A cross-modal, cross-view localization method and system based on local energy sorting sequence

Abstract

This invention discloses a cross-modal, cross-viewpoint localization method and system based on local energy ranking serialization. The method first uses a dual-branch feature extraction network to extract features from UAV infrared images and satellite visible light images; it then divides the two-dimensional feature map into multiple local windows and employs a window-level serpentine scanning strategy to maintain spatial continuity between windows; it calculates the semantic energy score of each pixel within a window using feature channel weights and performs pixel-level alternating ranking based on the semantic energy scores; it inputs the ranked feature sequence into a state space model for state space modeling; and finally, it calculates similarity and achieves localization matching. The system includes a feature extraction module, a local energy ranking serialization module, a sequence modeling module, a feature output module, and a matching module. By using this invention, the accuracy and robustness of matching and localizing UAV thermal infrared images with satellite visible light images are improved. This invention can be widely applied in the field of geolocation technology.

Description

Technical Field

[0001] This invention relates to the field of geolocation technology, and in particular to a cross-modal, cross-view positioning method and system based on local energy sorting sequence. Background Technology

[0002] When UAVs perform inspection, reconnaissance, search, and rescue missions in environments where GNAS signals are limited, such as at night, in low light, in smoke, or in severe weather, traditional visual positioning methods relying on visible light imagery often struggle to operate stably. Utilizing UAVs equipped with thermal infrared cameras to acquire real-time observational images and matching them with large-scale, high-precision satellite visible light maps to achieve cross-modal and cross-viewpoint geolocation has become a promising autonomous navigation solution. However, this task faces three major challenges: viewpoint differences, modal differences, and nighttime imaging differences. Significant differences in scale, rotation, and imaging geometry exist between the UAV's oblique viewpoint and the satellite's overhead viewpoint; thermal infrared images emphasize the surface temperature radiation characteristics of the target, while visible light images emphasize texture, color, and light reflection characteristics, resulting in a significant modal gap between the two.

[0003] Existing technologies for handling cross-viewpoint and cross-modal matching tasks typically employ convolutional neural networks or visual Transformers for feature extraction and obtain image descriptors through methods such as global pooling. In recent years, state-space models have been introduced into visual tasks due to their linear time complexity and strong long-sequence modeling capabilities. However, traditional state-space models usually require directly flattening two-dimensional feature maps into a one-dimensional sequence. This process disrupts crucial local spatial adjacency relationships in convolutional features, scattering spatially adjacent landmark structures in the one-dimensional sequence and thus destroying these relationships. For cross-modal and cross-viewpoint geolocation tasks, local landmark structures such as building boundaries, road inflections, and regular texture fragments are often important bases for accurate matching. If these local structures are separated during the serialization process, state-space models struggle to continuously model these key matching cues. Furthermore, if highly discriminative regions are separated by a large number of low-information background regions in the sequence, historical information during state updates gradually decays with the sequence length, weakening the association between key landmarks, increasing the risk of mismatches or location offsets, and ultimately reducing positioning accuracy. Summary of the Invention

[0004] In view of this, in order to address the technical problem that most existing cross-modal and cross-view methods directly flatten two-dimensional feature maps into one-dimensional sequences, thereby destroying local spatial adjacency relationships and leading to low localization accuracy, this invention proposes, firstly, a cross-modal and cross-view localization method based on local energy sorting and serialization. This method includes the following steps: A dual-branch feature extraction network is constructed to receive infrared images from the perspective of a UAV and visible light images from the perspective of a satellite, and extract the corresponding two-dimensional feature maps. Perform local energy sorting and serialization processing on the two-dimensional feature map; The local energy ranking serialization process includes: dividing the two-dimensional feature map into multiple non-overlapping local windows, and performing window-level serpentine scanning on the local windows to maintain the adjacency relationship of spatially adjacent windows in the one-dimensional sequence; calculating the semantic energy score of pixels in each local window, which is obtained based on the weighted result of pixel features and channel weights; and rearranging the pixels in each local window according to the semantic energy score, and adopting a pixel-level alternating sorting strategy to make high-energy pixel segments in adjacent windows distributed adjacently in the overall sequence. The one-dimensional feature sequence obtained after local energy sorting and serialization is input into the state space model for state space modeling. Before performing state space modeling, a temperature-scaled Softmax attention map is generated based on the semantic energy score, and the one-dimensional feature sequence is weighted using the Softmax attention map to enhance the input amplitude of high-energy regions during the state update process. Subsequently, the weighted one-dimensional feature sequence is input into one or more state space modeling units to capture long-distance dependencies across windows and regions, thereby obtaining the state space output sequence.

[0005] The output sequence is pooled to generate a global feature vector, and geolocation is achieved by calculating the similarity between the feature vector of the UAV thermal infrared image and the feature vector of the satellite visible light image.

[0006] Applying the above method, this invention also proposes a cross-modal, cross-view localization system based on local energy sorting serialization, which includes: a feature extraction module, a local energy sorting serialization module, a sequence modeling module, a feature output module, and a matching module.

[0007] In an overall technical approach, this invention employs a dual-branch processing framework to extract features from UAV thermal infrared images and satellite visible light images, respectively, to obtain corresponding two-dimensional depth feature maps. Subsequently, the two-dimensional feature maps are subjected to local energy sorting and serialization processing, converting them into one-dimensional feature sequences more suitable for state-space modeling. Then, the one-dimensional feature sequences are input into the state-space model for state-space modeling, and the output results are pooled and mapped to obtain a global feature vector for cross-modal matching. Finally, by calculating the similarity between the query image and the candidate satellite images, the optimal matching result and its corresponding geographical location are output.

[0008] Based on the above scheme, this invention provides a cross-modal and cross-viewpoint geolocation method and system based on local energy sorting sequence. Through a joint design of local window partitioning, window-level serpentine scanning, and pixel-level alternating sorting, it effectively preserves the spatial continuity of the two-dimensional feature map and shortens the relative distance between salient landmark regions in the sequence without significantly increasing computational complexity. Furthermore, by introducing a semantic energy evaluation mechanism based on channel weights, the sequence construction process prioritizes the retention of landmark features with high discriminative power for cross-modal matching, suppressing the adverse effects of low-value background areas on state propagation. Since cross-modal and cross-viewpoint geolocation tasks mainly rely on matching local salient landmarks such as building boundaries, road inflections, and regular texture fragments, the above processing enables key landmark features to be expressed more continuously and stably during state space modeling, reducing matching interference and long sequence information attenuation caused by low-information background areas. This reduces the probability of mismatches and positioning offsets, improving the matching accuracy between infrared and visible light images and the final positioning precision. Attached Figure Description

[0009] Figure 1 This is a flowchart of the steps of a cross-modal and cross-view localization method based on local energy sorting serialization according to the present invention; Figure 2 This is a schematic diagram of the data flow of a cross-modal, cross-view localization method based on local energy sorting and serialization according to the present invention; Figure 3 This is a schematic diagram of the data flow during local energy sorting serialization processing in a specific embodiment of the present invention; Figure 4 This is a sample image of the Boson-nighttime dataset in a specific embodiment of the present invention; Figure 5 This is an example diagram of satellite images, real thermal infrared images, and generated thermal infrared images in the test area in a specific embodiment of the present invention; Figure 6 This is a schematic diagram of a typical difficult matching example in the Boson-nighttime dataset in a specific embodiment of the present invention. Detailed Implementation

[0010] In addition to the technical issues mentioned in the background, current processing in this field also suffers from the problem of information decay over long sequences. Geolocation tasks often rely on sparse but salient key landmark areas. If these highly discriminative regions are separated by a large number of low-information background areas in the sequence, the historical information during the state update process will gradually decay as the sequence length increases, thereby weakening key matching clues. Especially in cross-modal tasks involving nighttime thermal infrared imagery and satellite visible light imagery, salient regions are often even sparser, and their response patterns are not consistent with visible light textures, further amplifying the aforementioned problems.

[0011] Therefore, there is an urgent need in this field for a new cross-modal geolocation technology that can perform a more spatially structured and task-semantic sequential reconstruction of the state space model before the two-dimensional visual features are input into it. This would preserve local spatial coherence, shorten the sequence distance between significant landmarks, and reduce information decay during the propagation of long sequence states.

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] It should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0014] It should be understood that the terms "system," "apparatus," "unit," and / or "module" used in this application are a method of distinguishing different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions.

[0015] Unless the context explicitly indicates an exception, words such as "a," "an," "a kind," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list; a method or apparatus may also include other steps or elements. An element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, product, or apparatus that includes the element.

[0016] In the description of the embodiments of this application, "a plurality of" refers to two or more. The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0017] Furthermore, flowcharts are used in this application to illustrate the operations performed by the system according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed precisely in sequence. Instead, the steps can be processed in reverse order or simultaneously. Additionally, other operations can be added to these processes, or one or more steps can be removed from them.

[0018] Reference Figure 1 This is a flowchart illustrating an optional example of the cross-modal, cross-view localization method based on local energy sorting serialization proposed in this invention. This method can be applied to computer devices, and the localization method proposed in this embodiment may include, but is not limited to, the following steps: Step S1: Acquire infrared images from the perspective of the UAV and visible light images from the perspective of the satellite, and extract features from them to obtain corresponding two-dimensional feature maps; Step S2: Perform local energy sorting and serialization on the two-dimensional feature map to obtain a one-dimensional feature sequence; Step S3: Perform state-space modeling on the one-dimensional feature sequence to obtain the output sequence; Step S4: Perform pooling on the output sequence to generate a global feature vector; Step S5: Calculate the similarity between the global feature vector of the infrared image and the global feature vector of the visible light image, and confirm the location data based on the similarity.

[0019] This invention provides a dual-branch network framework for cross-modal and cross-view geolocation, wherein the data flow direction of the framework refers to... Figure 2 The framework receives UAV thermal infrared images and satellite visible light images as inputs, and performs feature extraction, fusion, sequential modeling, and global representation learning on the two modal images through two processing branches with shared or partially shared parameters, thereby achieving cross-modal matching and geolocation.

[0020] During the training phase, such as Figure 2 As shown in the lower right, the system constructs anchor samples, positive samples, and negative samples based on the features of UAV thermal infrared images and satellite visible light images. It then uses InfoNCE loss for supervised training to narrow the feature distance between matching samples and widen the feature distance between unmatched samples, thereby improving cross-modal feature alignment capabilities.

[0021] During the testing phase, such as Figure 2 As shown in the upper right corner, the system takes the thermal infrared image of the UAV to be located as the query input, and inputs the visible light satellite images in the candidate area into the corresponding branch for feature extraction and encoding. Then, it uses cosine similarity to measure the similarity between the query features and the candidate features, and outputs the geographical location corresponding to the satellite image with the highest similarity as the positioning result.

[0022] In some feasible embodiments, step S1 involves a feature extraction module, which specifically includes: Using a LoRA-tuned DINOv3 and ConvNeXt-B backbone structure, feature encoding is performed on the input UAV thermal infrared image and satellite visible light image to obtain the corresponding multi-level two-dimensional feature representation. Figure 2 The dual-branch structure shown can employ a shared weight design to enhance the semantic consistency of different modal inputs in a unified feature space. The input image can be pre-adjusted to a uniform size, such as 256×256, but the present invention is not limited to this specific size.

[0023] Furthermore, after obtaining multi-level features, the features from different levels are fused using a Semantic Gated Residual Fusion (SGRF) module to generate a two-dimensional fused feature map that combines high-level semantic information with low-level spatial detail information. Preferably, the Semantic Gated Residual Fusion module includes processing units such as multi-scale feature adaptation, global average pooling, channel weight generation, convolution transformation, and gating fusion, which are used to suppress redundant background responses and enhance significant structural regions that contribute to cross-modal geolocation.

[0024] In some feasible embodiments, step S2 involves a local energy sorting serialization module that receives a two-dimensional fused feature map and converts it into a one-dimensional feature sequence suitable for state-space modeling. Specifically, the fused feature map is first energy-guided according to channel weights, and then local window partitioning, in-window sorting, window-level serpentine scanning, and pixel-level alternating sorting are performed on the feature map to obtain a rearranged sequence, thereby enhancing the continuous distribution of high-discriminative regions in the sequence and shortening the state propagation distance between key landmark features. Step S2 specifically includes: S2.1 Local window division and window-level snake scanning.

[0025] First, the received size is... The two-dimensional feature map is divided into multiple non-overlapping local windows, each with a size of [size missing]. This local partitioning method preserves local structural information in the original two-dimensional features as much as possible before serialization. To avoid the forced separation of spatially adjacent windows during line breaks in traditional raster scanning, the module further employs a window-level serpentine scanning strategy. Even-numbered rows of windows are scanned in forward order, while odd-numbered rows are scanned in reverse order, ensuring that adjacent windows in the two-dimensional plane maintain a relatively close relative position in the converted one-dimensional sequence. The reason for using local window partitioning instead of global flattening is that discriminative structures in cross-modal and cross-view geolocation tasks often exist in the form of local regions, such as building boundaries, road corners, or regular texture fragments. Direct global expansion easily destroys local adjacency relationships in long sequences; however, local window partitioning preserves spatial structure within a certain range with lower overhead. Furthermore, using window-level serpentine scanning instead of fixed line-by-line scanning reduces spatial breaks between the end of a line and the beginning of the next line, ensuring strong local coherence of adjacent windows in the two-dimensional plane after serialization, thus providing a more reasonable sequence order for subsequent state space modeling.

[0026] S2.2 Semantic energy calculation.

[0027] To avoid misclassifying large background areas as high-value regions based solely on pixel amplitude, this invention further introduces a semantic energy evaluation mechanism. Within each local window, for regions at position... pixel features Utilizing channel weights They are weighted channel by channel, and their semantic energy scores are calculated. The semantic energy score is used to measure the discriminative contribution of the corresponding pixel position to the cross-modal matching task and serves as the basis for subsequent pixel rearrangement within the window. The preferred calculation method is as follows: in, This represents element-wise multiplication. A weight vector, reflecting the semantic importance of each channel, is used to represent the semantic importance of each feature channel. The channel weights can be derived from global statistics of the preceding feature map, attention weights, or other equivalent methods. This semantic energy score allows for a more accurate distinction between high-discriminative regions with localization value, such as building edges, road structures, and intersections, and low-discriminative background regions, such as grasslands, water bodies, and shadows, thus providing a task-related basis for subsequent ranking. In a preferred embodiment, the channel weights can be generated by performing global average pooling on the two-dimensional feature map, followed by linear transformation, nonlinear activation, and normalization operations; in another embodiment, the channel weights can also be output from existing channel attention mechanisms or feature importance evaluation modules. Regardless of the specific implementation, the goal is to assign semantic importance related to cross-modal localization to different channels, thereby enabling the semantic energy score to more accurately reflect the actual contribution of pixel location to the final matching task.

[0028] S2.3, pixel-level alternating sorting.

[0029] After completing the semantic energy calculation, this invention further rearranges the pixels within each local window. Specifically, the pixel sequence within the window is sorted according to the semantic energy score, either from low to high or from high to low. To reduce the probability of high-energy regions being separated by low-energy regions between adjacent windows, this invention further employs a pixel-level alternating sorting strategy. That is, for windows at even positions in the sequence, their internal pixels are sorted from low to high energy; for windows at odd positions in the sequence, their internal pixels are sorted from high to low energy. This can be expressed as: Through the above alternating sorting method, the high-energy tail of the previous window can be directly connected to the high-energy head of the next window, so that the significant landmark features form more compact high-information segments in the overall sequence, shortening the state propagation path between key areas.

[0030] In some feasible embodiments, steps S3-S4 specifically include: The feature sequence, after being serialized by local energy sorting, is input into the state space model for context modeling. Subsequently, GeM pooling is used to obtain global feature representations, which are then further mapped by a multilayer perceptron to generate fixed-length feature vectors for matching, such as 1×512-dimensional feature vectors. The UAV thermal infrared image branch and the satellite visible light image branch output corresponding global feature descriptors, respectively.

[0031] After completing local window scanning and pixel rearrangement, this invention inputs the recombined one-dimensional feature sequence into a state-space model for sequence modeling. In a preferred embodiment, a temperature-scaled Softmax attention map can be generated based on the semantic energy score and applied to the input features to enhance the input amplitude of high-energy regions during state update. Subsequently, the feature sequence is input into one or more state-space modeling units to capture long-distance dependencies across windows and regions. Finally, pooling is performed on the output sequence of the state-space model, and a linear mapping or normalization layer is combined to generate a global feature vector for cross-modal image matching and geolocation. Since state-space models typically employ recursive or selective propagation mechanisms in the sequence dimension, information from preceding positions gradually decays as the step size increases during propagation. Therefore, if highly discriminative landmark regions are far apart in the original sequence, their interactions will be weakened in subsequent modeling. This invention shortens the relative distance between high-value regions in the reconstructed sequence by using local energy sorting and cross-window alternating stitching, thereby enabling the state propagation process to cover key landmark segments more continuously and improving the modeling stability of salient structures in cross-modal matching.

[0032] In some feasible embodiments, it also includes: During the training phase, the system also utilizes an image generation model to convert satellite visible light images into pseudo-thermal infrared images to construct satellite visible light-pseudo-infrared training sample pairs, which are then used together with real thermal infrared-visible light sample pairs for network training.

[0033] To address the issues of high cost, limited coverage area, and difficulty in supporting large-scale cross-modal training of real UAV thermal infrared aerial photography data, this invention further introduces a generative data augmentation strategy during the training phase. Specifically, an image generation model is first trained using a small number of real paired thermal infrared UAV images and visible light satellite images, preferably an image-to-image mapping model based on Pix2Pix. Subsequently, a large number of satellite image slices are cropped from the visible light regions of satellites not yet actually covered by UAV flights, and these slices are converted into pseudo-infrared images with a style similar to real thermal infrared images using the generative model, thereby constructing a large-scale "satellite visible light - pseudo-infrared" training sample pair. The pseudo-samples and real thermal infrared - visible light paired samples are used together for network training to enhance the alignment and generalization capabilities of the method in the cross-modal feature space. Furthermore, when constructing the generated samples, it is preferable to remove the geographical range corresponding to the test area from the generated source area to avoid spatial leakage interfering with the evaluation results. During the inference phase, the system still uses real UAV thermal infrared images and satellite visible light images for direct matching.

[0034] UAV thermal infrared images and satellite visible light images can be uniformly scaled to a preset resolution before being input into a two-branch feature extraction network. The local window size can be set to 4, 8, 12, or 16 depending on the feature map resolution. The state space model can adopt a single-layer or multi-layer stacked structure. Generalized mean pooling is preferred for pooling operations. The final output feature vector can be further processed by linear mapping and normalization before being used for similarity calculation. The backbone network preferably adopts a DINOv3-ConvNeXt structure with low-rank adaptation fine-tuning to reduce training overhead while maintaining pre-trained representation capabilities. During training, random flipping, color perturbation, random erasure, or other data augmentation operations can be performed on the thermal infrared and visible light images respectively, according to task requirements, to further improve the model's adaptability to changes in viewpoint, modal differences, and environmental noise.

[0035] In this embodiment, pseudo-infrared training samples are constructed by combining a generative model to expand cross-modal training data and enhance the model's generalization ability.

[0036] During the inference phase, the thermal infrared image of the UAV to be located is used as the query image input system, and the set of satellite visible light images within the candidate region is input into the corresponding branch for feature extraction and sequential modeling. Subsequently, global feature vectors of the query image and each candidate satellite image are obtained, and matched and ranked according to cosine similarity or other distance metrics. The geographical location corresponding to the satellite image with the highest similarity can be used as the positioning result of the UAV's current position; when higher robustness is required, the top few candidate positions can also be output as Top-k retrieval results for subsequent position fusion, trajectory smoothing, or temporal positioning optimization. Because this invention prioritizes the preservation of highly discriminative landmark regions and enhances their continuous distribution in the sequence during the sequentialization phase, it can maintain high matching stability under complex backgrounds and significant modal differences during the inference phase.

[0037] Based on the overall process described above, this invention also provides relevant experimental data examples: To verify the effectiveness of this invention, experiments were conducted on the Boson-nighttime cross-modal dataset, specifically designed for the constrained environment of global navigation satellite systems at night. This dataset is designed for cross-modal, cross-view geolocation tasks involving UAV thermal infrared and satellite visible light. The real data was collected by UAVs equipped with thermal imaging devices during multiple nighttime flights, and spatial alignment with satellite visible light maps was achieved through 3D reconstruction and orthoregistration. The dataset is divided into training, validation, and test sets according to the actual flight areas to avoid spatial leakage. Furthermore, pseudo-infrared enhanced samples constructed by a generative model are introduced during the training phase to expand the scale of the cross-modal training data.

[0038] Boson-nighttime is a real-world nighttime dataset constructed for cross-modal, cross-view geolocation tasks using UAV thermal infrared imagery and satellite visible light imagery. This dataset serves UAV visual positioning research under conditions of nighttime, low light, smoke obstruction, or limitations imposed by the Global Navigation Satellite System, and effectively reflects the target application scenarios of this invention. According to publicly available information, this dataset was acquired through nighttime flight data collection missions using UAVs equipped with thermal imaging sensors. The thermal imaging device used was the FLIR Boson thermal imager, with a focal length of approximately 8.7 mm, a resolution of 640p, and a horizontal field of view of approximately 50°. During data collection, the UAV performed six flight missions between 9:00 PM and 4:00 AM, acquiring thermal infrared images from an approximate top-down perspective with a ground spatial resolution of approximately 1 m / px. After acquisition, researchers reconstructed the multi-view thermal infrared images using 3D reconstruction methods and further aligned the thermal infrared maps with the corresponding satellite visible light maps using orthorectification, thus forming a data foundation usable for cross-modal matching and geolocation. Figure 4 As shown, the Boson-nighttime dataset contains both satellite visible light images and nighttime thermal infrared images that correspond to the UAV thermal infrared image space, which can intuitively reflect the cross-modal imaging differences in this task.

[0039] In terms of scene composition, the Boson-nighttime dataset covers a total area of ​​approximately 33 km², primarily consisting of desert terrain, while also including a small number of structured target areas such as farmland, roads, and buildings. Compared to conventional daytime visible light remote sensing data, a significant characteristic of this dataset is that under thermal infrared imaging conditions, a large number of areas exhibit low contrast, weak texture, and self-similarity characteristics. Particularly in desert areas, different geographical locations often display highly similar thermal response patterns, leading to potential confusion during cross-modal matching. Figure 5 As shown, the structural texture of some areas in real thermal infrared images is weak, but after contrast enhancement, the local texture and edge information become more obvious. At the same time, generative thermal infrared samples can also simulate the appearance features of real thermal infrared images to a certain extent. This indicates that the task cannot be completed simply by relying on global appearance information, but requires highlighting a small number of highly discriminative structural regions, such as road boundaries, building outlines, farmland block textures, and local thermal anomaly areas.

[0040] In terms of data organization, Boson-nighttime crops thermal infrared orthophotos and corresponding satellite images into 512×512 pixel patches, and uses a tiling step of approximately 35 pixels to divide them, so that each thermal infrared patch forms a sample pair with a spatially aligned satellite patch. Publicly available information indicates that the training set, validation set, and test set of this dataset contain 10256, 13011, and 26568 pairs of image patches, respectively. In addition to real paired samples, the original study also used unpaired satellite images to construct generative pseudo-thermal infrared samples to expand the scale of cross-modal training data and improve the model's generalization ability under the limited conditions of real thermal infrared samples. This data organization method is highly consistent with the design idea of ​​this invention, which combines generative pseudo-thermal infrared sample expansion strategy during the training phase.

[0041] The Boson-nighttime dataset has the following characteristics. First, it is derived from real-world UAV nighttime flight data, realistically reflecting imaging noise, structural gaps, and environmental complexity in nighttime thermal infrared localization tasks. Second, it exhibits significant cross-modal and cross-viewpoint differences; the query source is UAV thermal infrared imagery, while the reference source is satellite visible light imagery, both showing marked differences in imaging mechanisms, texture representation, and spatial perspective. Third, the dataset is predominantly desert, with large background areas exhibiting low contrast, self-similarity, and weak texture, making traditional methods relying on global flattening or local texture matching prone to mismatches and localization shifts. Finally, high-discriminative regions in this dataset are often locally sparsely distributed; truly useful structural information for geolocation exists only in a few windows or local landmark locations, while the large background areas provide limited discriminative information. Therefore, if traditional global raster flattening is used before inputting the 2D feature map into the state-space model, key salient regions are easily separated by low-value background features in long sequences, thus weakening the long-range dependency modeling effect.

[0042] Based on the aforementioned data characteristics, the Boson-nighttime dataset can provide a representative validation scenario for the local energy ranking serialization method proposed in this invention. For example... Figure 6As shown, in low-contrast, self-similar, and weak-texture scenes, baseline models are prone to matching offsets or mismatches, while this invention achieves more stable and accurate matching results. On one hand, this invention preserves local adjacency relationships in two-dimensional space as much as possible through local window partitioning and window-level serpentine scanning; on the other hand, through semantic energy score calculation and pixel-level alternating sorting, sparsely distributed high-discriminative regions form more continuous high-information segments in a one-dimensional sequence, thereby shortening the transmission distance between key landmarks and mitigating information attenuation during long-sequence state propagation. Therefore, this dataset effectively demonstrates the applicability and advantages of this invention in nighttime thermal infrared-satellite visible light cross-modal cross-view geolocation tasks.

[0043] The performance of this invention (DINO-GFESM) was rigorously compared with current mainstream baseline networks (such as ResNet-18, which combines cross-entropy CE and domain adversarial neural network DANN), and the experimental results are shown in Table 1.

[0044] Table 1: Performance comparison of the proposed method with existing baseline methods on the Boson-nighttime dataset. Experimental results show that, without introducing generative pseudo-infrared data augmentation, the present invention achieves an R@1 accuracy of 89.86%, an R@5 accuracy of 95.16%, and a positioning error of 15.2 meters, which is superior to the traditional baseline model compared to it. This indicates that the local energy ranking serialization mechanism proposed in this invention can effectively improve the feature discrimination capability in infrared-visible cross-modal matching. Furthermore, after introducing generative pseudo-infrared data augmentation, the performance of the present invention is further improved, with R@1 reaching 93.93%, R@5 reaching 96.2%, and R512@1 reaching 98.93%, and the actual positioning error reduced to 8.2 meters. The above results demonstrate that the present invention can not only improve the accuracy of cross-modal image retrieval but also significantly reduce the actual geographic positioning error. The above experimental results further illustrate that the local energy ranking serialization mechanism proposed in this invention not only improves the global discriminativeness of feature representation but, more importantly, improves the information transmission efficiency of key landmark areas in the sequence modeling process. Especially in scenarios where there are significant differences in response between thermal infrared and visible light, this invention can prioritize and enhance structural information with practical positioning value in subsequent state space modeling through semantic energy evaluation and sequence rearrangement strategies, thereby demonstrating stability and effectiveness superior to traditional global flattening methods.

[0045] In summary, the present invention provides a cross-modal and cross-view geolocation method and system based on local energy sorting sequence. It focuses on the problems of unreasonable direct flattening of two-dimensional features, destruction of local spatial continuity, and information decay of long sequences in existing state space models for visual positioning tasks. It proposes a sequence reconstruction scheme for nighttime thermal infrared-visible light matching tasks, which has clear structural innovation and application value.

[0046] Furthermore, this invention proposes a local energy sorting serialization scheme for state-space models. This scheme preserves spatial adjacency relationships in the two-dimensional feature map through local window partitioning and window-level serpentine scanning, and reconstructs the original sequence generation order in a targeted manner through semantic energy evaluation and pixel-level alternating sorting. This allows high-energy, highly discriminative landmark features to be more densely distributed in the sequence, effectively alleviating the spatial structure destruction and long-sequence information attenuation problems caused by directly flattening the two-dimensional feature map in traditional state-space models.

[0047] This invention possesses strong cross-modal matching capabilities, adapting to significant modal differences between UAV thermal infrared images and satellite visible light images. Experimental results on the Boson-nighttime dataset demonstrate that this invention significantly reduces actual geolocation errors while maintaining high retrieval accuracy. Specifically, after introducing generative pseudo-infrared data augmentation, the R@1 reaches 93.93%, and the positioning error is reduced to 8.2 meters, indicating that this invention can provide effective technical support for UAV visual positioning and autonomous navigation in nighttime or environments where global navigation satellite systems are restricted.

[0048] This invention further incorporates a generative pseudo-infrared data augmentation strategy. Under the condition of limited real thermal infrared samples, it enhances the model's alignment and generalization capabilities across modal feature spaces by constructing satellite visible light-pseudo-infrared training sample pairs. This design helps reduce the limitation of nighttime thermal infrared aerial data acquisition costs on model training effectiveness, thereby improving the feasibility of deploying this invention in real-world complex environments.

[0049] A cross-modal, cross-view localization system based on local energy sorting sequencer includes: The feature extraction module is used to perform step S1; The local energy sorting serialization module is used to execute step S2; Sequence modeling module, used to execute step S3; The feature output module is used to execute step S4; The matching module is used to perform step S5.

[0050] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0051] A cross-modal, cross-view localization device based on local energy sorting sequence: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements a cross-modal, cross-view localization method based on local energy sorting serialization as described above.

[0052] The content of the above method embodiments is applicable to the device embodiments. The specific functions implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0053] A storage medium storing processor-executable instructions, which, when executed by a processor, are used to implement a cross-modal, cross-view localization method based on local energy sorting serialization as described above.

[0054] The content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0055] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A cross-modal, cross-view localization method based on local energy sorting sequencer, characterized in that, Includes the following steps: Infrared images from the perspective of a drone and visible light images from the perspective of a satellite are acquired, and features are extracted from each to obtain the corresponding two-dimensional feature maps. The two-dimensional feature map is subjected to local energy sorting and serialization processing to obtain a one-dimensional feature sequence; The one-dimensional feature sequence is modeled in a state space to obtain an output sequence; The output sequence is pooled to generate a global feature vector; Calculate the similarity between the global feature vector of the infrared image and the global feature vector of the visible light image, and confirm the location data based on the similarity.

2. The cross-modal, cross-view localization method based on local energy sorting serialization according to claim 1, characterized in that, The step of performing local energy sorting and serialization on the two-dimensional feature map to obtain a one-dimensional feature sequence specifically includes: The two-dimensional feature map is divided into multiple local windows, and window-level scanning is performed on the local windows; Calculate the semantic energy score of each pixel within each local window; Based on the semantic energy score, the pixels within each local window are rearranged to obtain a one-dimensional feature sequence.

3. The cross-modal, cross-view localization method based on local energy sorting sequence according to claim 2, characterized in that, The rules for window-level scanning are as follows: Even-numbered rows are scanned in forward order, while odd-numbered rows are scanned in reverse order.

4. The cross-modal, cross-view localization method based on local energy sorting serialization according to claim 2, characterized in that, The rearrangement employs a pixel-level alternating sorting strategy, specifically as follows: For a local window at an even position in the sequence, its internal pixels are sorted from low to high according to their semantic energy scores; For local windows at odd positions in the sequence, the pixels inside are sorted from high to low according to their semantic energy scores, so that high-energy pixel segments in adjacent windows are distributed adjacently in the overall sequence.

5. The cross-modal, cross-view localization method based on local energy sorting sequence according to claim 2, characterized in that, The formula for calculating the semantic energy score is as follows: in, Indicates the position within the local window. The semantic energy score of each pixel; This represents the pixel feature at that location, i.e., the first pixel. Line 1 The feature vector corresponding to the column pixels; Indicates channel weight; This represents element-wise multiplication; This represents the square norm of the weighted pixel feature vector.

6. The cross-modal, cross-view localization method based on local energy sorting serialization according to claim 2, characterized in that, Before the step of performing state-space modeling on the one-dimensional feature sequence, the following steps are also included: A temperature-scaled Softmax attention map is generated based on the semantic energy score, and the Softmax attention map is used to weight the one-dimensional feature sequence.

7. The cross-modal, cross-view localization method based on local energy sorting serialization according to claim 1, characterized in that, The pooling process is generalized mean pooling.

8. The cross-modal, cross-view localization method based on local energy sorting serialization according to claim 1, characterized in that: A dual-branch feature extraction network is used to extract features from infrared images from the perspective of UAVs and visible light images from the perspective of satellites. The backbone network of the dual-branch feature extraction network is a DINOv3-ConvNeXt network that has been fine-tuned with low-rank adaptation. A state-space model is used to model the state space of a one-dimensional feature sequence. The state-space model is a Mamba structure.

9. A cross-modal, cross-viewpoint localization system based on local energy sorting sequencer, characterized in that, include: The feature extraction module is used to acquire infrared images from the perspective of the UAV and visible light images from the perspective of the satellite, and to extract features from them to obtain corresponding two-dimensional feature maps. The local energy sorting serialization module is used to perform local energy sorting serialization processing on the two-dimensional feature map to obtain a one-dimensional feature sequence; The sequence modeling module is used to perform state-space modeling on the one-dimensional feature sequence to obtain an output sequence; The feature output module is used to pool the output sequence to generate a global feature vector; The matching module is used to calculate the similarity between the global feature vector of the infrared image and the global feature vector of the visible light image, and to confirm the location data based on the similarity.

10. A cross-modal, cross-viewpoint positioning device based on local energy sorting serialization, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements a method as described in any one of claims 1-8.

Images