Cross-view image matching method and system fused with eye movement attention mechanism

Abstract

The invention relates to the technical field of remote sensing image processing in computer vision, in particular to a cross-view image matching method and system fused with an eye movement attention mechanism, and the method comprises the steps: training two independent Transfomer encoder branches through a ground-remote sensing cross-view image data set; collecting eye movement track data when a human executes a cross-view image matching task, and generating a human visual attention map; performing image-level fusion on the remote sensing human visual attention map and the remote sensing self-attention map generated by the trained remote sensing Transfomer encoder through a mixed attention module to obtain a mixed attention map; processing the original remote sensing image and the mixed attention map through an image adaptive cutting and resolution enhancement module to obtain a high-resolution remote sensing image; and inputting the high-resolution remote sensing image and the ground image into two pre-trained independent Transfomer encoder branches for secondary training. The attention of the guiding model is more focused on the image key area beneficial to matching, so that the matching precision of the ground image and the remote sensing image is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing in computer vision, and in particular to a cross-viewing image matching method and system integrating an eye movement attention mechanism. Background Art

[0002] Cross-view image matching refers to the process of accurately matching images from different perspectives by determining whether images taken from different perspectives have a common mapping of the same area or location. This paper focuses on the cross-view matching problem between horizontal ground maps and overhead remote sensing images. This technology plays an important role in autonomous driving, intelligent navigation, 3D reconstruction and other fields. Due to the different imaging principles of images from different perspectives and the differences in perspective, it is difficult to effectively extract and match key features, making it a challenging task.

[0003] The difficulty of cross-view image matching lies in the extraction and alignment of key features of images from different viewpoints. Deep learning technology is currently the main means of extracting features of images from different viewpoints. Lin et al. used AlexNet to extract image features, Workman et al. used GoogleNet to extract image features, and Hu et al. used VGG16 to extract image features. Since CNN tends to mine fine-grained information of images from different viewpoints, it ignores the extraction of robust features and feature alignment of images, making it difficult to establish a global correlation model. Since Vision Transformer (ViT) can encode the position of images, has excellent global correlation modeling capabilities and self-attention mechanisms, Vision Transformer is introduced into the field of cross-view image matching. Its self-attention mechanism is conducive to extracting robust features from images and establishing a global correlation model of images. Dai et al. proposed a feature segmentation and region alignment (FSRA) structure, which uses the attention distribution of the Transformer feature map to divide the image region, and realizes the precise alignment of specific regions between multiple views, enhancing the model's ability to capture global information. Yang et al. designed a network based on self-cross attention mechanism, which uses the self-attention mechanism of Transformer itself to model global correlation and reduce the visual ambiguity between images of different perspectives. Zhuang et al. proposed a Transformer network based on pixel attention mechanism, which achieves accurate matching of the same semantic parts in two images by finely classifying each pixel in the image. Zhu et al. used Data-efficient image Transformers (DeiT) to replace the original ViT as the backbone network, reducing the high dependence on the amount of data.

[0004] In addition, the existing cross-view image matching models based on deep learning tend to over-focus on the significant features of images such as cars, roads, and skies during feature extraction and alignment, which interferes with the key features that are conducive to image matching. This preference is largely a manifestation of the "shortcut learning" phenomenon, that is, the model tends to quickly capture and rely on features that are easy to learn but not essential to the task as "shortcuts", rather than focusing on deep mining and accurate alignment of truly discriminative key features in the image, thereby weakening the model's matching accuracy and generalization ability. As Sun et al. pointed out, the image matching model based on deep learning is affected by "shortcut learning", and the matching results often do not match the user's goals and do not work well on new data sets. Summary of the invention

[0005] The present invention aims to solve the problems existing in the prior art and proposes a cross-view image matching method and system integrating eye movement attention mechanism, which utilizes the image-level fusion of remote sensing self-attention map and remote sensing human visual attention map to guide the model's attention to be more focused on the key areas of the image that are conducive to matching, so as to improve the matching accuracy between ground images and remote sensing images.

[0006] To achieve the above purpose, the technical solution adopted is:

[0007] A cross-view image matching method integrating eye movement attention mechanism, including:

[0008] Using the existing ground-remote sensing cross-view image dataset, two independent Transfomer encoder branches are trained to generate self-attention maps for ground images and remote sensing images respectively;

[0009] Design an experimental paradigm for cross-view image matching, collect eye movement data of humans when performing cross-view image matching tasks, and generate human visual attention maps;

[0010] The human visual attention map corresponding to the remote sensing image and the remote sensing self-attention map generated by the trained remote sensing Transfomer encoder are fused at the image level through the hybrid attention module to obtain a hybrid attention map;

[0011] The original remote sensing image and the mixed attention map are processed through image adaptive cropping and resolution enhancement module to obtain a high-resolution remote sensing image.

[0012] The high-resolution remote sensing images and ground images are input into two independent pre-trained Transformer encoder branches for secondary training.

[0013] According to the cross-view image matching method integrating the eye movement attention mechanism of the present invention, further, two independent Transfomer encoder branches generate self-attention maps for the ground map and the remote sensing map respectively, including:

[0014] Both independent Transformer encoder branches use the VisionTransformer model, which first uses a feature embedding layer to convert ground images and remote sensing images into low-dimensional vectors, which are then sent to the Transformer encoder for feature encoding. Finally, a multi-layer perceptron head is used to perform cross-view image matching on the output features of the two branches. The self-attention map of the training set at the last training round is exported and saved.

[0015] According to the cross-view image matching method integrating the eye movement attention mechanism of the present invention, further, two independent Transfomer encoder branches are trained using weighted soft margin triplet loss as the objective function.

[0016] According to the cross-view image matching method integrating the eye movement attention mechanism of the present invention, further, an experimental paradigm of cross-view image matching is designed, and the process of generating a human visual attention map is as follows:

[0017] Randomly extract a certain number of remote sensing-ground image pairs from the ground-remote sensing cross-view image dataset;

[0018] The extracted remote sensing-ground image pairs are preprocessed to make the width of the remote sensing image consistent with that of the ground image, and then spliced ​​with the remote sensing image at the bottom and the ground image at the top as the stimulus image for the eye movement experiment;

[0019] The stitched images are presented to the subjects one by one using a desktop eye tracker, and the subjects are asked to judge whether the aerial perspective content and the ground perspective content contained in the image are the same location. At the same time, the subjects' eye movement trajectories are collected, and the collected eye movement data are used to generate a human visual attention map, including a human visual attention map corresponding to the remote sensing image and a human visual attention map corresponding to the ground image.

[0020] According to the cross-view image matching method integrating the eye movement attention mechanism of the present invention, further, the remote sensing human visual attention map I g Remote Sensing Self-Attention Map I n The hybrid attention map a(I g ,I n ) is expressed as:

[0021]

[0022] in, is the query vector of the human visual attention map for remote sensing, is the key value vector of the remote sensing self-attention map, is the value vector of the remotely sensed human visual attention map.

[0023] According to the cross-view image matching method integrating the eye movement attention mechanism of the present invention, further, the calculated mixed attention information is updated by the following formula, and the updated mixed attention map is expressed as:

[0024] u(I g ,I n )=LayerNorm(I g +a(I g ,I n ))

[0025] Among them, LayerNorm represents layer normalization.

[0026] According to the cross-view image matching method integrating the eye movement attention mechanism of the present invention, further, the working process of the image adaptive cropping and resolution enhancement module is as follows:

[0027] First, the original remote sensing image and the hybrid attention map are grayed out and divided into a series of image blocks of equal size. Then, the average pixel value of the hybrid attention map and the average pixel value of each image block of the hybrid attention map are calculated.

[0028] Then, the image blocks of the hybrid attention map are traversed. If there is an image block whose average pixel value is less than the average pixel value of the hybrid attention map, the pixel values ​​of the corresponding image block in the remote sensing image are all set to zero; if there is no image block, the image block of the remote sensing image remains unchanged, realizing adaptive image cropping;

[0029] Finally, the cropped image resolution is enhanced.

[0030] Furthermore, the present invention also provides a cross-view image matching system integrating an eye movement attention mechanism, comprising a pre-training module, a human visual attention map generation module, a hybrid attention map generation module, a high-resolution remote sensing image module and a secondary training module, wherein:

[0031] A pre-training module is used to train two independent Transfomer encoder branches using the existing ground-remote sensing cross-view image dataset to generate self-attention maps for ground images and remote sensing images respectively;

[0032] The human visual attention map generation module is used to design the experimental paradigm of cross-view image matching, collect the eye movement trajectory data of humans when performing cross-view image matching tasks, and generate the human visual attention map;

[0033] A hybrid attention map generation module is used to fuse the human visual attention map corresponding to the remote sensing image with the remote sensing self-attention map generated by the trained remote sensing Transfomer encoder through the hybrid attention module at the image level to obtain a hybrid attention map;

[0034] High-resolution remote sensing image module, used to obtain high-resolution remote sensing images through image adaptive cropping and resolution enhancement modules of original remote sensing images and mixed attention maps;

[0035] The secondary training module is used to input high-resolution remote sensing images and ground images into two pre-trained independent Transfomer encoder branches for secondary training.

[0036] The beneficial effects achieved by adopting the above technical solution are:

[0037] The cross-view image matching method that integrates the eye movement attention mechanism proposed in the present invention simulates the visual focus cognitive processing mode of the human visual system, designs a hybrid attention module, and utilizes the image-level fusion of the remote sensing self-attention map and the remote sensing human visual attention map to guide the model's attention to be more focused on the key areas of the image that are conducive to matching. In order to reduce the interference of redundant information, an image adaptive cropping and resolution enhancement module is designed. This module draws on the "attention and magnification" mechanism in human visual focus, and adaptively removes interference from unimportant areas in remote sensing images according to the dynamic changes of the hybrid attention map, and combines image resolution enhancement to further strengthen the model's attention to the key areas of the image. The present invention not only reduces the waste of computing resources, but also enhances the model's ability to capture detailed information, avoids the "shortcut learning" trap, and improves the accuracy and efficiency of the model in processing complex visual tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings of the embodiments of the present invention, wherein the drawings are only used to illustrate some embodiments of the present invention, but not to limit all embodiments of the present invention thereto.

[0039] Figure 1 is a flow chart of a cross-view image matching method integrating an eye movement attention mechanism according to an embodiment of the present invention;

[0040] Figure 2 is an experimental picture obtained by splicing a remote sensing image and a ground image according to an embodiment of the present invention;

[0041] Figure 3 is an example diagram of positive samples and negative samples in an embodiment of the present invention;

[0042] Figure 4 is a flow chart of an eye movement experiment according to an embodiment of the present invention;

[0043] Figure 5 is a stimulus picture and a corresponding human visual attention map according to an embodiment of the present invention;

[0044] Figure 6is a schematic diagram of the structure of a hybrid attention module according to an embodiment of the present invention;

[0045] Figure 7 It is a visualization example of the attention mechanism of an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings of specific embodiments of the present invention to clearly and completely describe the exemplary scheme of the embodiment of the present invention. Unless otherwise defined, the technical terms or scientific terms used in the present invention should be the common meanings understood by people with ordinary skills in the field.

[0047] At present, the cross-view image matching model based on deep learning still faces challenges in overcoming the semantic barriers caused by perspective differences, which makes it easy to ignore the core features of the task during the model learning process. When the human brain processes visual tasks, it can flexibly respond to changes in perspective, understand and accurately focus on the features of the same location under different perspectives. This embodiment introduces the eye movement data of the human brain during the cross-view matching task, and proposes a cross-view image matching method that integrates the eye movement attention mechanism for cross-view image matching. Figure 1 As shown, the method comprises:

[0048] Step S101: Use the existing ground-remote sensing cross-view image dataset to train two independent Transformer encoder branches T s ,T a , generate self-attention maps for ground maps and remote sensing images respectively, that is, the first stage; export and save the self-attention map of the training set in the last training round.

[0049] The two independent Transformer encoder branches both use the Vision Transformer model. Each Transformer encoder branch contains a feature embedding layer and a Transformer encoder. For the feature embedding layer, first use Patch Embedding to transform the input image I∈R H×W×C (H, W, C are the height, width and number of channels of the image respectively) converted into Patch Tokens (patch labels) I t ∈R N×D (D is the feature dimension, N is the number of patches.) These patch tokens are then concatenated with the preset class tokens and distillation tokens to form the token set required for subsequent processing. t ′∈R (N+2)×DThis combination not only retains the global information of the image (through Class Token), but also incorporates local details (through Patch Tokens) and possible distillation knowledge (through Distillation Token), providing a rich source of information for the model. In order to improve the model's sensitivity to the spatial position relationship of elements in the image, each branch introduces a learnable position encoding, which is obtained through a learnable matrix R (N +2)×D It is embedded into the network, providing location information for each Patch Token, so that the model can recognize and utilize the spatial layout in the image. Finally, the Transfomer encoder is used to encode the features of the entire branch. The core of the Transfomer encoder is the Multi-Head Attention module, which can calculate attention on multiple subspaces in parallel to capture the complex interactions between image features. The Transfomer encoder can maintain scale consistency between input and output, ensuring lossless transmission and efficient processing of information.

[0050] Step S102: designing an experimental paradigm for cross-view image matching, collecting eye movement trajectory data of humans when performing a cross-view image matching task, and generating a human visual attention map.

[0051] (1) Eye movement data collection

[0052] The eye tracking device used in this embodiment is Tobii Pro Spectrum, the stimulus playback software is TobiiPro Lab, and the sampling rate is 300Hz. The ground map and the remote sensing map are spliced ​​into a single visual stimulus and presented directly to the subjects to capture their eye movement trajectory data during the cross-view image matching task.

[0053] 3000 groups of positive samples and 1000 groups of negative samples were randomly selected from the ground-remote sensing cross-view image dataset (such as CVUSA, CVACT) as the eye movement experiment sample library to form the eye movement experiment database. The width of the remote sensing image of each sample is consistent with the width of the ground image. The remote sensing image is presented to the subjects in a splicing manner with the ground image at the bottom and the remote sensing image at the top. Figure 2 shown.

[0054] Positive samples and negative samples: define each ground map to be matched and its corresponding remote sensing image describing the same area in the remote sensing image library as a positive sample, such as Figure 3 (a), any remote sensing image that does not match any other remote sensing image in the remote sensing image library constitutes a negative sample, such as Figure 3(b) As can be seen from the example figure, due to the complexity of remote sensing image scenes, there are many image targets with significant features, while there is relatively little usable information in ground images. If we cannot focus on key image features during the matching process, a large amount of invalid information will cause a decrease in image matching performance.

[0055] The eye movement experiment process is as follows Figure 4 As shown. Each experimental unit consists of 10 modules. There is a 2-minute break between any two adjacent modules. There is a 30-minute break between every 10 modules. Each module contains 20 random test tasks. Each experimental unit is expected to take about 60 minutes. In order to ensure the quality and reliability of the data, each subject collects data from 4 experimental units.

[0056] During the experiment, the subjects were asked to carefully judge whether the street view image and the remote sensing image in the mosaic image displayed on the screen depicted the same location. If the judgment was yes, they pressed the number key "1"; if not, they pressed the number key "2".

[0057] (2) Data preprocessing

[0058] The eye movement data is filtered. In order to intuitively show the human eye movement attention allocation pattern in the cross-view image matching task, the human visual attention map is generated using the public Python tool library pyHeatMap. Figure 2 The layout of the spliced ​​image in is used to accurately segment the generated human visual attention map, and the attention distribution information is mapped to the street view image and the remote sensing image respectively, generating their own independent human visual attention maps, such as Figure 5 As shown. Figure 5 In the figure, the red boxes outline the original image pairs input to the second-stage model, namely the street view and remote sensing images without any processing; and the green boxes show the human visual attention maps corresponding to these original images, which clearly reveal the subjects' gaze focus and attention distribution patterns when performing the matching task.

[0059] Step S103: remote sensing human visual attention map I g The remote sensing self-attention map I generated by the trained remote sensing Transformer encoder n The hybrid attention map is obtained by performing image-level fusion through the hybrid attention module.

[0060] Both the eye movement attention mechanism and the self-attention mechanism in the Transformer encoder use the scaled dot product attention function as the core calculation formula, as shown in formula (1). Here, Q (queries), K (keys), and V (values) are the input features obtained by linear transformation, dk It represents the dimension of the input feature.

[0061]

[0062] The structure of the hybrid attention module designed in this embodiment is as follows Figure 6 As shown in Figure 1, this module mixes eye movement and self-attention at the image level, aiming to improve the model's ability to identify and focus on the most important and relevant areas in the image. g As a direct reflection of human visual attention, it is used if it exists; if it does not exist (such as in some automated processing or preliminary training stages), the remote sensing self-attention map I of the last training round of the first stage is used g ' As an alternative, the specific rules are as follows:

[0063]

[0064] In order to further combine I g (or I g ′) and the model’s own remote sensing self-attention map I n , introduce formula (3) to calculate the mixed attention map a(I g ,I n ) to capture the correlation information between the two.

[0065]

[0066] in, is the query vector of the human visual attention map for remote sensing, is the key value vector of the remote sensing self-attention map, is the value vector of the remotely sensed human visual attention map.

[0067] The calculated hybrid attention information is updated by formula (4), and Layer Normalization is applied to ensure the stability of the data. The updated hybrid attention map is expressed as:

[0068] u(I g ,I n )=LayerNorm(I g +a(I g ,I n ))(4)

[0069] Finally, by integrating formulas (2), (3), and (4), we obtain formula (5), which fully describes how to flexibly calculate and update the final hybrid attention map u(I g ,I n ).

[0070]

[0071] Step S104: The original remote sensing image and the mixed attention map are subjected to image adaptive cropping and resolution enhancement modules to obtain a high-resolution remote sensing image.

[0072] In the work of Zhu et al., they used the network attention mechanism to guide the image to be non-uniformly segmented, which improved the model's ability to focus on key areas and effectively reduced the errors caused by information redundancy. However, this method relies on a pre-set cropping rate, which may cause some critical image blocks to be mistakenly removed, thereby limiting the further improvement of model performance. To this end, this case proposed an image adaptive cropping and resolution enhancement module, and the core workflow is as follows:

[0073] First, for the mixed attention map and remote sensing images Grayscale, and I h and I a Divide the image into a series of 16×16 blocks and Then use formula (6) to calculate I h and The average pixel value and

[0074]

[0075] Where W and H are the height and width of the image block, respectively, x,y is the pixel value.

[0076] Then, traverse the image blocks of the mixed attention map, if there is an average pixel value of the image block Smaller than the average pixel value of the mixed attention map Then the corresponding image block in the remote sensing image If it does not exist, then This step ensures that the key information areas are fully preserved while removing unimportant background information.

[0077] Finally, in order to further improve the model's attention to key areas, the cropped image resolution is increased from 256×256 to 320×320. This adjustment not only enlarges the important relevant areas in the image, but also helps the model capture more details, thereby further improving the model's geolocation performance. The details of the image adaptive cropping and resolution enhancement module are shown in Algorithm 1.

[0078]

[0079] Step S105: input the high-resolution remote sensing image and the ground image into two pre-trained independent Transformer encoder branches for secondary training.

[0080] At each stage, the model uses weighted soft-margin triplet loss as the objective function, which is defined as:

[0081]

[0082] Where α is a hyperparameter used to adjust the sensitivity of the distance difference between positive and negative samples. pos With d neg They represent the Euclidean distance between the features of the positive sample and the negative sample images, respectively. The specific calculation method is as shown in formulas (8) and (9):

[0083]

[0084] f in formula (8) and (9) i A 、f i P and f i N They refer to the feature vectors of the anchor point (usually a street view image), the positive example (the remote sensing image corresponding to the anchor point), and the negative example (other non-corresponding remote sensing images). 2 Refers to the 2-norm, which is used to calculate the Euclidean distance between two points. During the training process with a batch size of N, a comprehensive sampling strategy is adopted, that is, all possible N×(N-1) triplets are enumerated and their losses are calculated. This strategy ensures that the model can fully learn the subtle differences between different positive and negative samples. Finally, in order to improve the stability and convergence of training, the calculated loss values ​​are usually normalized. This design enables the model to more accurately distinguish which remote sensing images truly reflect the same geographical area as the ground image, thereby improving the accuracy of cross-view image matching.

[0085] Corresponding to the above method, this embodiment also proposes a cross-view image matching system integrating an eye movement attention mechanism, including a pre-training module, a human visual attention map generation module, a hybrid attention map generation module, a high-resolution remote sensing image module and a secondary training module, wherein:

[0086] The pre-training module is used to train two independent Transfomer encoder branches using the existing ground-remote sensing cross-view image dataset to generate self-attention maps for ground images and remote sensing images respectively.

[0087] The human visual attention map generation module is used to design the experimental paradigm of cross-view image matching, collect eye movement trajectory data of humans when performing cross-view image matching tasks, and generate human visual attention maps.

[0088] The hybrid attention map generation module is used to obtain a hybrid attention map by performing image-level fusion of the human visual attention map corresponding to the remote sensing image and the remote sensing self-attention map generated by the trained remote sensing Transfomer encoder through the hybrid attention module.

[0089] The high-resolution remote sensing image module is used to obtain a high-resolution remote sensing image by combining the original remote sensing image and the mixed attention map with the image adaptive cropping and resolution enhancement module.

[0090] The secondary training module is used to input high-resolution remote sensing images and ground images into two pre-trained independent Transfomer encoder branches for secondary training.

[0091] In order to verify the effectiveness of this solution, further explanation is given below in combination with experimental data.

[0092] 1. Datasets and evaluation metrics

[0093] 1) CVACT dataset: Experiments were conducted on the benchmark dataset CVACT to verify the effectiveness of the proposed model. The dataset contains 35,532 sets of image pairs for training, 8,884 sets of image pairs for verification (CVACT_val), and 92,802 sets of image pairs for testing (CVACT_test). The performance of the model on CVACT_val is regarded as an important indicator to measure its cross-view image matching ability, thus ensuring the fairness and wide recognition of the evaluation results.

[0094] 2) Evaluation indicators: The "Top-k" recall rate indicator (denoted as "R@k") is used to comprehensively report the performance of the model. This indicator intuitively reflects the probability of successfully matching the real corresponding image in the top k image sequences of the retrieval results. Specifically, for each ground image, the closest k remote sensing images are retrieved in its embedding space. If the real matching image corresponding to the ground image is among the top k in these retrieval results, it is considered a successful retrieval, thereby accurately measuring the accuracy and efficiency of the model in the cross-modal image matching task.

[0095] (II) Model implementation details

[0096] The model of the present invention is deployed in Pytorch, the optimizer uses SAM, and the experimental parameters of each stage are shown in Table 1.

[0097] Table 1 Experimental parameter settings

[0098]

[0099] (III) Comparison with other methods

[0100] 1) Image matching in the same area

[0101] In real life, we sometimes have the following requirements: how to accurately locate a specific street view image in a certain area based on a wide remote sensing image library of a certain area. In order to simulate this process, the benchmark dataset CVACT was selected as the experimental platform. The dataset covers a vast area of ​​about 300 square kilometers in Canberra, providing rich and real image resources for the experiment. The model of the present invention was trained on the CVACT dataset, and its performance was evaluated by the independent validation set CVACT_val. Table 2 lists the experimental results in detail and compares them intuitively with other top works in the current field. Table 2 shows that the method of the present invention achieves the best performance in R@k (i.e., the proportion of the target image included in the first k results returned), especially R@1 reaches a significant level of 75.76%, and the method of the present invention achieves a performance improvement of 1.2% compared with the previous best model TransGeo. Therefore, it is believed that the introduction of the eye movement attention mechanism enhances the model's ability to capture and extract key features across different perspectives, so that the model can more accurately lock the target position when facing complex and changeable street view images. The verification of the experimental results in the same area not only confirmed the technological advancement of the model of the present invention, but also further demonstrated its great potential and value in practical applications.

[0102] Table 2 Experimental results on the benchmark dataset CVACT

[0103] Model R@1 R@5 R@10 R@1％ CVM-Net 20.15 45.00 56.87 87.57 VGG global pooling 28.98 57.04 67.96 91.72 Hydrid-Net 36.89 - - 87.81 Instance Loss USAM 40.53 - - 89.12 Orientation 46.96 68.28 75.48 92.01 CVFT 61.05 81.33 86.52 95.93 Rodrigues 73.19 90.39 93.38 97.45 TransGeo 74.36 90.67 93.73 98.26 Ours 75.56 93.77 96.28 98.85

[0104] 2) Cross-region image matching

[0105] In order to verify the location inference ability of the model under extreme challenges, a cross-regional and cross-view image matching experiment was designed. In this scenario, the model needs to accurately locate the target street view image in a completely unfamiliar region, CVUSA (an image set covering a wide area of ​​the United States), relying only on the training experience of CVACT. In Table 3, the experimental results are systematically presented, and the data clearly reveal the remarkable performance of the model in the cross-regional image matching task. The significant improvement of the R@k value (i.e., the probability that the first k retrieval results contain the correct match) of the model of the present invention demonstrates its strong generalization ability. Compared with the advanced L2LTR method, the model of the present invention achieved an increase of 11.05% in R@1, an indicator that directly reflects the accuracy of the first retrieval. Therefore, it is believed that incorporating the eye movement attention mechanism into the model design not only enhances the model's ability to deeply understand the image content, but also gives it a more flexible and changeable perspective adaptation ability, so that it can show higher generalization ability and robustness when facing complex and changeable cross-view image matching tasks.

[0106] Table 3 Experimental results on cross-datasets

[0107] Model R@1 R@5 R@10 R@1％ Orientation 18.47 34.46 42.28 69.01 TransGeo 18.99 38.24 46.91 88.94 SAFA 21.45 36.55 43.79 69.83 GeoDTR 29.85 49.25 57.11 82.47 L2LTR 33.00 51.87 60.63 84.79 Ours 44.05 69.10 78.61 96.13

[0108] (IV) Ablation experiment

[0109] Hybrid attention module: Table 4 shows the effect of the hybrid attention module on the performance improvement of the model on CVACT. Specifically, the application of this module has achieved a positive effect on the evaluation indicators of the model: R@1 increased by 1.01%, R@5 increased by 0.6%, and R@10 increased by 0.29%. Even under the broader R@1% evaluation standard, an increase of 0.01% was achieved. This series of data shows that the hybrid attention module has made a substantial contribution to optimizing model performance and enhancing its performance in complex visual tasks, further verifying its effectiveness in improving recognition accuracy and matching efficiency.

[0110] Table 4 Ablation experimental results of hybrid attention module on CVACT

[0111]

[0112] Image Adaptive Cropping and Resolution Enhancement Module: In Table 5, the positive effect of the Image Adaptive Cropping and Resolution Enhancement module on the model performance on CVACT is shown. The implementation of this module enhances the performance of the model, with the R@1 indicator improving by up to 5.04%, the R@5 indicator rising by 2.30%, and the R@10 indicator also achieving a 1.38% increase. Even under broader accuracy requirements, such as the R@1% evaluation, it also showed a positive improvement of 0.04%. These data not only clearly prove the key role of the Image Adaptive Cropping and Resolution Enhancement module in optimizing model performance, but also further emphasize its unique advantages in improving matching accuracy and efficiency.

[0113] Table 5 Ablation experiment results of image adaptive cropping and resolution enhancement module on CVACT

[0114]

[0115] (V) Analysis of attention mechanism

[0116] In order to prove that the introduction of eye movement attention alleviates the "shortcut learning" phenomenon, Grad-CAM is used to visualize the model's attention on CVACT_val. By comparing four examples, it shows the significant changes in the model's attention distribution on the image before and after the introduction of eye movement information, such as Figure 7 As shown. Figure 7 In the figure, we can clearly observe the positive effect that eye movement information brings to the model. For each set of image pairs, the introduction of eye movement data prompts the model to focus more on the core elements in the image, such as recognizable elements such as street trees, house outlines, and traffic signs in street view and remote sensing images, rather than disturbing backgrounds such as distorted roads and vast skies. Specifically, in street view images, the model's focus is pulled away from distracting or worthless roadside clutter or backgrounds, and instead locked on relatively stable static features such as street trees and forests; while in remote sensing images, the guidance of eye movement information prompts the model to shift its attention from the non-critical area in the lower half of the image to the upper half. The information in this area is highly consistent with the main information of the street view image, further verifying the effectiveness of eye movement guidance.

[0117] In summary, the present invention proposes a cross-view image matching method that integrates the eye movement attention mechanism, which alleviates the "shortcut learning" problem of the deep learning cross-view image matching model in a novel way. The network guides the model's attention to focus on the key areas of the image by constructing a hybrid module of eye movement and self-attention. In addition, the image adaptive cropping and resolution enhancement module is used to simulate the "attention and magnification" mechanism of human visual focusing behavior, eliminate redundant information in the image, and further guide the model to focus on the key areas of the image. Experiments have shown that the introduction of eye movement information plays a positive role in the model's same-region and cross-region image matching, with R@1 reaching 75.76% and 44.05% respectively, and the effectiveness of the hybrid attention module, image adaptive cropping and resolution enhancement module are verified through ablation experiments.

[0118] Unless otherwise specifically stated, the relative steps, numerical expressions and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present invention.

[0119] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0120] The units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person of ordinary skill in the art may use different methods to implement the described functions for each specific application, but such implementation is not considered to be beyond the scope of the present invention.

[0121] Those skilled in the art will appreciate that all or part of the steps in the above method can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk. Optionally, all or part of the steps in the above embodiment can also be implemented using one or more integrated circuits, and accordingly, each module / unit in the above embodiment can be implemented in the form of hardware or in the form of software function modules. The present invention is not limited to any specific form of combination of hardware and software.

[0122] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A cross-view image matching method integrating eye movement attention mechanism, characterized in that: Include: Using the existing ground-remote sensing cross-view image dataset, two independent Transfomer encoder branches are trained to generate self-attention maps for ground images and remote sensing images respectively; Design an experimental paradigm for cross-view image matching, collect eye movement data of humans when performing cross-view image matching tasks, and generate human visual attention maps; The human visual attention map corresponding to the remote sensing image and the remote sensing self-attention map generated by the trained remote sensing Transfomer encoder are fused at the image level through the hybrid attention module to obtain a hybrid attention map; The original remote sensing image and the mixed attention map are used to obtain a high-resolution remote sensing image through image adaptive cropping and resolution enhancement modules; The high-resolution remote sensing images and ground images are input into two pre-trained independent Transformer encoder branches for secondary training.

2. The cross-view image matching method integrating eye movement attention mechanism according to claim 1, characterized in that: Two independent Transformer encoder branches generate self-attention maps for ground maps and remote sensing images respectively, including: Both independent Transformer encoder branches use the VisionTransformer model, which first uses a feature embedding layer to convert ground images and remote sensing images into low-dimensional vectors, which are then sent to the Transformer encoder for feature encoding. Finally, a multi-layer perceptron head is used to perform cross-view image matching on the output features of the two branches. The self-attention map of the training set at the last training round is exported and saved.

3. The cross-view image matching method integrating eye movement attention mechanism according to claim 1, characterized in that: The two independent Transformer encoder branches are trained using a weighted soft-margin triplet loss as the objective function.

4. The cross-view image matching method integrating eye movement attention mechanism according to claim 1, characterized in that: Design an experimental paradigm for cross-view image matching, and the process of generating human visual attention maps is as follows: A certain number of remote sensing-ground image pairs are randomly selected from the ground-remote sensing cross-view image dataset; The extracted remote sensing-ground image pairs are preprocessed to make the width of the remote sensing image consistent with that of the ground image, and then spliced ​​with the remote sensing image at the bottom and the ground image at the top as the stimulus image for the eye movement experiment; The stitched images are presented to the subjects one by one using a desktop eye tracker, and the subjects are asked to judge whether the aerial perspective content and the ground perspective content contained in the image are the same location. At the same time, the subjects' eye movement trajectories are collected, and the collected eye movement data are used to generate a human visual attention map, including a human visual attention map corresponding to the remote sensing image and a human visual attention map corresponding to the ground image.

5. The cross-view image matching method integrating eye movement attention mechanism according to claim 1, characterized in that: The remote sensing human visual attention map I g Remote Sensing Self-Attention Map I n The hybrid attention map a(I g ,I n ) is expressed as: in, is the query vector of the human visual attention map for remote sensing, is the key value vector of the remote sensing self-attention map, is the value vector of the remotely sensed human visual attention map.

6. The cross-view image matching method integrating eye movement attention mechanism according to claim 5, characterized in that: The calculated mixed attention information is updated by the following formula, and the updated mixed attention graph is expressed as: u(I g ,I n )=LayerNorm(I g +a(I g ,I n )) Among them, LayerNorm represents layer normalization.

7. The cross-view image matching method integrating eye movement attention mechanism according to claim 1, characterized in that: The working process of the image adaptive cropping and resolution enhancement module is as follows: First, the original remote sensing image and the hybrid attention map are grayed out and divided into a series of image blocks of equal size. Then, the average pixel value of the hybrid attention map and the average pixel value of each image block of the hybrid attention map are calculated. Then, the image blocks of the hybrid attention map are traversed. If there is an image block whose average pixel value is less than the average pixel value of the hybrid attention map, the pixel values ​​of the corresponding image block in the remote sensing image are all set to zero; if there is no image block, the image block of the remote sensing image remains unchanged, realizing adaptive image cropping; Finally, the cropped image resolution is enhanced.

8. A cross-view image matching system integrating eye movement attention mechanism, characterized in that: It includes a pre-training module, a human visual attention map generation module, a mixed attention map generation module, a high-resolution remote sensing image module and a secondary training module, among which: A pre-training module is used to train two independent Transfomer encoder branches using the existing ground-remote sensing cross-view image dataset to generate self-attention maps for ground images and remote sensing images respectively; The human visual attention map generation module is used to design the experimental paradigm of cross-view image matching, collect the eye movement trajectory data of humans when performing cross-view image matching tasks, and generate the human visual attention map; A hybrid attention map generation module is used to fuse the human visual attention map corresponding to the remote sensing image with the remote sensing self-attention map generated by the trained remote sensing Transfomer encoder through the hybrid attention module at the image level to obtain a hybrid attention map; High-resolution remote sensing image module, used to obtain high-resolution remote sensing images through image adaptive cropping and resolution enhancement modules for the original remote sensing images and mixed attention maps; The secondary training module is used to input high-resolution remote sensing images and ground images into two pre-trained independent Transfomer encoder branches for secondary training.

9. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images