Attention mechanism based cross-view geolocation method, medium and device

Abstract

The application discloses a cross-view geographical positioning method based on an attention mechanism, a medium and equipment, and the method comprises the following steps: receiving image information taken by a UAV or a satellite; inputting the image information into a trained neural network model, and outputting geographical position information corresponding to the image information. When training the neural network model, the application significantly improves the robustness of attention to target buildings or important areas in the image shift and rotation transformation in the geographical positioning task through multi-dimensional feature fusion, deep convolution optimization and cross-view feature alignment, so that the positioning difficulty in complex scenes such as view angle change and geometric shape difference is overcome.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and more specifically to a cross-perspective geolocation method, medium, and device based on an attention mechanism. Background Technology

[0002] In the fields of computer vision and pattern recognition, cross-view image matching is a task of significant research importance, playing a crucial role in various practical applications such as UAV image analysis, satellite remote sensing image processing, and scene reconstruction. However, due to the significant geometric deformations and illumination differences between cross-view images, extracting robust and discriminative features from them has become a challenge.

[0003] Currently, attention mechanisms have become an important tool for improving model performance. By simulating the selective attention mechanism of the human visual system, they can enhance the expressive power of salient regions in feature maps. However, existing attention mechanisms such as SENet, CBAM, and SKNet have certain limitations when handling cross-view tasks. For example, SENet only focuses on channel dimension information, ignoring the importance of spatial attention; although CBAM combines channel and spatial attention, it processes them separately and fails to achieve the fusion of cross-dimensional features; although SKNet uses multi-scale convolution to capture some directional features, it fails to fully integrate channel and spatial information. Summary of the Invention

[0004] In view of the above problems, the present invention provides a cross-view geolocation technical solution based on an attention mechanism to solve the problem that existing cross-view matching methods cannot accurately focus on target buildings in images from different viewpoints, resulting in a decrease in the matching accuracy of target buildings and thus affecting the accuracy of building location.

[0005] To achieve the above objectives, in a first aspect, this application provides a cross-view geolocation method based on an attention mechanism, comprising the following steps:

[0006] Receive image information captured by drones or satellites;

[0007] The image information is input into the trained neural network model, which outputs the geographic location information corresponding to the image information.

[0008] The neural network model is trained by including the following steps:

[0009] S1: Receive a first sample image acquired from a satellite perspective or a second sample image acquired from a drone perspective, and extract feature maps from the first sample image or the second sample image using a pre-trained ConvNeXt network;

[0010] S2: Divide the extracted feature map into a first branch tensor, a second branch tensor, and a third branch tensor. Perform ECA operation on the first branch tensor, the second branch tensor, and the third branch tensor respectively, and then perform global average pooling to obtain the first channel description vector, the second channel description vector, and the third channel description vector.

[0011] S3: The first branch tensor is weighted using the first channel description vector, the second branch tensor is weighted using the second channel description vector, and the third branch tensor is weighted using the third channel description vector to obtain the first attention feature map corresponding to the three branches. The three first attention feature maps are averaged to output the second attention feature map.

[0012] S4: Perform a depthwise convolution operation on the second attention feature map to obtain the enhanced third attention feature map;

[0013] S5: Determine geographical location information based on the third attention feature map.

[0014] Optionally, the feature map extracted from the first sample image or the second sample image using a pre-trained ConvNeXt network is represented by the following formula:

[0015]

[0016] Where i∈{1,2}, x1 represents the first sample image, x2 represents the second sample image, and L i ∈

[0017] R C×H×W This represents the extracted feature map, where C is the number of channels, and H and W represent the height and width of the image, respectively.

[0018] Furthermore, the feature map has a shape of C×H×W, where C is the number of channels, and H and W represent the height and width of the image, respectively.

[0019] The first branch tensor is represented as W×H×C, the second branch tensor is represented as H×C×W, and the third branch tensor retains the original shape of the feature map and is represented as C×H×W;

[0020] Steps S2 and S3 are expressed by the following formulas:

[0021]

[0022] Among them, L D Let X represent the input tensors of the first branch, the second branch, and the third branch. DX represents the first attention feature map of the output, X represents the second attention feature map, and the symbol σ represents the Sigmoid activation function.

[0023] Furthermore, step S4 includes:

[0024] Depth convolution operations are performed using convolution kernels of different lengths, and information exchange between channels is achieved through 1×1 convolutional layers to obtain an enhanced third attention feature map.

[0025] Step S4 is expressed by the following formula:

[0026]

[0027] Where DwConv represents a 5×5 depthwise convolution, Pathi is the result of two depthwise strip convolutions and the ontology after the 5×5 depthwise convolution, and Y is the enhanced third attention feature map.

[0028] Furthermore, the training of the neural network model also includes the following steps:

[0029] When calculating the KL divergence, the predicted probability of the positive class is ignored, and the softmax function is reapplied to calculate the predicted probability of the negative class. Specifically, this includes:

[0030] The third attention feature map is passed to the classifier after average pooling. The classifier obtains the logit score for each class, and the logit scores of each class are passed to the Log-Softmax layer to obtain the logarithm of the predicted probability for each class, which is expressed by the following formula:

[0031] L=Log_Softmax(Classifier(AvgPool(Y)));

[0032] Where L={l1,l2,...,l p ,...,l N}∈R 1×N This represents the logarithm of the predicted probability for each category, where N represents the total number of categories, and the subscript p indicates the positive class.

[0033] To calculate the negative class KL divergence loss, lp is ignored, therefore L is rewritten as:

[0034]

[0035] Next, L^ is passed through a Softmax layer to obtain the predicted probability distribution, as shown below:

[0036]

[0037] Where P represents the predicted probability distribution of all negative classes.

[0038] Furthermore, the method also includes:

[0039] The output results of the first sample image acquired from the satellite perspective and the second sample image acquired from the UAV perspective are used to learn each other's predicted probability distributions, thereby aligning the depth feature distributions of the two.

[0040] The calculation process for the negative class KL divergence is expressed by the following formula:

[0041]

[0042] Where P1 and P2 represent the negative class prediction probability distributions of the first sample image and the second sample image, respectively.

[0043] Furthermore, the method includes:

[0044] The final loss function of the neural network model is obtained by comprehensively calculating the KL divergence loss, contrast loss, and cross-entropy loss.

[0045] Furthermore, the contrast loss is expressed as follows:

[0046]

[0047] Where q is the feature vector of the query image, R is the set of reference images, and r + These are the positive samples corresponding to the query image, and τ is a temperature parameter used to control the scaling of the similarity score.

[0048] The cross-entropy loss is expressed as follows:

[0049]

[0050] in, It is the predicted probability of the neural network model, and lp(y) is the logarithm of the predicted probability of the positive class;

[0051] The formula for calculating the final loss function of the neural network model is as follows:

[0052] Loss = L infoNCE +λ1L CE +λ2L NCKL ;

[0053] Wherein, λ1 and λ2 are adjustable hyperparameters.

[0054] In a second aspect, the present invention also provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the method described in the first aspect.

[0055] In a third aspect, the present invention also provides an electronic device including a memory and a processor, the memory being used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method described in the first aspect.

[0056] Unlike existing technologies, the above solution provides a cross-view geolocation method, medium, and device based on an attention mechanism. The method includes: receiving image information captured by a drone or satellite; inputting the image information into a trained neural network model; and outputting the geographic location information corresponding to the image information. When training the neural network model, this application significantly improves the robustness of attention to target buildings or important areas in geolocation tasks under image shifts and rotations through multi-dimensional feature fusion, deep convolution optimization, and cross-view feature alignment, thereby overcoming the positioning difficulties in complex scenarios such as viewpoint changes and geometric differences.

[0057] The above description of the invention is merely an overview of the technical solution of the present invention. In order to enable those skilled in the art to better understand the technical solution of the present invention and to implement it based on the description and drawings, and to make the above-mentioned objectives and other objectives, features and advantages of the present invention easier to understand, the following description is provided in conjunction with the specific embodiments and drawings of the present invention. Attached Figure Description

[0058] The accompanying drawings are only used to illustrate the principles, implementation methods, applications, features, and effects of specific embodiments of the present invention and other related contents, and should not be considered as limitations on the present invention.

[0059] In the accompanying drawings of the instruction manual:

[0060] Figure 1 A flowchart illustrating a cross-view geolocation method based on an attention mechanism, as described in a first exemplary embodiment of the present invention.

[0061] Figure 2 This is a flowchart illustrating the neural network model training process according to a second exemplary embodiment of the present invention;

[0062] Figure 3 This is a framework diagram of a cross-view geolocation method based on an attention mechanism according to a specific embodiment of the present invention.

[0063] Figure 4 This is a schematic diagram illustrating the principle of the attention mechanism involved in a specific embodiment of the present invention;

[0064] Figure 5 A schematic diagram of a visualization model heatmap generated by existing cross-view matching methods;

[0065] Figure 6 This is a schematic diagram of a visualization model heatmap generated by a cross-view geolocation method based on an attention mechanism according to a specific embodiment of the present invention.

[0066] Figure 7 This is a comparison chart of the output results of the method according to a specific embodiment of the present invention on the University-1652 dataset and the method according to the prior art.

[0067] Figure 8 This is a comparison chart of the output results of the method according to a specific embodiment of the present invention on the SUES-200 dataset and the method according to the prior art.

[0068] Figure 9 This is a comparison diagram showing the processing results of the attention mechanism involved in a specific embodiment of the present invention and the attention mechanism involved in the prior art.

[0069] Figure 10 This is a comparison chart showing the processing results of the loss function involved in a specific embodiment of the present invention and the loss function involved in the prior art;

[0070] Figure 11 This is a schematic diagram of an electronic device according to a specific embodiment of the present invention.

[0071] The reference numerals used in the above figures are explained as follows:

[0072] 10. Electronic devices;

[0073] 101. Processor;

[0074] 102. Storage medium. Detailed Implementation

[0075] To illustrate the possible application scenarios, technical principles, implementable specific solutions, and achievable objectives and effects of this invention in detail, the following description, in conjunction with the listed specific embodiments and accompanying drawings, provides a detailed explanation. The embodiments described herein are merely illustrative of the technical solutions of this invention and are therefore intended only as examples, not as limiting the scope of protection of this invention.

[0076] In this document, the term "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The term "embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment, nor does it specifically limit its independence or connection with other embodiments. In principle, in this invention, as long as there are no technical contradictions or conflicts, the technical features mentioned in each embodiment can be combined in any way to form corresponding implementable technical solutions.

[0077] Unless otherwise defined, the technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains; the use of related terms herein is merely for the purpose of describing particular embodiments and is not intended to limit the invention.

[0078] In the description of this invention, the term "and / or" is used to describe the logical relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A exists, B exists, and A and B exist simultaneously. Additionally, the character " / " generally indicates that the preceding and following objects have an "or" logical relationship.

[0079] In this invention, terms such as “first” and “second” are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any actual quantity, hierarchy, or order between these entities or operations.

[0080] Without further limitations, the use of terms such as “comprising,” “including,” “having,” or other similar open-ended expressions in this invention is intended to cover non-exclusive inclusion, which does not exclude the presence of additional elements in a process, method, or product that includes the stated elements, such that a process, method, or product that includes a list of elements may include not only those defined elements but also other elements not expressly listed, or elements inherent to such a process, method, or product.

[0081] In this invention, expressions such as "greater than", "less than", and "exceeding" are understood to exclude the stated number; expressions such as "above", "below", and "within" are understood to include the stated number. Furthermore, in the description of the embodiments of this invention, "multiple" means two or more (including two), and similar expressions related to "multiple" are also understood in this way, such as "multiple groups" and "multiple times", unless otherwise explicitly specified.

[0082] In the first aspect, such as Figure 1 As shown, this application provides a cross-view geolocation method based on an attention mechanism, including the following steps:

[0083] First, proceed to step S101: Receive image information captured by a drone or satellite;

[0084] Then proceed to step S102: input the image information into the trained neural network model, and output the geographical location information corresponding to the image information.

[0085] In this embodiment, the image information is image information containing the target building. By accurately matching the image information, the specific location of the target building can be accurately marked.

[0086] like Figure 2 As shown, the neural network model includes the following steps during training:

[0087] S1: Receive a first sample image acquired from a satellite perspective or a second sample image acquired from a drone perspective, and extract feature maps from the first sample image or the second sample image using a pre-trained ConvNeXt network;

[0088] S2: Divide the extracted feature map into a first branch tensor, a second branch tensor, and a third branch tensor. Perform ECA operation on the first branch tensor, the second branch tensor, and the third branch tensor respectively, and then perform global average pooling to obtain the first channel description vector, the second channel description vector, and the third channel description vector.

[0089] S3: The first branch tensor is weighted using the first channel description vector, the second branch tensor is weighted using the second channel description vector, and the third branch tensor is weighted using the third channel description vector to obtain the first attention feature map corresponding to the three branches. The three first attention feature maps are averaged to output the second attention feature map.

[0090] S4: Perform a depthwise convolution operation on the second attention feature map to obtain the enhanced third attention feature map;

[0091] S5: Determine geographical location information based on the third attention feature map.

[0092] In some embodiments, this invention employs a ConvNeXt network pre-trained on ImageNet as a feature extraction module to extract features from images captured from both satellite and drone perspectives. Considering the similar pattern structures of satellite and drone images, the feature extraction is implemented by sharing weights. Specifically, the neural network model framework of this application is as follows: Figure 3 As shown, the feature map extracted from the first sample image or the second sample image using a pre-trained ConvNeXt network is represented by the following formula:

[0093]

[0094] Where i∈{1,2}, x1 represents the first sample image, x2 represents the second sample image, and L i ∈

[0095] RC×H×W This represents the extracted feature map, where C is the number of channels, and H and W represent the height and width of the image, respectively.

[0096] To overcome the challenges posed by geometrical variations between images, this application proposes an innovative Shape Adaptive and Importance-Aware Attention Module (SSA). This module comprises two key sub-modules: a cross-dimensional feature fusion (CFF) module and a directed guided deep convolution (DDC) module. These modules enhance the fusion and alignment capabilities of cross-viewpoint features through operations in different dimensions. An illustrative diagram of the Shape Adaptive and Importance-Aware Attention Module is shown below. Figure 4 As shown.

[0097] The main function of the CFF module is to perform cross-dimensional operations on the input feature map to achieve effective feature fusion. In some embodiments, the feature map has a shape of C×H×W, where C is the number of channels, and H and W represent the height and width of the image, respectively.

[0098] The first branch tensor is represented as W×H×C, the second branch tensor is represented as H×C×W, and the third branch tensor retains the original shape of the feature map and is represented as C×H×W;

[0099] Each tensor is processed using ECA (Efficient Channel Attention), followed by global average pooling to obtain three channel description vectors: a first channel description vector, a second channel description vector, and a third channel description vector. Then, the first branch tensor is weighted using the first channel description vector, the second branch tensor using the second channel description vector, and the third branch tensor using the third channel description vector, resulting in a channel attention map and a spatial attention map (i.e., the first attention feature map). Finally, the three feature maps are simply averaged to obtain the output X of the CFF module. The processing of the CFF module is the same as steps S2 and S3 described above, which are expressed by the following formulas:

[0100]

[0101] Among them, L D Let X represent the input tensors of the first branch, the second branch, and the third branch. D X represents the first attention feature map of the output, X represents the second attention feature map, and the symbol σ represents the Sigmoid activation function.

[0102] The DDC module employs a 5×5 depthwise convolution operation to capture spatial feature relationships. This operation reduces computational complexity while preserving the relationships between channels. To enhance the spatial feature extraction capability of the convolution operation, the DDC module also uses convolutional kernels of different lengths and implements inter-channel information exchange through 1×1 convolutional layers. Therefore, in some embodiments, step S4 includes:

[0103] Depth convolution operations are performed using convolution kernels of different lengths, and information exchange between channels is achieved through 1×1 convolutional layers to obtain an enhanced third attention feature map.

[0104] Step S4 is expressed by the following formula:

[0105]

[0106] Where DwConv represents a 5×5 depthwise convolution, Pathi is the result of two depthwise strip convolutions and the ontology after the 5×5 depthwise convolution, and Y is the enhanced third attention feature map.

[0107] In this way, through the synergistic effect of the SSA module, the model can effectively extract and fuse deep features across dimensions from images from different perspectives, thereby achieving more accurate matching.

[0108] KL divergence measures the difference between two probability distributions. In traditional KL divergence loss, the model tends to focus on the positive class, thus ignoring the importance of the negative class. To enhance the model's attention to the negative class, this invention proposes Negative Class KL Divergence Loss (NCKL). In some embodiments, during the training of the neural network model, the following step is further included: when calculating the KL divergence, the predicted probability of the positive class is ignored, and the softmax function is reapplied to calculate the predicted probability of the negative class.

[0109] Specifically, it includes:

[0110] The third attention feature map (i.e., the output feature map of the SSA module) is passed to the classifier after average pooling. The classifier obtains the logit score for each class, and the logit scores of each class are passed to the Log-Softmax layer to obtain the logarithm of the predicted probability for each class, which is expressed by the following formula:

[0111] L=Log_Softmax(Classifier(AvgPool(Y)));

[0112] Where L={l1,l2,...,l p ,...,l N}∈R 1×NThis represents the logarithm of the predicted probability for each category, where N represents the total number of categories, and the subscript p indicates the positive class (i.e., the true class).

[0113] To calculate the negative class KL divergence loss, lp is ignored, therefore L is rewritten as:

[0114]

[0115] Next, L^ is passed through a Softmax layer to obtain the predicted probability distribution, as shown below:

[0116]

[0117] Where P represents the predicted probability distribution of all negative classes.

[0118] In some embodiments, the method further includes:

[0119] The output results of the first sample image acquired from the satellite perspective and the second sample image acquired from the UAV perspective are used to learn each other's predicted probability distributions, thereby aligning the depth feature distributions of the two.

[0120] The calculation process for the negative class KL divergence is expressed by the following formula:

[0121]

[0122] Where P1 and P2 represent the negative class prediction probability distributions of the first sample image and the second sample image, respectively.

[0123] In some embodiments, the method includes:

[0124] The final loss function of the neural network model is obtained by comprehensively calculating the KL divergence loss, contrast loss, and cross-entropy loss.

[0125] In short, to optimize feature alignment across viewpoints, this invention introduces contrastive loss (InfoNCE loss) and cross-entropy loss. Through these loss functions, the model can effectively learn the differences between positive and negative classes, further improving the accuracy of cross-viewpoint geolocation.

[0126] The contrast loss is expressed as follows:

[0127]

[0128] Where q is the feature vector of the query image, R is the set of reference images, and r + These are the positive samples corresponding to the query image, and τ is a temperature parameter used to control the scaling of the similarity score.

[0129] The cross-entropy loss is expressed as follows:

[0130]

[0131] in, It is the predicted probability of the neural network model, and lp(y) is the logarithm of the predicted probability of the positive class;

[0132] The optimization objective of this invention is to improve the accuracy of cross-view image matching by combining contrastive learning, cross-entropy loss, and negative class KL divergence loss, enabling the network to learn effective representations of deep features across viewpoints. The final loss function of the neural network model is calculated as follows:

[0133] Loss = L infoNCE +λL CE +λ2L NCKL ;

[0134] Wherein, λ1 and λ2 are adjustable hyperparameters.

[0135] The above scheme effectively improves the matching accuracy of satellite and UAV viewpoint images by comprehensively utilizing the proposed shape adaptive and importance-aware attention module (SSA) and negative class KL divergence loss (NCKL). Through multi-dimensional feature fusion, deep convolution optimization, and cross-viewpoint feature alignment, this invention significantly enhances the robustness of focusing on target buildings or important areas in geolocation tasks when subjected to image shifts and rotations.

[0136] This invention proposes a deep learning method for image matching based on satellite and UAV perspectives. By introducing a novel SSA module and NCKL loss function, it significantly improves the performance of cross-view image matching tasks. Specific technical effects are as follows:

[0137] 1. Enhanced robustness to image translation and rotation: By using the SSA module, this invention effectively achieves cross-dimensional interaction of channel and spatial features, enabling the model to better focus on the features of the target building, especially when facing image displacement and rotation, while still maintaining attention to the target building region in the image. Simultaneously, it allows the model to accurately perceive the shape of buildings in satellite and UAV view images, improving the robustness of image matching. The attention mechanism method of this application, compared with the existing MCCG method, generates model heatmaps that, during image processing, as shown below. Figure 5-6 As shown.

[0138] Figure 5 and Figure 6The top two and bottom two rows of images are selected from two scene categories in the University-1652 dataset. The first column contains the input image, while the second, third, and fourth columns contain the generated heatmaps. The heatmap in the third column was generated after shifting the input image 80 pixels to the right, while the heatmap in the fourth column was generated after rotating the input image 90 degrees clockwise. By comparing the images in the second and third columns, it can be observed that MCCG focuses on areas of inconsistency between the original input image and the shifted image, and its attention remains focused on the center of the transformed image. Furthermore, by comparing the images in the third and fourth columns, MCCG fails to focus on the target building regions in both the original and rotated images, indicating a poor ability to perceive the shape of target buildings.

[0139] 2. Improved Cross-View Image Matching Performance: Compared to existing techniques, this invention achieves significant performance improvements on multiple benchmark datasets (such as University-1652 and SUES-200). On the University-1652 dataset, the proposed method outperforms existing methods across all evaluation metrics, particularly in cases of image rotation and displacement, effectively preserving building shape perception. On the SUES-200 dataset, this invention demonstrates superior performance compared to traditional methods (such as CCR and Sample4Geo) across all settings, especially in improving accuracy for UAV target localization tasks. Results are as follows: Figure 7 and Figure 8 As shown.

[0140] 3. Comparison with some attention mechanisms: such as Figure 9 As shown, this application conducted ablation experiments on different attention mechanisms. Compared to the multi-scale convolutional kernels and channel attention used in SKNet, the proposed SSA integrates channel attention maps and spatial attention maps, and utilizes multi-scale bar convolutions to capture the directional features of the feature maps, thus achieving better performance. SENet performs worse than SSA because it ignores the importance of spatial attention. Although CBAM combines channel attention and spatial attention, it separates the two and fails to achieve cross-dimensional feature fusion. In contrast, the proposed SSA integrates channel and spatial information simultaneously in both stages, thus enriching the representation of information.

[0141] 4. Comparison of Different Loss Functions: Traditional KL divergence loss considers the alignment of positive and negative class distributions. However, in cross-perspective tasks, the probability distribution of the positive class often dominates, masking the information of the negative class. The proposed NCKL loss focuses only on the negative class distribution, mitigating the interference of the positive class on feature learning by excluding it. Figure 10As shown, the NCKL loss significantly improves performance on the SUES-200 dataset compared to the traditional KL divergence loss. Furthermore, the proposed NCKL loss, when combined with cross-entropy loss, is more effective than the Triplet loss in learning discriminative features across viewpoints. Compared to these two loss functions, the NCKL loss improves Recall@1 by approximately +4.5% and AP by +4% on the Drone→Satellite task on the SUES-200 dataset.

[0142] In a second aspect, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the attention-based cross-view geolocation method as described in the first aspect of the present invention.

[0143] The computer-readable storage medium may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory.

[0144] The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD ROM); the magnetic surface memory may be a disk storage device or a magnetic tape storage device.

[0145] The volatile memory may be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synclink dynamic random access memory (SLDRAM), and direct memory bus random access memory (DRRAM). The computer-readable storage media described in the embodiments of the present invention are intended to include these and any other suitable types of memory.

[0146] like Figure 11 As shown, in a third aspect, the present invention provides an electronic device 10, including a processor 101 and a storage medium 102, wherein a computer program is stored on the storage medium, and the computer program, when executed by the processor, implements the attention-based cross-view geolocation method as described in the first aspect of the present invention.

[0147] In some embodiments, the processor may be implemented by software, hardware, firmware, or a combination thereof, and may use at least one of the following: circuit, single or multiple application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), central processing units (CPUs), controllers, microcontrollers, and microprocessors, thereby enabling the processor to execute some or all of the steps or any combination of the steps in the attention-based cross-view geolocation method described in the various embodiments of this application.

[0148] Finally, it should be noted that although the above embodiments have been described in the description and drawings of this invention, this should not limit the scope of patent protection of this invention. Any technical solutions that are based on the essential concept of this invention, utilize the content described in the description and drawings of this invention to make equivalent structural or procedural substitutions or modifications, as well as the direct or indirect application of the technical solutions of the above embodiments to other related technical fields, are all included within the scope of patent protection of this invention.

Claims

1. A cross-view geolocation method based on an attention mechanism, characterized in that, Includes the following steps: Receive image information captured by drones or satellites; The image information is input into the trained neural network model, which outputs the geographic location information corresponding to the image information. The neural network model is trained by including the following steps: S1: Receive a first sample image acquired from a satellite perspective or a second sample image acquired from a drone perspective, and extract feature maps from the first sample image or the second sample image using a pre-trained ConvNeXt network; S2: Divide the extracted feature map into a first branch tensor, a second branch tensor, and a third branch tensor. Perform ECA operation on the first branch tensor, the second branch tensor, and the third branch tensor respectively, and then perform global average pooling to obtain the first channel description vector, the second channel description vector, and the third channel description vector. S3: The first branch tensor is weighted using the first channel description vector, the second branch tensor is weighted using the second channel description vector, and the third branch tensor is weighted using the third channel description vector to obtain the first attention feature map corresponding to the three branches. The three first attention feature maps are averaged to output the second attention feature map. S4: Perform a depthwise convolution operation on the second attention feature map to obtain the enhanced third attention feature map; S5: Determine geographical location information based on the third attention feature map.

2. The cross-view geolocation method based on attention mechanism as described in claim 1, characterized in that, The feature map extracted from the first sample image or the second sample image using a pre-trained ConvNeXt network is represented by the following formula: ； Where i∈{1,2}, x1 represents the first sample image, x2 represents the second sample image, and L i ∈R C×H×W This represents the extracted feature map, where C is the number of channels, and H and W represent the height and width of the image, respectively.

3. The cross-view geolocation method based on attention mechanism as described in claim 1, characterized in that, The feature map has a shape of C×H×W, where C is the number of channels, and H and W represent the height and width of the image, respectively. The first branch tensor is represented as W×H×C, the second branch tensor is represented as H×C×W, and the third branch tensor retains the original shape of the feature map and is represented as C×H×W; Steps S2 and S3 are expressed by the following formulas: ； ； Among them, L D Let X represent the input tensors of the first branch, the second branch, and the third branch. D X represents the first attention feature map of the output, X represents the second attention feature map, and the symbol σ represents the Sigmoid activation function.

4. The cross-view geolocation method based on attention mechanism as described in claim 1, characterized in that, Step S4 includes: Depth convolution operations are performed using convolution kernels of different lengths, and information exchange between channels is achieved through 1×1 convolutional layers to obtain an enhanced third attention feature map. Step S4 is expressed by the following formula: ； Where DwConv represents a 5×5 depthwise convolution, and Path i The output shows the result of two depthwise strip convolutions and the ontology after a 5×5 depthwise convolution. Y is the enhanced third attention feature map.

5. The cross-view geolocation method based on attention mechanism as described in claim 1, characterized in that, The training of the neural network model also includes the following steps: When calculating the KL divergence, the predicted probability of the positive class is ignored, and the softmax function is reapplied to calculate the predicted probability of the negative class. Specifically, this includes: The third attention feature map is passed to the classifier after average pooling. The classifier obtains the logit score for each class, and the logit scores of each class are passed to the Log_Softmax layer to obtain the logarithm of the predicted probability for each class, which is expressed by the following formula: ； Where L = {l1, l2, ..., l p ,...,l N }∈R 1×N This represents the logarithm of the predicted probability for each category, where N represents the total number of categories, and the subscript p indicates the positive class. To calculate the negative class KL divergence loss, we ignore l. p Therefore, L is rewritten as: ； then, The predicted probability distribution obtained through the Softmax layer is represented as follows: ； in, This represents the predicted probability distribution for all negative classes.

6. The cross-view geolocation method based on attention mechanism as described in claim 5, characterized in that, The method further includes: The output results of the first sample image acquired from the satellite perspective and the second sample image acquired from the UAV perspective are used to learn each other's predicted probability distributions, thereby aligning the depth feature distributions of the two. The calculation process for the negative class KL divergence is expressed by the following formula: ； in, and Let represent the negative class prediction probability distributions of the first sample image and the second sample image, respectively.

7. The cross-view geolocation method based on attention mechanism as described in claim 6, characterized in that, The method includes: The final loss function of the neural network model is obtained by comprehensively calculating the KL divergence loss, contrast loss, and cross-entropy loss.

8. The cross-view geolocation method based on attention mechanism as described in claim 7, characterized in that, The contrast loss is expressed as follows: ； Where q is the feature vector of the query image, R is the set of reference images, and r + These are the positive samples corresponding to the query image, and τ is a temperature parameter used to control the scaling of the similarity score. The cross-entropy loss is expressed as follows: ； in, It is the predicted probability of the neural network model, l p (y) is the logarithm of the predicted probability of the positive class; The formula for calculating the final loss function of the neural network model is as follows: ； Wherein, λ1 and λ2 are adjustable hyperparameters.

9. A computer-readable storage medium storing computer program instructions thereon, characterized in that, The computer program instructions, when executed by a processor, implement the method as described in any one of claims 1 to 8.

10. An electronic device comprising a memory and a processor, characterized in that, The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in any one of claims 1 to 8.

Images