Cross-view target geolocation method and device based on cross-task knowledge transfer

Abstract

The present application relates to a cross-view target geolocation method and device based on cross-task knowledge transfer. The method comprises: constructing a pre-training feature extraction module, including a first feature map extraction module and a second feature map extraction module with the same structure and weight sharing; based on a first loss function, inputting a reference image and a query image into the first feature map extraction module and the second feature map extraction module respectively for comparative learning training, to obtain trained first feature map extraction module and second feature map extraction module; constructing a cross-view target geolocation model, and transferring the trained first feature map extraction module and the second feature map extraction module to the cross-view target geolocation model; based on a second loss function, inputting the reference image and the query image with the query target into the cross-view target geolocation model for training, to obtain a trained cross-view target geolocation model. The present application can improve the target positioning accuracy and generalization ability in the cross-view scene.

Description

Technical Field

[0001] This invention relates to the field of cross-view target geolocation technology, and in particular to a cross-view target geolocation method and apparatus based on cross-task knowledge transfer. Background Technology

[0002] Cross-view object geo-localization (CVOGL) is a technique that determines the geographic coordinates of a target by detecting its location within a ground or UAV image in satellite imagery. Compared to positioning methods that rely on GPS signals, this technique utilizes spatial relationships based on visual features to locate specific building targets when GPS information is missing or in areas where GPS is denied.

[0003] However, CVOGL faces a dual challenge compared to traditional Object Localization: First, significant viewpoint differences between query and reference images lead to drastic changes in the appearance of the same object, rendering template matching mechanisms based on shallow visual features ineffective. Second, CVOGL focuses on unique instance matching and localization across viewpoints, with each query instance corresponding to only a single paired sample. The lack of aggregation of similar instances prevents the model from leveraging large-scale class data (such as the common 1000 classes in ImageNet) for feature learning. This scarcity of supervision signals leads to weak model generalization and the risk of overfitting. Therefore, optimizing modules in Object Localization in isolation (such as feature fusion units or prediction heads) cannot fundamentally solve these problems. Summary of the Invention

[0004] Therefore, it is necessary to provide a cross-view target geolocation method and device based on cross-task knowledge transfer that can achieve target correspondence in cross-view scenarios and improve the model's generalization ability, in order to address the above-mentioned technical problems.

[0005] A cross-viewpoint target geolocation method based on cross-task knowledge transfer, the method comprising:

[0006] Obtain a first dataset, which includes a reference image and a query image;

[0007] A pre-trained feature extraction module is constructed, which includes a first feature map extraction module and a second feature map extraction module with the same structure and shared weights.

[0008] A first loss function is constructed based on the principle of maximizing mutual information within positive sample pairs and minimizing mutual information within negative sample pairs. Then, based on the first loss function, the reference image and the query image are respectively input into the first feature map extraction module and the second feature map extraction module for comparative learning and training, so as to obtain the trained first feature map extraction module and the second feature map extraction module.

[0009] Obtain a second dataset, which includes a reference image and a query image with the query target;

[0010] A cross-view target geolocation model is constructed, and the trained first feature map extraction module and second feature map extraction module are transferred to the cross-view target geolocation model;

[0011] The first loss function is added to the second loss function, and then, based on the second loss function, the reference image and the query image with the query target are input into the cross-view target geolocation model for training, so as to obtain the trained cross-view target geolocation model.

[0012] On the other hand, a cross-view target geolocation device based on cross-task knowledge transfer is also provided, including:

[0013] The first dataset acquisition module is used to acquire a first dataset, which includes a reference image and a query image.

[0014] A pre-trained feature extraction module construction module is used to construct a pre-trained feature extraction module, wherein the pre-trained feature extraction module includes a first feature map extraction module and a second feature map extraction module with the same structure and shared weights;

[0015] The pre-trained feature extraction module training module is used to construct a first loss function based on the principle of maximizing mutual information within positive sample pairs and minimizing mutual information within negative sample pairs; then, based on the first loss function, the reference image and the query image are respectively input into the first feature map extraction module and the second feature map extraction module for comparative learning training, so as to obtain the trained first feature map extraction module and the second feature map extraction module;

[0016] The second dataset acquisition module is used to acquire a second dataset, which includes a reference image and a query image with a query target.

[0017] A cross-view target geolocation model construction module is used to construct a cross-view target geolocation model and transfer the trained first feature map extraction module and second feature map extraction module to the cross-view target geolocation model.

[0018] The cross-view target geolocation model training module is used to add the first loss function to the second loss function, and then, based on the second loss function, input the reference image and the query image with the query target into the cross-view target geolocation model for training, so as to obtain the trained cross-view target geolocation model.

[0019] Compared with existing technologies, the cross-view target geolocation method and apparatus based on cross-task knowledge transfer provided by this invention have the following advantages:

[0020] 1. In the first stage, a pre-trained feature extraction module is trained using the first dataset to learn the basic matching rules of cross-view images. Then, this is transferred to the cross-view target geolocation model, which can improve the model's sample utilization efficiency and reduce the dependence on the location task annotation data.

[0021] 2. In the pre-training feature extraction module, training is performed using the extreme value constraint of mutual information between positive and negative samples. This endows the model with the ability to distinguish between positive and negative samples, capturing the commonalities between cross-view image pairs and improving the overall discriminative power of the model. After performing cross-task knowledge transfer, the robustness of the cross-view target geolocation model can be enhanced, improving the target positioning accuracy and generalization ability in cross-view scenarios, and adapting to the positioning needs of complex scenarios.

[0022] 3. By incorporating the first loss function into the second loss function, the cross-view target geolocation model avoids catastrophic forgetting of knowledge from the first stage, thus ensuring knowledge continuity and enhancing task adaptability. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention, and those skilled in the art can obtain other related drawings based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating the cross-perspective target geolocation method based on cross-task knowledge transfer provided in Example 1.

[0025] Figure 2 This is a structural block diagram of the cross-view target geolocation method based on cross-task knowledge transfer provided in Example 1;

[0026] Figure 3 This is a schematic diagram of cross-view feature fusion provided in Example 1;

[0027] Figure 4 This is a schematic diagram of the three parts of the target-scene joint loss provided in Example 1;

[0028] Figure 5 This is a structural block diagram of the cross-view target geolocation device based on cross-task knowledge transfer provided in Example 2.

[0029] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0031] It is understood that the technical solutions of the various embodiments of the present invention can be combined with each other, but only if they are based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0032] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0033] Example 1

[0034] like Figure 1 and Figure 2 As shown, the cross-view target geolocation method based on cross-task knowledge transfer provided in this embodiment includes the following steps:

[0035] Step 201: Obtain the first dataset, which includes the reference image and the query image.

[0036] Step 202: Construct a pre-trained feature extraction module, which includes a first feature map extraction module and a second feature map extraction module with the same structure and shared weights.

[0037] Step 203: Construct a first loss function based on the principle of maximizing mutual information within positive sample pairs and minimizing mutual information within negative sample pairs; then, based on the first loss function, input the reference image and the query image into the first feature map extraction module and the second feature map extraction module respectively for comparative learning training to obtain the trained first feature map extraction module and second feature map extraction module.

[0038] Step 204: Obtain the second dataset, which includes the reference image and the query image with the query target.

[0039] Step 205: Construct a cross-view target geolocation model and transfer the trained first feature map extraction module and second feature map extraction module into the cross-view target geolocation model.

[0040] Step 206: Add the first loss function to the second loss function, and then, based on the second loss function, input the reference image and the query image with the query target into the cross-view target geolocation model for training, to obtain the trained cross-view target geolocation model.

[0041] It is understandable that the first and second datasets primarily use the CVOGL dataset, which contains 12,478 instances from three perspectives: ground, drone, and satellite. Satellite images are used as reference images, with each image sampling from 0.09 to 1 km. 2 Internally, the resolution has been uniformly adjusted to 1024. 1024 pixels. Each reference image corresponds to one ground image and one drone image containing the query target. The ground image and drone image, used as query images, are both uniformly adjusted to 256 pixels. 512 pixels and 256 256 pixels. Three types of images are used to solve ground-based problems. Satellites and drones This involves a cross-view target geolocation task across two satellite scenarios. Each scenario has 6239 instances used for training, validation, and testing, with a ratio of approximately 7:1.5:1.5.

[0042] In addition, the first dataset also incorporates two additional open-source cross-view datasets, CVUSA and SUES-200, for pre-training the first-stage network. The SUES-200 dataset contains 200 scenes from both UAV and satellite perspectives; the CVUSA dataset contains 35,532 cross-view sample pairs from ground and satellite perspectives. SUES-200 is primarily used for UAVs. In its satellite missions, CVUSA is primarily used on the ground. In satellite missions, the introduction of additional datasets CVUSA and SUES-200 helps the model learn richer features, enhancing its generalizability across different instances. Furthermore, expanding the training set broadens the data, mitigating the scarcity of supervision signals and model overfitting issues in CVOGL tasks to some extent, thus improving model generalization.

[0043] In the first stage of pre-training, cross-view invariant features are learned by minimizing the feature distance between samples of the same class and maximizing the distance between samples of different classes, thereby reducing feature differences caused by viewpoint variations. It attempts to mine view-invariant features from multiple perspectives, including building outlines and road topology, to identify high-level semantic information common to cross-view images and encode this information into feature vectors to achieve accurate feature matching. It is trained on a first dataset and uses contrastive learning to guide the model to learn discriminative features between samples, rather than simply memorizing the distribution of positive samples. This avoids overfitting of the pre-trained feature extraction module to positive samples and improves the discriminativeness and robustness of the pre-trained feature extraction module's representation.

[0044] In the second stage of fine-tuning, the image classification task knowledge of the pre-trained feature extraction module is transferred to the target detection task of the cross-view target geolocation model. This inherits the global association information from the first stage, significantly improving the performance of semi-supervised target detection in the target domain (unlabeled data) and providing a scalable solution for cross-domain target detection.

[0045] The cross-view target geolocation model builds upon the scene-level knowledge learned in the first stage, incorporating target geolocation supervision knowledge to enable the model to focus globally on matching target instances across different viewpoints. Furthermore, to prevent catastrophic forgetting of first-stage knowledge during the second-stage training, a new second loss function is proposed. This second loss function balances the global constraints of scene-level features with the refined supervision of target-level localization, achieving multi-level joint optimization of the model and improving its stability.

[0046] This invention employs a two-stage training approach. In the first stage, a pre-trained feature extraction module is trained to improve the matching ability of cross-view image samples and identify view-invariant features. In the second stage, the pre-trained feature extraction module, which includes cross-view scene-level associations, is reused in the cross-view target geolocation model for target detection and matching. To prevent forgetting the scene-level association knowledge from the first stage during the second stage of target bounding box prediction training, a second loss function is designed to achieve multi-level supervision of scene-target joint training. The cross-view target geolocation method based on cross-task knowledge transfer (STONet) proposed in this invention enables a transition from image-level to target-level localization.

[0047] In one embodiment, the first loss function is expressed as:

[0048] (1);

[0049] (2);

[0050] (3);

[0051] In the formula, Represents the first loss function; This represents an anchor query sample; This indicates a positive satellite image sample from the same geographic location as the anchor query sample; This indicates that the anchor query sample comes from satellite imagery negative samples from different geographical locations; max is a function that takes the maximum value among all parameters. This indicates reducing the feature distance between positive sample pairs; This indicates that the feature distance of negative sample pairs is increased.

[0052] It is understandable that the first loss function constructs anchor points. Positive samples negative samples Triples aim to reduce the feature distance between positive sample pairs of the anchor query image and the reference image at the same geographic location. Increase the distance between the query image and negative samples of the reference image that are not in the same geographical location. The first loss function can drive the model to learn view-invariant semantic features between cross-view images from the same geographical location, making the features between positive sample pairs similar, thereby achieving cross-view scene-level semantic alignment (such as building cluster distribution, road topology).

[0053] In one embodiment, based on a first loss function, the reference image and the query image are respectively input into the first feature map extraction module and the second feature map extraction module for comparative learning training, including:

[0054] The query image is input into the first feature map extraction module for processing, and the first feature map is output; the reference image is input into the second feature map extraction module for processing, and the second feature map is output.

[0055] Perform an inverse polar coordinate feature transformation on the first feature map to obtain the transformed first feature map;

[0056] Using the first loss function As a constraint, the transformed first feature map With the second feature map Perform similarity-based comparative learning.

[0057] Specifically, the pre-trained feature extraction module constructs a Siamese network based on contrastive learning to capture common features (such as illumination invariance and geometric transformation robustness) between cross-viewpoint image pairs. This Siamese network includes a first feature map extraction module and a second feature map extraction module, which share weights. The first feature map extraction module processes the query image, and the second feature map extraction module processes the reference image. A first loss function based on cosine similarity is used to measure the model training loss, and then the module parameters are updated through backpropagation until the module loss converges, resulting in the trained first and second feature map extraction modules. During pre-training, to ensure that the pre-trained feature extraction module can learn general and robust feature representations, it is trained using a first dataset containing the CVOGL, CVUSA, and SUES-200 datasets, which improves the generalization ability of the pre-trained feature extraction module.

[0058] The query image and reference image are abstracted to a feature space level. Based on this, the input query image data space in the pre-trained feature extraction module is... The reference image data space is The corresponding data distribution is The goal of the pre-trained feature extraction module is to find a mapping. , so that:

[0059] (4)

[0060] In the formula, express One of the query image samples; This represents a reference image sample in X. f This is used to map the correspondence between query images and reference images. During the training process of the pre-trained feature extraction module, it typically constructs positive and negative sample pairs, maximizing the mutual information within positive sample pairs (ground and satellite images from the same geographical location) and minimizing the mutual information within negative sample pairs (ground and satellite images from different geographical locations). The information gain is expressed as:

[0061] (5)

[0062] In the formula, This involves introducing a discriminative gain specific to negative samples. The introduction of negative samples endows the model with the ability to distinguish between positive and negative samples, thereby improving the model's overall discriminative power. ; This represents the positive sample corresponding to X within the reference data space; This represents the negative sample corresponding to X within the reference data space.

[0063] In the pre-training feature extraction module, an inverse polar coordinate feature transformation was also performed on the first feature map. This inverse polar coordinate feature transformation addresses the feature space mismatch problem encountered when fusing features from ground images and satellite images with significant viewpoint differences. It transforms the distribution of ground image features from a horizontal distribution to a rotational distribution, improving the consistency of target distribution positions under orthogonal viewpoint differences.

[0064] In one embodiment, the cross-view target geolocation model includes a location encoder, a feature extraction module, a cross-view feature fusion module, and a target bounding box prediction head; wherein, the trained first feature map extraction module and second feature map extraction module are transferred to the feature extraction module.

[0065] In one embodiment, based on a second loss function, a reference image and a query image containing the query target are input into a cross-view target geolocation model for training, resulting in a trained cross-view target geolocation model, including:

[0066] The query image containing the query target is input into the location encoder for encoding, and the output is the location feature.

[0067] The location features are added to the query image containing the query target and then input into the first feature map extraction module for processing, outputting the third feature map.

[0068] The reference image is input into the second feature map extraction module for processing, and the fourth feature map is output.

[0069] Perform an inverse polar coordinate feature transformation on the third feature map to obtain the transformed third feature map.

[0070] The transformed third feature map and the fourth feature map are input into the cross-view feature fusion module for fusion to obtain the fused features.

[0071] The fused features are input into the target bounding box prediction head for processing, and the predicted bounding box and confidence score are output.

[0072] Based on the second loss function, the difference between the predicted bounding box and the true bounding box is calculated by combining the confidence score. The model parameters are then updated through backpropagation until the model loss converges, resulting in a trained cross-view target geolocation model. Finally, the trained cross-view target geolocation model is used for target localization in cross-view scenarios.

[0073] It is understandable that the feature extraction module in the cross-view target geolocation model originates from the pre-trained feature extraction module trained in the first stage. The scene-level associations and view-invariant features learned in the pre-trained feature extraction module are transferred to the cross-view target geolocation model, preserving the ability to perceive macroscopic scene structures such as road topology and building outlines. Then, based on the transferred scene-level understanding knowledge, the pre-trained feature extraction module is fine-tuned to further establish target-level associations.

[0074] After obtaining the third and fourth feature maps through the transferred feature extraction module, the third feature map undergoes inverse polar coordinate feature transformation to obtain the transformed third feature map. Then, the transformed third and fourth feature maps are input into the cross-view feature fusion module for fusion to obtain the fused features. Figure 3 As shown, the cross-view feature fusion module is mainly for realizing the interaction between ground images and UAV images. The transformed third feature map is processed by global average pooling and then multiplied with the fourth feature map as the weight for cross-view interaction. The weight is then applied to the fourth feature map element by element to obtain fused features containing information from both views, which are then used in the subsequent target box prediction header to predict the bounding box.

[0075] Furthermore, since the almost orthogonal viewpoint difference between ground and satellite images is significantly greater than that between UAV and satellite images, directly calculating attention may lead to cross-viewpoint feature misalignment. Therefore, to alleviate the orthogonal geometric difference problem in ground-satellite scenarios, the third feature map in the cross-viewpoint target geolocation model is also subjected to inverse polar coordinate feature transformation. This transforms the original ground panoramic image features into a geometric distribution from a top-down viewpoint, significantly mitigating the dual geometric and semantic differences between ground-satellite cross-viewpoint features. This makes the cross-viewpoint interactive attention weight distribution of ground-satellite tasks more reasonable and improves the consistency of target distribution positions under orthogonal viewpoint differences.

[0076] In the cross-view target geolocation model, the input query image data space is: The target data space for the query is The reference image data space is The reference target data space for prediction is Output predicted target location information The probability distribution is Its main goal is to find a mapping. , so that:

[0077] (6)

[0078] in, Used to map the correspondence between query image, query target, reference image and reference target.

[0079] In traditional cross-view target geolocation methods, Its positive samples The samples are input together into the model to learn viewpoint-invariant features between positive samples, and their information gain is expressed as:

[0080] (7)

[0081] This invention utilizes a knowledge transfer method to transfer the knowledge learned by the pre-trained feature extraction module in the first stage. The knowledge was transferred to the cross-view target geolocation model, and in the second stage, the cross-view target geolocation model was also constructed during training. and The constructed positive and negative sample pairs prevent the cross-perspective target geolocation model from catastrophically forgetting the knowledge from the first stage. Therefore, the total information gain of the method proposed in this invention can be expressed as:

[0082] (8)

[0083] In the formula, This represents the information gain in the first stage. This indicates the information gain in the second stage.

[0084] As can be seen from the above formula, the cross-view target geolocation model contains more... and Inter-image information includes viewpoint-invariant features and global associations within the query and reference images. Therefore, and The gain in mutual information can obviously help with the target The information gain can be expressed as:

[0085] (9)

[0086] In the formula, This represents the change in information gain.

[0087] in, Depend on It is determined by three conditions.

[0088] According to the conditional entropy decomposition formula (9), the gain information after migration is obtained, which is expressed as:

[0089] (10);

[0090] In the formula, This represents the known two-stage, cross-perspective target geolocation method STONet based on cross-task knowledge transfer. Three types of input variables Uncertainty; In the absence of any reference information Uncertainty; Represents the reference target data space for prediction after migration. With joint variable group ( Mutual information between ) where X is the query image data space, It is to query the target data space. It is the reference image data space; Represents the reference target data space for prediction before migration. With joint variable group ( Mutual information between them; This indicates that the migration was known prior to the migration. Three types of input variables Uncertainty.

[0091] As can be seen, compared with the traditional single-stage cross-view target geolocation method, the two-stage STONet based on cross-task knowledge transfer proposed in this invention can reduce the uncertainty of target localization, thereby improving the decision reliability of the model and the credibility of reasoning knowledge.

[0092] In one embodiment, the expression for the second loss function is:

[0093] (11);

[0094] In the formula, This represents the second loss function; This represents the loss function for predicting the bounding box location; Represents the confidence loss function; Represents the first loss function; , This represents a hyperparameter, which is typically set to 0.5.

[0095] In one embodiment, the expression for the loss function for predicting the bounding box location is:

[0096] (12);

[0097] In the formula, Indicates the coordinates of the predicted bounding box center point; This represents the width and height values ​​of the predicted bounding box; Represents the coordinates of the center point of the actual bounding box; Represents the width and height values ​​of the actual bounding box; This represents the sigmoid function. This represents the floor function.

[0098] In one embodiment, the confidence loss function is expressed as follows:

[0099] (13);

[0100] In the formula, This represents the confidence level of all predicted bounding boxes for the k-th instance. Indicates the binary label corresponding to the predicted bounding box; Indicates the number of instances.

[0101] It is understandable that the cross-view target geolocation model outputs the predicted bounding box of the target in the satellite image (containing the center coordinates and width and height values ​​of the predicted bounding box) and the confidence level of the predicted bounding box. Therefore, the second loss function includes the predicted bounding box position loss, which reflects the positional difference between the predicted bounding box and the ground truth box. And the confidence loss in the accuracy of target existence prediction. Furthermore, in order to inherit cross-view scene-level knowledge from the weights of the first-stage training and to avoid catastrophic forgetting, the first loss function is... It is also incorporated into the training of cross-view target geolocation models. For example... Figure 4 As shown, the loss for predicting the bounding box location With confidence loss In model training, it refers to the constraints imposed on model training at the target level; the first loss function. It refers to the constraints on model training at the scene level during model training.

[0102] Confidence loss It is used to solve the problem of grid prediction errors, when the model predicts the target location to be on the correct grid. The prediction is affirmed, while it is suppressed when the model predicts the target location to an incorrect grid. Among all anchor boxes, those with the largest intersection-union area with the true bounding box are labeled "1" and given high confidence as positive samples to ensure the target is detected; other anchor boxes are labeled "0" and given low confidence as negative samples to avoid false detections. This loss enables the model to distinguish between background regions (no target) and foreground regions (target present), thus fulfilling the task of confirming whether the target is within the anchor box.

[0103] Predicting bounding box location loss This implementation achieves the task of optimizing the position and size of the bounding box within a confirmed grid containing the target. Among all candidate anchor boxes, the one with the largest intersection-union area is selected as the predicted bounding box. express.

[0104] Predicting bounding box location loss The absolute differences between predicted coordinates and dimensions and actual coordinates and dimensions are transformed into the calculation of relative offset differences. That is, the absolute difference between coordinates is transformed into the difference between the offset ratio of the predicted bounding box center point coordinates relative to the top left corner of the current grid and the offset ratio of the actual bounding box center point coordinates relative to the top left corner of the current grid; the absolute difference between dimensions is transformed into the difference between the logarithmic scaling of the predicted bounding box dimensions relative to the anchor box dimensions and the logarithmic scaling of the actual bounding box dimensions relative to the anchor box dimensions.

[0105] As can be seen, the second loss function designed in this invention implements a multi-level supervision strategy through global coarse-grained supervision of scene-level features and fine-grained supervision of target-level localization. Among these, the scene-level loss... The guided model captures semantic consistency across perspectives (such as building layout and road topology), providing spatial priors for target localization; target-level loss. and The confidence-location co-optimization mechanism drives the model to accurately focus on key targets in complex backgrounds, ultimately achieving precise cross-view target geolocation.

[0106] In one embodiment, the method proposed in this invention is verified.

[0107] First, a pre-trained feature extraction module was constructed using PyTorch, employing a two-branch network architecture. The two branches take the query image and reference image as inputs, respectively. For ease of reproduction, the pre-trained feature extraction module used convNeXt_tiny from torchvision, loaded with ImageNet pre-trained weights, with the two branches sharing weights. Ground images, drone images, and satellite images were resized to 256 × 512 pixels, 256 × 256 pixels, and 1024 × 1024 pixels, respectively, before input, and augmented using common techniques such as rotation, noise addition, and random cropping. The first stage of pre-training used the SGD optimizer with an initial learning rate of 1e-3, adjusted using cosine annealing, and trained for 100 generations. The second stage used the RMSProp optimizer, fine-tuning the model with an initial learning rate of 1e-4, trained for 30 generations, and again adjusted using cosine annealing. Both training stages were performed on two NVIDIA RTX4090D graphics cards with a batch size of 12. (Ground view) The "satellite perspective" training data comes from a subset of ground and satellite perspective image pairs in the CVUSA and CVOGL datasets; the "drone perspective" training data comes from a subset of ground and satellite perspective image pairs. The "satellite perspective" training data comes from a subset of drone and satellite perspective image pairs in the SUES-200 and CVOGL datasets.

[0108] After the model was trained, the computational efficiency and accuracy of the proposed method (Ours), DetGeo, AttenGeo, and AMPNet were systematically evaluated, as shown in Table 1. The proposed method achieves a significant reduction in computational efficiency while maintaining the current best positioning accuracy. Compared to the baseline method DetGeo, the proposed method reduces the computational load by an average of 50% and the number of parameters by 60%. Furthermore, it achieves comparable target positioning accuracy to AMPNet with only 43.73% of the parameters and 50% of the computational load. In summary, the proposed method provides a real-time, high-precision target geolocation solution for edge computing devices.

[0109] Table 1 Computational Complexity

[0110]

[0111] This invention proposes a cross-view target geolocation method based on cross-task knowledge transfer, which solves the problem of establishing cross-view target associations by combining cross-view scene-level knowledge with target-level knowledge. First, a two-stage training method is designed so that the model can predict local target locations based on learning global cross-view associations. Second, the proposed second loss function achieves multi-level supervision at the scene and target levels, while avoiding catastrophic forgetting of the pre-trained knowledge from the first stage. Furthermore, to address the potential geometric inconsistency between ground image features and satellite image features, an inverse polar coordinate feature transformation method is proposed. This method guides feature alignment through geometric constraints, assisting in subsequent cross-view feature fusion. Finally, through thorough experimental verification, the proposed method significantly improves performance while maintaining the advantages of fewer parameters and higher computational efficiency.

[0112] It should be understood that, although this embodiment Figure 1 The steps are shown sequentially as indicated by the arrows, but they are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order in which these steps are performed; they can be executed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0113] Example 2

[0114] Based on the cross-task knowledge transfer-based cross-view target geolocation method in Embodiment 1, this embodiment discloses a cross-task knowledge transfer-based cross-view target geolocation device, such as... Figure 5 As shown, the cross-view target geolocation device based on cross-task knowledge transfer includes: a first dataset acquisition module 401, a pre-trained feature extraction module construction module 402, a pre-trained feature extraction module training module 403, a second dataset acquisition module 404, a cross-view target geolocation model construction module 405, and a cross-view target geolocation model training module 406, wherein:

[0115] The first dataset acquisition module 401 is used to acquire the first dataset, which includes a reference image and a query image.

[0116] The pre-trained feature extraction module construction module 402 is used to construct the pre-trained feature extraction module, which includes a first feature map extraction module and a second feature map extraction module with the same structure and shared weights.

[0117] The pre-trained feature extraction module training module 403 is used to construct a first loss function based on the principle of maximizing mutual information within positive sample pairs and minimizing mutual information within negative sample pairs. Then, based on the first loss function, the reference image and the query image are respectively input into the first feature map extraction module and the second feature map extraction module for comparative learning training, so as to obtain the trained first feature map extraction module and the second feature map extraction module.

[0118] The second dataset acquisition module 404 is used to acquire a second dataset, which includes a reference image and a query image with a query target.

[0119] The cross-view target geolocation model construction module 405 is used to construct a cross-view target geolocation model and transfer the trained first feature map extraction module and second feature map extraction module to the cross-view target geolocation model.

[0120] The cross-view target geolocation model training module 406 is used to add the first loss function to the second loss function, and then, based on the second loss function, input the reference image and the query image with the query target into the cross-view target geolocation model for training, so as to obtain the trained cross-view target geolocation model.

[0121] In this embodiment, the specific working process and working principle of the first dataset acquisition module 401, the pre-trained feature extraction module construction module 402, the pre-trained feature extraction module training module 403, the second dataset acquisition module 404, the cross-view target geolocation model construction module 405, and the cross-view target geolocation model training module 406 are the same as those in Embodiment 1, and therefore will not be described again in this embodiment. Each unit module can be implemented entirely or partially through software, hardware, or a combination thereof. Each unit module can be embedded in or independent of the processor in a computer device in hardware form, or it can be stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each of the above unit modules.

[0122] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0123] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0124] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A cross-perspective target geolocation method based on cross-task knowledge transfer, characterized in that, The method includes: Obtain a first dataset, which includes a reference image and a query image; A pre-trained feature extraction module is constructed, which includes a first feature map extraction module and a second feature map extraction module with the same structure and shared weights. A first loss function is constructed based on the principle of maximizing mutual information within positive sample pairs and minimizing mutual information within negative sample pairs. Then, based on the first loss function, the reference image and the query image are respectively input into the first feature map extraction module and the second feature map extraction module for comparative learning and training, so as to obtain the trained first feature map extraction module and the second feature map extraction module. Obtain a second dataset, which includes a reference image and a query image with the query target; A cross-view target geolocation model is constructed, and the trained first feature map extraction module and second feature map extraction module are transferred to the cross-view target geolocation model; The first loss function is added to the second loss function, and then based on the second loss function, the reference image and the query image with the query target are input into the cross-view target geolocation model for training to obtain the trained cross-view target geolocation model. Based on the first loss function, the reference image and the query image are respectively input into the first feature map extraction module and the second feature map extraction module for comparative learning training, including: The query image is input into the first feature map extraction module for processing, and a first feature map is output; the reference image is input into the second feature map extraction module for processing, and a second feature map is output. Perform an inverse polar coordinate feature transformation on the first feature map to obtain the transformed first feature map; Using the first loss function as a constraint, a similarity-based comparative learning is performed on the transformed first feature map and the second feature map.

2. The cross-view target geolocation method based on cross-task knowledge transfer according to claim 1, characterized in that, The first loss function expression is: ； ； ； In the formula, Represents the first loss function; This represents an anchor query sample; This indicates a positive satellite image sample from the same geographic location as the anchor query sample; This indicates that the satellite imagery samples from different geographical locations are negative samples compared to the anchor query samples; `max` is a function that takes the maximum value among all arguments. This indicates reducing the feature distance between positive sample pairs; This indicates that the feature distance of negative sample pairs is increased.

3. The cross-perspective target geolocation method based on cross-task knowledge transfer according to claim 1 or 2, characterized in that, The cross-view target geolocation model includes a location encoder, a feature extraction module, a cross-view feature fusion module, and a target bounding box prediction head; wherein, the trained first feature map extraction module and the second feature map extraction module are transferred to the feature extraction module.

4. The cross-view target geolocation method based on cross-task knowledge transfer according to claim 3, characterized in that, The trained first feature map extraction module and second feature map extraction module are migrated to the feature extraction module, and the migrated gain information is represented as follows: ； In the formula, This represents the known two-stage, cross-perspective target geolocation method STONet based on cross-task knowledge transfer. Three types of input variables Uncertainty; In the absence of any reference information Uncertainty; Represents the reference target data space for prediction after migration. With joint variable group ( Mutual information between ) where X is the query image data space, It is to query the target data space. It is a reference image data space Represents the reference target data space for prediction before migration. With joint variable group ( Mutual information between ) This indicates that the migration was known prior to the migration. Three types of input variables Uncertainty.

5. The cross-perspective target geolocation method based on cross-task knowledge transfer according to claim 3, characterized in that, The expression for the second loss function is: ； In the formula, This represents the second loss function; This represents the loss function for predicting the bounding box location; Represents the confidence loss function; Represents the first loss function; , This represents hyperparameters.

6. The cross-view target geolocation method based on cross-task knowledge transfer according to claim 5, characterized in that, The expression for the loss function for predicting the bounding box location is: ； In the formula, Indicates the coordinates of the predicted bounding box center point; This represents the width and height values ​​of the predicted bounding box; Represents the coordinates of the center point of the actual bounding box; Represents the width and height values ​​of the actual bounding box; This represents the sigmoid function. This represents the floor function; Indicates the number of instances.

7. The cross-view target geolocation method based on cross-task knowledge transfer according to claim 5, characterized in that, The confidence loss function is expressed as follows: ； In the formula, This represents the confidence level of all predicted bounding boxes for the k-th instance. Indicates the binary label corresponding to the predicted bounding box; Indicates the number of instances.

8. The cross-view target geolocation method based on cross-task knowledge transfer according to claim 5, characterized in that, Based on the second loss function, the reference image and the query image with the query target are input into the cross-view target geolocation model for training, resulting in a trained cross-view target geolocation model, including: The query image containing the query target is input into a location encoder for encoding, and the location features are output. The location features are added to the query image with the query target and then input into the first feature map extraction module for processing, and a third feature map is output. The reference image is input into the second feature map extraction module for processing, and a fourth feature map is output. Perform an inverse polar coordinate feature transformation on the third feature map to obtain the transformed third feature map; The transformed third feature map and the fourth feature map are input into the cross-view feature fusion module for fusion to obtain the fused feature; The fused features are input into the target bounding box prediction head for processing, and the predicted bounding box and confidence score are output. Based on the second loss function, the difference between the predicted bounding box and the true bounding box is calculated by combining the confidence score. The model parameters are updated through backpropagation until the model loss converges, and a trained cross-view target geolocation model is obtained.

9. A cross-view target geolocation device based on cross-task knowledge transfer, characterized in that, The apparatus employing the cross-view target geolocation method based on cross-task knowledge transfer as described in any one of claims 1 to 8 includes: The first dataset acquisition module is used to acquire a first dataset, which includes a reference image and a query image. A pre-trained feature extraction module construction module is used to construct a pre-trained feature extraction module, wherein the pre-trained feature extraction module includes a first feature map extraction module and a second feature map extraction module with the same structure and shared weights; The pre-trained feature extraction module training module is used to construct a first loss function based on the principle of maximizing mutual information within positive sample pairs and minimizing mutual information within negative sample pairs; then, based on the first loss function, the reference image and the query image are respectively input into the first feature map extraction module and the second feature map extraction module for comparative learning training, so as to obtain the trained first feature map extraction module and the second feature map extraction module; The second dataset acquisition module is used to acquire a second dataset, which includes a reference image and a query image with a query target. A cross-view target geolocation model construction module is used to construct a cross-view target geolocation model and transfer the trained first feature map extraction module and second feature map extraction module to the cross-view target geolocation model. The cross-view target geolocation model training module is used to add the first loss function to the second loss function, and then, based on the second loss function, input the reference image and the query image with the query target into the cross-view target geolocation model for training, so as to obtain the trained cross-view target geolocation model.

Images