Cross-view geographic positioning method and device
By using a twin network model with shared weights in cross-view geolocation, feature extraction and loss calculation of drone images and satellite images is solved, and the problem of poor matching effect and unbalanced sample is improved, and the positioning accuracy and inference speed are improved. Suitable for edge computing devices.
Patent Information
- Application Number
- CN202411086555.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-08
- Publication Date
- 2024-11-29
AI Technical Summary
In the existing cross-view geolocation technology, the matching effect of drone images and satellite images is poor, and the proportion of positive and negative samples is unbalanced, resulting in inaccurate positioning.
Using the initial twin network model with shared weights, after rotating, grid enhancement, random discarding and color jittering on the drone and satellite images, they are input into the SDCL loss function for training, and learn effective representations to alleviate positive and negative samples. Imbalance problem improves matching effect.
It significantly improves the matching effect of cross-view geolocation, reduces computing costs and inference time, and is suitable for edge computing devices such as drones and satellites with limited computing power.
Smart Images

Figure CN119048775A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of cross-view geolocation, and in particular to a cross-view geolocation method and device. Background Art
[0002] Conventional positioning methods rely on satellite GPS signals, but in densely populated cities or remote forests, GPS signals are often weak, resulting in inaccurate positioning. Therefore, in order to meet the needs of high-precision geolocation, cross-view geolocation technology has emerged.
[0003] Cross-view geolocation compares and analyzes the similarity of images from different perspectives (such as drone images and satellite images), and uses the geographic location information contained in satellite images to locate the target object of the drone. Specifically, taking the drone-view image as input, the system will find the satellite image that is most similar to the drone image, and determine the location of the drone through the geographic information of the satellite image.
[0004] However, there are large visual differences between drone images and satellite images, resulting in poor matching results. In addition, the ratio of positive and negative samples between drone images and satellite images is unbalanced, and how to effectively balance positive and negative samples is a challenge. Existing technologies usually transform drone images into a perspective similar to that of satellite images for matching, but this direct geometric transformation and brute-force feature matching often do not produce ideal results. Summary of the invention
[0005] The present invention solves the technical problem of poor matching effect of cross-view geolocation in the prior art by providing a cross-view geolocation method and device.
[0006] The present invention provides a cross-view geolocation method, comprising:
[0007] Inputting the training UAV image into the first Siamese sub-network of the initial Siamese network model to obtain an embedded vector feature of the training UAV image;
[0008] Inputting the training satellite image into the second twin sub-network of the initial twin network model to obtain the embedded vector feature of the training satellite image;
[0009] The embedded vector features of the training drone image and the embedded vector features of the training satellite image are input into the SDCL loss function , the loss between the two is calculated, and training is performed through gradient backpropagation. After the training, the optimal twin network model is obtained; wherein q is the embedding vector feature of the training drone image, g is the embedding vector feature of the training satellite image, G is the set of embedding vector features of the training drone image, Q is the set of embedding vector features of the training satellite image, and g + is a positive sample matching the q, is a negative sample that does not match the q, q + is the positive sample matched by g, is a negative sample that does not match the g, τ is a temperature parameter, and λ is a weight coefficient of the symmetric term;
[0010] The real-time UAV image is input into the optimal twin network model to obtain the matching satellite image.
[0011] Specifically, before inputting the training drone image into the first twin sub-network of the initial twin network model to obtain the embedded vector feature of the training drone image, the method further includes:
[0012] Rotating the training UAV image and the training satellite image;
[0013] Perform grid enhancement on the rotated drone images and satellite images;
[0014] Randomly discard some grids in grid-enhanced drone and satellite images;
[0015] Color jittering is performed on the randomly discarded UAV images and satellite images to obtain enhanced training UAV images and training satellite images.
[0016] Specifically, the weights of the first twin network and the second twin network in the initial twin network model are shared.
[0017] Specifically, the first twin sub-network and the second twin sub-network in the initial twin network model are DaViT.
[0018] The present invention also provides a cross-view geographic positioning device, comprising:
[0019] A drone image embedding vector feature acquisition module is used to input the training drone image into the first twin sub-network of the initial twin network model to obtain the embedding vector feature of the training drone image;
[0020] A satellite image embedding vector feature acquisition module, used to input the training satellite image into the second twin sub-network of the initial twin network model to obtain the embedding vector feature of the training satellite image;
[0021] A model training module is used to input the embedded vector features of the training drone image and the embedded vector features of the training satellite image into the SDCL loss function , the loss between the two is calculated, and training is performed through gradient backpropagation. After the training, the optimal twin network model is obtained; wherein q is the embedding vector feature of the training drone image, g is the embedding vector feature of the training satellite image, G is the set of embedding vector features of the training drone image, Q is the set of embedding vector features of the training satellite image, and g + is a positive sample matching the q, is a negative sample that does not match the q, q + is the positive sample matched by g, is a negative sample that does not match the g, τ is a temperature parameter, and λ is a weight coefficient of the symmetric term;
[0022] The cross-view geolocation module is used to input real-time drone images into the optimal twin network model to obtain matching satellite images.
[0023] Specifically, it also includes:
[0024] A rotation module, used to rotate the training UAV image and the training satellite image;
[0025] The grid enhancement module is used to perform grid enhancement on the rotated UAV images and satellite images;
[0026] A random discarding module is used to randomly discard some grids in the grid-enhanced drone images and satellite images;
[0027] The color jittering module is used to perform color jitter on the randomly discarded drone images and satellite images to obtain enhanced training drone images and training satellite images.
[0028] Specifically, the weights of the first twin network and the second twin network in the initial twin network model are shared.
[0029] Specifically, the first twin sub-network and the second twin sub-network in the initial twin network model are DaViT.
[0030] One or more technical solutions provided in the present invention have at least the following technical effects or advantages:
[0031] 1. Input the training drone image into the first twin network of the initial twin network model to obtain the embedded vector features of the training drone image; input the training satellite image into the second twin network of the initial twin network model to obtain the embedded vector features of the training satellite image; input the embedded vector features of the training drone image and the embedded vector features of the training satellite image into the SDCL loss function
[0032] In the process, the loss between the two is calculated, and training is performed through gradient backpropagation. After the training, the optimal twin network model is obtained; the real-time drone image is input into the optimal twin network model to obtain the matching satellite image. The present invention introduces a symmetric decoupled contrastive learning loss (SDCL), which learns effective representations by maximizing the similarity between positive samples and minimizing the similarity between negative samples, effectively alleviating the problem of imbalance between positive and negative samples, greatly improving the network's learning ability for positive and negative samples, and improving the matching effect of cross-view geolocation.
[0033] 2. The weights of the two sub-networks of the twin network are shared, which can greatly reduce the number of parameters of the twin network. Therefore, there is no need for too much computing cost, and the inference speed is faster. It is very suitable for edge computing devices with limited computing power such as drones and satellites. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 A flowchart of a cross-view geolocation method provided by an embodiment of the present invention;
[0035] Figure 2 A schematic diagram of a cross-view geolocation method provided by an embodiment of the present invention;
[0036] Figure 3 This is an enhanced effect diagram of a drone image and a satellite image in an embodiment of the present invention;
[0037] Figure 4 A cross-view geolocation result diagram of the cross-view geolocation method provided by an embodiment of the present invention;
[0038] Figure 5 A module diagram of a cross-view geo-positioning device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0039] The embodiments of the present invention solve the technical problem of poor matching effect of cross-view geolocation in the prior art by providing a cross-view geolocation method and device.
[0040] The technical solution in the embodiment of the present invention is to solve the above technical problems, and the overall idea is as follows:
[0041] Obtain drone images and satellite images, preprocess the original drone image and satellite image data sets to obtain diversified drone images and satellite images, and use the cross-validation method to divide them into two parts: training set and test set. Use drone images as queries and satellite images as retrieval libraries. Perform image enhancement operations on the training set data, extract features from drone images and satellite images through weight-sharing twin neural networks, and obtain drone image features and satellite image features. Calculate the loss between the two through comparative learning loss, and obtain the optimal model through back-propagation training. Obtain drone-view images in real time and send them to the trained optimal model for image retrieval to obtain the best matching satellite image, and use the geolocation information on the satellite image to complete the geolocation of the drone.
[0042] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0043] like Figure 1 and Figure 2 As shown, the cross-view geolocation method provided by the embodiment of the present invention includes:
[0044] Step S110: input the training drone image into the first twin sub-network of the initial twin network model to obtain the embedded vector feature of the training drone image;
[0045] In order to ensure the invariance of geometric structures such as buildings and roads in the image, and to solve the problem of insufficient sample diversity caused by insufficient data volume of input drone images and satellite images, and to improve the training accuracy of the model, before inputting the training drone image into the first twin network of the initial twin network model and obtaining the embedded vector features of the training drone image, it also includes:
[0046] Rotate training drone images and training satellite images;
[0047] Specifically, the training UAV image and the training satellite image are horizontally flipped and rotated. During the horizontal flipping and rotation process, it is ensured that the horizontal flipping and rotation directions of the UAV image and the satellite image are consistent, so as to obtain the UAV image and the satellite image after horizontal flipping and rotation.
[0048] Perform grid enhancement on the rotated drone images and satellite images;
[0049] Randomly discard some grids in grid-enhanced drone and satellite images;
[0050] The randomly discarded drone images and satellite images are color-jittered to obtain enhanced training drone images and training satellite images, such as Figure 3 shown.
[0051] Step S120: input the training satellite image into the second twin sub-network of the initial twin network model to obtain the embedded vector feature of the training satellite image;
[0052] Step S130: Input the embedded vector features of the training drone image and the embedded vector features of the training satellite image into the SDCL loss function In the above example, the loss between the two is calculated, and training is performed through gradient backpropagation. After the training, the model parameters with high cross-view geolocation accuracy are obtained, that is, the optimal twin network model is obtained; where q is the embedding vector feature of the training drone image, the so-called query, and g is + is a positive sample matching q, is a negative sample that does not match q, g is the embedding vector feature of the training satellite image, and q + is a positive sample matched by g, is a negative sample that does not match g, G is a collection of embedding vector features of training drone images, called Gallery. The Gallery collection contains a positive sample g that matches the query q + And multiple mismatched negative samples Q is the set of embedding vector features of the training satellite image, called Query, and τ is the temperature parameter, which can be set as a learnable or fixed value as a hyperparameter. By adjusting the temperature parameter τ, the sensitivity of the model to similarity differences can be controlled, and the performance of the model can be further optimized. λ is the weight coefficient of the symmetric term;
[0053] In this embodiment, λ is set to 0.5.
[0054] Step S140: Input the real-time UAV image into the optimal twin network model to obtain a satellite image that matches the input UAV image. Use the geolocation information on the satellite image to complete the geolocation of the UAV, that is, obtain the satellite information of the UAV at this time. The cross-view geolocation result is as follows: Figure 4 shown.
[0055] In this embodiment, the weights of the first twin network and the second twin network in the initial twin network model are shared. The first twin network and the second twin network are DaViT.
[0056] The experimental verification was completed on a Linux server, and the server operating system used was Ubuntu 22.04.2. In terms of hardware environment, the CPU is Intel Core i7-12700KF, and the GPU is NVIDIA GeForce RTX 3090 (24GB); in terms of software environment, the Python version is 3.9.13, the CUDA version is 11.7, and it is implemented based on the PyTorch 2.0.0 deep learning framework.
[0057] Table 1. Comparative test results on the University-1652 dataset
[0058]
[0059] Table 2. Comparison of inference time on the University-1652 validation set
[0060]
[0061] As shown in Table 1, the method of the embodiment of the present invention achieves the most advanced results with the least parameters. Compared with the best CNN-based method PCL, this method considers global feature extraction through the DaViT structure, and the effect is better than the CNN-based method. Compared with the transformer-based method FSRA that previously achieved the best performance, at a resolution of 256x256, the tiny version of this method improves R@1 by 6.97 and AP by 6.22. The reason is that the SDCL loss proposed in the embodiment of the present invention can well solve the problem of unbalanced samples of drone images and satellite images, making the calculation of loss more reasonable. The embodiment of the present invention also significantly reduces the parameters of the network through a parameter sharing strategy, while achieving better results.
[0062] As shown in Table 2, the model based on contrastive loss has a relatively long inference time due to the large size and large number of parameters of VGG-16. In contrast, the triple loss and LPN models use the ResNet-50 structure, and their inference speed is faster than that of the VGG-16-based model due to the reduction in the number of parameters. The backbone network of the FSRA model adopts the ViT-S structure. Although the model has fewer parameters, its inference speed is significantly reduced due to its high computational complexity. The method of the embodiment of the present invention is based on a weight sharing strategy, which eliminates the need for the twin network to perform model initialization twice, thereby significantly improving the inference speed while reducing the model parameters. The tiny model inference speed reaches an astonishing 1min 13s, which is nearly 5 times faster than FSRA. This advantage is particularly important for edge computing devices such as drones and satellites.
[0063] like Figure 5As shown, the cross-view geographic positioning device provided by the embodiment of the present invention includes:
[0064] The UAV image embedding vector feature acquisition module 100 is used to input the training UAV image into the first twin sub-network of the initial twin network model to obtain the embedding vector feature of the training UAV image;
[0065] The satellite image embedding vector feature acquisition module 200 is used to input the training satellite image into the second twin sub-network of the initial twin network model to obtain the embedding vector feature of the training satellite image;
[0066] In order to ensure the invariance of geometric structures such as buildings and roads in the image, and solve the problem of insufficient sample diversity caused by insufficient input drone image and satellite image data, and improve the training accuracy of the model, it also includes:
[0067] The rotation module is used to rotate the training UAV image and the training satellite image; specifically, the training UAV image and the training satellite image are horizontally flipped and rotated. During the horizontal flipping and rotation process, it is ensured that the horizontal flipping and rotation directions of the UAV image and the satellite image are consistent, so as to obtain the UAV image and the satellite image after horizontal flipping and rotation.
[0068] The grid enhancement module is used to perform grid enhancement on the rotated UAV images and satellite images;
[0069] A random discarding module is used to randomly discard some grids in the grid-enhanced drone images and satellite images;
[0070] The color jittering module is used to perform color jitter on the randomly discarded UAV images and satellite images to obtain enhanced training UAV images and training satellite images.
[0071] Model training module 300, for inputting the embedded vector features of the training drone image and the embedded vector features of the training satellite image into the SDCL loss function
[0072] In the above example, the loss between the two is calculated, and training is performed through gradient backpropagation. After the training, the model parameters with high cross-view geolocation accuracy are obtained, that is, the optimal twin network model is obtained; where q is the embedding vector feature of the training drone image, the so-called query, and g is + is a positive sample matching q, is a negative sample that does not match q, g is the embedding vector feature of the training satellite image, and q + is a positive sample matched by g, is a negative sample that does not match g, G is a collection of embedding vector features of training drone images, called Gallery. The Gallery collection contains a positive sample g that matches the query q + And multiple mismatched negative samples Q is the set of embedding vector features of the training satellite image, called Query, and τ is the temperature parameter, which can be set as a learnable or fixed value as a hyperparameter. By adjusting the temperature parameter τ, the sensitivity of the model to similarity differences can be controlled, and the performance of the model can be further optimized. λ is the weight coefficient of the symmetric term;
[0073] In this embodiment, λ is set to 0.5.
[0074] The cross-view geolocation module 400 is used to input the real-time drone image into the optimal twin network model to obtain a satellite image that matches the input drone image, and use the geolocation information on the satellite image to complete the geolocation of the drone, that is, to obtain the satellite information of the drone at this time.
[0075] In this embodiment, the weights of the first twin network and the second twin network in the initial twin network model are shared. The first twin network and the second twin network are DaViT.
[0076] In summary, the embodiments of the present invention adopt a similar symmetric strategy to optimize the decoupled contrastive learning loss function (SDCL) to ensure that information flow can be fully utilized in both directions, that is, the transmission from drone images to satellite images and from satellite images to drone images, which solves the problem of sample imbalance between drone images and satellite images and improves the matching effect of cross-view geolocation. Compared with the loss function used in the existing method, the embodiments of the present invention overcome the challenge of small positive sample ratio during training and improve the overall performance and robustness of the model.
[0077] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0078] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0079] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0080] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0081] The parts not described in detail in the embodiments of the present invention are all known technologies to those skilled in the art. Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention is described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions of the present invention, which should be included in the scope of the claims of the present invention.
Claims
1. A cross-view geolocation method, characterized in that: include: Inputting the training UAV image into the first Siamese sub-network of the initial Siamese network model to obtain an embedded vector feature of the training UAV image; Inputting the training satellite image into the second twin sub-network of the initial twin network model to obtain the embedded vector feature of the training satellite image; The embedded vector features of the training drone image and the embedded vector features of the training satellite image are input into the SDCL loss function , the loss between the two is calculated, and training is performed through gradient backpropagation. After the training, the optimal twin network model is obtained; wherein q is the embedding vector feature of the training drone image, g is the embedding vector feature of the training satellite image, G is the set of embedding vector features of the training drone image, Q is the set of embedding vector features of the training satellite image, and g + is a positive sample matching the q, is a negative sample that does not match the q, q + is the positive sample matched by g, is a negative sample that does not match the g, τ is a temperature parameter, and λ is a weight coefficient of the symmetric term; The real-time UAV image is input into the optimal twin network model to obtain the matching satellite image.
2. The cross-view geolocation method according to claim 1, characterized in that: Before inputting the training drone image into the first twin sub-network of the initial twin network model to obtain the embedded vector feature of the training drone image, the method further includes: Rotating the training UAV image and the training satellite image; Perform grid enhancement on the rotated drone images and satellite images; Randomly discard some grids in grid-enhanced drone and satellite images; Color jittering is performed on the randomly discarded UAV images and satellite images to obtain enhanced training UAV images and training satellite images.
3. The cross-view geolocation method according to claim 1, characterized in that: The weights of the first twin network and the second twin network in the initial twin network model are shared.
4. The cross-view geolocation method according to claim 1, characterized in that: The first twin sub-network and the second twin sub-network in the initial twin network model are DaViT.
5. A cross-view geographic positioning device, characterized in that: include: A drone image embedding vector feature acquisition module is used to input the training drone image into the first twin sub-network of the initial twin network model to obtain the embedding vector feature of the training drone image; A satellite image embedding vector feature acquisition module, used to input the training satellite image into the second twin sub-network of the initial twin network model to obtain the embedding vector feature of the training satellite image; A model training module is used to input the embedded vector features of the training drone image and the embedded vector features of the training satellite image into the SDCL loss function , the loss between the two is calculated, and training is performed through gradient backpropagation. After the training, the optimal twin network model is obtained; wherein q is the embedding vector feature of the training drone image, g is the embedding vector feature of the training satellite image, G is the set of embedding vector features of the training drone image, Q is the set of embedding vector features of the training satellite image, and g + is a positive sample matching the q, is a negative sample that does not match the q, q + is the positive sample matched by g, is a negative sample that does not match the g, τ is a temperature parameter, and λ is a weight coefficient of the symmetric term; The cross-view geolocation module is used to input real-time drone images into the optimal twin network model to obtain matching satellite images.
6. The cross-view geolocation device according to claim 5, characterized in that: Also includes: A rotation module, used to rotate the training UAV image and the training satellite image; The grid enhancement module is used to perform grid enhancement on the rotated UAV images and satellite images; A random discarding module is used to randomly discard some grids in the grid-enhanced drone images and satellite images; The color jittering module is used to perform color jitter on the randomly discarded drone images and satellite images to obtain enhanced training drone images and training satellite images.
7. The cross-view geolocation device according to claim 5, characterized in that: The weights of the first twin network and the second twin network in the initial twin network model are shared.
8. The cross-view geo-positioning device according to claim 5, characterized in that: The first twin sub-network and the second twin sub-network in the initial twin network model are DaViT.
Citation Information
Patent Citations
Cross-view-angle geographic positioning method based on unmanned aerial vehicle-satellite
CN113361508A
Cross-view geographic positioning method based on optimal transmission theory
CN114926827A
Visual target tracking method and device based on twin network
CN115588030A
High-resolution remote sensing image land coverage classification method and device and storage medium
CN117036936A
Unmanned aerial vehicle cross-view positioning navigation method and device based on semi-supervised feature enhancement
CN117746231A
