Cross-viewing image geolocation method and device under direction misalignment conditions

By randomly offsetting and generative network processing of ground panoramic images, combined with convolutional neural network and feature aggregation structure, geographic positioning of cross-view angle images with unaligned directions is achieved, and the direction of ground panoramic images must be aligned in the prior art. Restrictions improve the robustness and accuracy of positioning.

CN118674785BActive Publication Date: 2024-12-03AEROSPACE INFORMATION RES INST CAS
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202411132754.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-19
Publication Date
2024-12-03
Estimated Expiration
2044-08-19

AI Technical Summary

Technical Problem

In the existing cross-view image geolocation method, the directions of the ground panoramic image and the remote sensing image must be aligned, which limits the application scenarios of geolocation, and is difficult to achieve accurate positioning especially when the direction of the ground image is unknown in actual applications.

Method used

By randomly offsetting the ground panoramic image within 360 degrees, multiple offset ground panoramic images are generated, and input them with the original ground panoramic image to a pre-trained generative network model to generate a simulated polar coordinate transformed remote sensing image , combined with convolutional neural network and feature aggregation structure, feature extraction and matching are performed to achieve geolocation of images with misaligned orientations.

Benefits of technology

It improves the robustness of the model to input in different directions, narrows the difference in view angles, and can accurately geo-locate the image when the ground image direction is not aligned, overcomes the direction alignment limitations, and improves the reliability of the positioning results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118674785B_ABST
    Figure CN118674785B_ABST
Patent Text Reader

Abstract

The present invention provides a cross-view image geolocation method and device under the condition of direction misalignment, which is applied to the field of image data processing technology. The method comprises: inputting a ground panoramic image and a target number of offset ground panoramic images into a pre-trained generative network model respectively to obtain a simulated polar coordinate transformed remote sensing image output by the pre-trained generative network model; performing polar coordinate transformation on a remote sensing image sample set to obtain a polar coordinate transformed remote sensing image data set; calling a preset convolutional neural network to perform feature extraction to obtain a first feature vector set and a second feature vector set; calling a preset feature aggregation structure to perform feature aggregation to obtain a first output set and a second output set; performing feature matching on the first output set and the second output set to obtain a sorting result for image geolocation; the present invention can better grasp the spatial position relationship of the objects contained in the image at different perspectives.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and in particular to a cross-viewing angle image geolocation method and device under the condition of direction misalignment. Background Art

[0002] Cross-view image geolocation is a method of geolocation based on multi-source images, which can provide technical support for subsequent tasks such as robot navigation, autonomous driving and 3D reconstruction.

[0003] In most existing cross-view geolocation datasets, the directions of ground panoramic images and remote sensing images are known and aligned. For ground panoramic images, the direction indicated by the center of the image is the true north direction, which corresponds to the direction of the remote sensing image after polar coordinate transformation. Based on the existing cross-view geolocation datasets, most of the current cross-view geolocation research is conducted under the assumption that the image directions are known and aligned, treating the geolocation task as a pure position estimation problem while ignoring the direction alignment.

[0004] It can be seen that the cross-view image geolocation method in the related art has the limitation that the directions of the ground panoramic images must be aligned. Summary of the invention

[0005] The present invention provides a cross-view image geolocation method and device under the condition of misaligned directions, so as to solve the limitation that the directions of ground panoramic images and remote sensing images must be aligned in the cross-view image geolocation method in the related art, and realize geolocation under the condition that the directions of ground images and remote sensing images are not aligned.

[0006] The present invention provides a cross-view image geolocation method under the condition of misaligned direction, comprising the following steps. Performing random offsets of different degrees within a 360-degree range of a target number of times on a ground panoramic image, generating a target number of offset ground panoramic images corresponding to the target number, wherein the direction of the ground panoramic image is an unknown direction, and the viewing angle of the ground panoramic image is 360 degrees; inputting the ground panoramic image and the target number of offset ground panoramic images into a pre-trained generative network model respectively, to obtain a simulated polar coordinate transformed remote sensing image output by the pre-trained generative network model, wherein the number of the simulated polar coordinate transformed remote sensing images is the sum of the target number and a constant 1; performing polar coordinate transformation on each remote sensing image in a remote sensing image sample set, to obtain a polar coordinate transformed remote sensing image data set; Call a preset convolutional neural network to perform feature extraction on the simulated remote sensing image after polar coordinate transformation and the remote sensing image data set after polar coordinate transformation, respectively, to obtain a first feature vector set and a second feature vector set; call a preset feature aggregation structure to perform feature aggregation on the first feature vector set and the second feature vector set, respectively, to obtain a first output set and a second output set, wherein the preset feature aggregation structure has spatial perception capability; perform feature matching on the first output set and the second output set to obtain a sorting result, wherein the number of the sorting results is the sum of the target number and a constant 1; make a decision based on the sorting result to obtain a final sorting result for image geolocation.

[0007] According to a cross-view image geolocation method under direction misalignment conditions provided by the present invention, polar coordinate transformation is performed on each remote sensing image in a remote sensing image sample set to obtain a remote sensing image data set after polar coordinate transformation, including: for each pixel point coordinate of each remote sensing image in the remote sensing image sample set, polar coordinate transformation is performed with the center pixel as the center to obtain the remote sensing image data set after polar coordinate transformation.

[0008] According to a cross-view image geolocation method under directional misalignment conditions provided by the present invention, the ground panoramic image is randomly offset to different degrees within a 360-degree range for a target number of times to generate a target number of offset ground panoramic images corresponding to the target number, including: obtaining a ground panoramic image; randomly obtaining the pixel coordinates of the starting position of the ground panoramic image through a preset random function; cropping with the horizontal coordinates of the pixel coordinates of the starting position as the axis of symmetry to obtain a first partial image and a second partial image; and obtaining a offset ground panoramic image after splicing the first partial image to the second partial image.

[0009] According to a cross-view image geolocation method under direction misalignment conditions provided by the present invention, in the step of respectively inputting the ground panoramic image and the offset ground panoramic image of the target number into a pre-trained generative network model, the method further includes: constructing a generative network based on a preset objective function and a preset loss function, the generative network including a generative network and a discriminative network; obtaining a ground panoramic image training set and a remote sensing image training set after polar coordinate transformation; converting the ground panoramic image training set into a simulated remote sensing image training set after polar coordinate transformation based on the generative network; discriminating the input remote sensing image according to the preset objective function based on the discriminative network, determining a first probability that the input remote sensing image belongs to the remote sensing image training set after polar coordinate transformation and a second probability that the input remote sensing image belongs to the remote sensing image training set after polar coordinate transformation; updating the generative network based on the first probability, the second probability and the preset loss function to obtain a pre-trained generative network model.

[0010] According to a cross-view image geolocation method under direction misalignment conditions provided by the present invention, the preset feature aggregation structure includes multiple position embedding modules, wherein each position embedding module in the multiple position embedding modules has the same architecture and different weights; the position embedding module includes channel-directional maximum pooling and a spatial-aware importance generator.

[0011] According to a cross-view image geolocation method under direction misalignment conditions provided by the present invention, making a decision based on the sorting result to obtain a final sorting result includes: determining the Euclidean distance between each first output feature vector in the first output set and each second output feature vector in the second output set, wherein the number of the Euclidean distances is the number of remote sensing images after the simulated polar coordinate transformation; and sorting based on the Euclidean distances to obtain a sorting result.

[0012] The present invention also provides a cross-view image geo-positioning device under the condition of direction misalignment, comprising the following modules: a generation module, used to perform random offsets of different degrees within a 360-degree range of a target number of times on a ground panoramic image, and generate a target number of offset ground panoramic images corresponding to the target number, wherein the direction of the ground panoramic image is an unknown direction, and the viewing angle of the ground panoramic image is 360 degrees; an output module, used to input the ground panoramic image and the target number of offset ground panoramic images into a pre-trained generative network model respectively, to obtain a simulated polar coordinate transformed remote sensing image output by the pre-trained generative network model, wherein the number of the simulated polar coordinate transformed remote sensing images is the sum of the target number and a constant 1; a transformation module, used to perform polar coordinate transformation on each remote sensing image in a remote sensing image sample set, to obtain A remote sensing image data set after polar coordinate transformation; an extraction module, used to call a preset convolutional neural network to perform feature extraction on the simulated remote sensing image after polar coordinate transformation and the remote sensing image data set after polar coordinate transformation, respectively, to obtain a first feature vector set and a second feature vector set, respectively; an aggregation module, used to call a preset feature aggregation structure to perform feature aggregation on the first feature vector set and the second feature vector set, respectively, to obtain a first output set and a second output set, wherein the preset feature aggregation structure has spatial perception capability; a matching module, used to perform feature matching on the first output set and the second output set, to obtain a sorting result, wherein the number of the sorting results is the sum of the target number and a constant 1; a decision module, used to make a decision based on the sorting result, to obtain a final sorting result for image geolocation.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a cross-viewing image geolocation method under any of the above-mentioned direction misalignment conditions is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a cross-viewing image geolocation method under any of the above-mentioned direction misalignment conditions.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the cross-viewing image geolocation method under any of the above-mentioned direction misalignment conditions is implemented.

[0016] The cross-perspective image geolocation method under the condition of misaligned direction provided by the present invention performs multiple random direction offsets on the input ground panoramic image, and inputs the ground panoramic image and the offset ground panoramic image into the pre-trained generative network model, thereby improving the robustness of the model to inputs in different directions; a total of two operations are performed to reduce the difference between the two perspective images: the first operation is to perform polar coordinate transformation on each remote sensing image in the remote sensing image sample set to obtain a remote sensing image data set after polar coordinate transformation, and the second operation is to output the simulated polar coordinate transformation remote sensing image output by the pre-trained generative network model. The two reductions in perspective differences are helpful for the subsequent model to match images with excessive perspective differences; the feature aggregation structure with spatial perception capability is used to embed the spatial position information of the image, which can better grasp the spatial position relationship of the objects contained in the image at different perspectives. This solves the problem that the cross-perspective image geolocation method in the related art has the limitation that the direction of the ground panoramic image must be aligned. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 The invention provides a cross-viewing image geolocation method under the condition of direction misalignment.

[0019] Figure 2 The present invention provides a cross-view image geolocation flow chart for directionally misaligned input ground images.

[0020] Figure 3 It is a schematic diagram of the structure of the generative adversarial network provided by the present invention.

[0021] Figure 4 It is a schematic diagram of a feature aggregation structure with spatial perception capability provided by the present invention.

[0022] Figure 5 It is a flowchart of the ranking output comprehensive decision provided by the present invention.

[0023] Figure 6 It is a structural schematic diagram of a cross-viewing angle image geolocation device under a direction misalignment condition provided by the present invention.

[0024] Figure 7 It is a schematic diagram of the physical structure of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] Cross-view image geolocation is a method of geolocation based on multi-source images, which can provide technical support for subsequent tasks such as robot navigation, autonomous driving and 3D reconstruction. The goal of cross-view image geolocation is to find the most relevant image or a region of an image (usually a remote sensing image) from another perspective from a large-scale candidate dataset for a given query image (usually a ground image) taken from one perspective.

[0027] Since the candidate images in the dataset, especially the bird's-eye view images from air and space platforms, are pre-annotated with geotags, the geolocation of the query image or the sensor that took the image can be determined based on the geotags of the paired images. With the popularity of drones, satellites, and other aerial photography equipment, we can obtain high-quality image data from multiple perspectives; at the same time, the rapid development of the Internet has also provided a platform for the sharing of rich ground images, providing a data basis for cross-perspective image geolocation.

[0028] In addition, powerful computing resources make it possible to process and analyze large amounts of image data, support the implementation of complex image processing and machine learning algorithms, and provide hardware support for cross-view image geolocation research.

[0029] The data sets for this technology are ground panoramic images and satellite images. Compared with drone images, the perspectives of ground panoramic images and remote sensing images are more different, so the matching and positioning process is more challenging; in addition, since ground images are easier to obtain than drone images, the research on ground-satellite cross-perspective geolocation methods is more valuable.

[0030] In most existing cross-view geolocation datasets, the directions of ground panoramic images and remote sensing images are known and aligned. For ground panoramic images, the direction indicated by the center of the image is the true north direction, which corresponds to the direction of the remote sensing image after polar coordinate transformation. Based on the existing cross-view geolocation datasets, most of the current cross-view geolocation research is conducted under the assumption that the image directions are aligned, treating the geolocation task as a pure position estimation problem while ignoring the direction alignment.

[0031] In actual application scenarios, although the direction of remote sensing images is usually unified, the direction of ground images is often unknown. On the one hand, if the queried ground image comes from the user's on-site shooting, it is difficult to ensure that the image is completely aligned to the north during the shooting process of the ground image; on the other hand, if the queried ground image comes from the Internet, it generally does not carry the shooting direction information. Even if it carries the direction information, it may be inaccurate or wrong. Therefore, assuming that the direction of the ground image is known is not valid in many cases. In view of the fact that most of the ground panoramic images obtained in actual applications are unknown in direction, geolocation of cross-view images with misaligned directions has become a new research direction.

[0032] An embodiment of the present invention proposes a cross-view image geolocation method for directionally misaligned input ground images. The method comprehensively utilizes the spectral, spatial relationship and other characteristics of ground panoramic images and remote sensing images, and at the same time uses statistical methods to improve the reliability of positioning results.

[0033] refer to Figure 1 , Figure 1 The invention provides a cross-viewing image geolocation method under the condition of direction misalignment, which specifically includes the following steps.

[0034] Step 101, randomly shifting the ground panoramic image at different degrees within a range of 360 degrees for a target number of times, generating a target number of shifted ground panoramic images corresponding to the target number of times, wherein the direction of the ground panoramic image is an unknown direction and the viewing angle of the ground panoramic image is 360 degrees.

[0035] In the embodiment of the present invention, an input ground panoramic image with unknown viewing angle is subjected to n random shifts of different degrees to generate n ground panoramic images with unknown directions, and the viewing angle is kept unchanged at 360° during the transformation process.

[0036] Through the embodiments of the present invention, the input ground image is subjected to multiple random directional shifts, which are subsequently input into the model to obtain multiple groups of sorting results, and a sorting aggregation model is used to comprehensively output different sorting results to improve the robustness of the model to inputs in different directions, thereby enabling geographic positioning under conditions where the ground image direction is not aligned.

[0037] Step 102, input the ground panoramic image and the target number of offset ground panoramic images into the pre-trained generative network model respectively, and obtain the simulated polar coordinate transformed remote sensing image output by the pre-trained generative network model, wherein the number of simulated polar coordinate transformed remote sensing images is the sum of the target number and a constant 1.

[0038] In an embodiment of the present invention, an original input ground panoramic image and n ground panoramic images generated after random offset are respectively input into a pre-trained generative network model to generate a total of n+1 simulated remote sensing images after polar coordinate transformation.

[0039] Step 103, performing polar coordinate transformation on each remote sensing image in the remote sensing image sample set to obtain a remote sensing image data set after polar coordinate transformation.

[0040] In the embodiment of the present invention, each remote sensing image in the remote sensing image sample set is transformed by a polar coordinate transformation method to generate a remote sensing image after polar coordinate transformation, and construct a remote sensing image data set after polar coordinate transformation.

[0041] Step 104, calling a preset convolutional neural network to perform feature extraction on the simulated remote sensing image after polar coordinate transformation and the remote sensing image data set after polar coordinate transformation, respectively, to obtain a first feature vector set and a second feature vector set.

[0042] In the embodiment of the present invention, the simulated polar coordinate transformed remote sensing image obtained in step 102 and the polar coordinate transformed remote sensing image data set obtained in step 103 are respectively input into a preset convolutional neural network.

[0043] Step 105 , calling a preset feature aggregation structure to perform feature aggregation on the first feature vector set and the second feature vector set respectively to obtain a first output set and a second output set, wherein the preset feature aggregation structure has spatial perception capability.

[0044] In an embodiment of the present invention, a feature aggregation structure with spatial perception capability is used to aggregate the features extracted by the convolutional neural network in step 104.

[0045] Step 106: perform feature matching on the first output set and the second output set to obtain a sorting result, wherein the number of the sorting results is the sum of the target number and a constant 1.

[0046] In the embodiment of the present invention, feature matching is performed on the output features of the two branches in the feature aggregation structure with spatial perception in step 105, and a total of n+1 sorting results are output.

[0047] Step 107 , making a decision based on the sorting result to obtain a final sorting result for image geolocation.

[0048] In the embodiment of the present invention, a ranking aggregation technology is used to make a comprehensive decision on the n+1 ranking outputs in step 106 and output a final ranking result.

[0049] Through the embodiment of the present invention, the final overall ranking result is output according to the comprehensive sorting result, so as to match the actual remote sensing image closest to the ground panoramic image according to the overall ranking result.

[0050] refer to Figure 2 , Figure 2 The present invention provides a flow chart of cross-view image geolocation for directionally misaligned input ground images, which specifically includes the following steps.

[0051] Obtain a ground panoramic image, perform random offset on the ground panoramic image, and obtain ground panoramic images in different directions.

[0052] Acquire a remote sensing image, perform polar coordinate transformation on the remote sensing image, and obtain a remote sensing image after polar coordinate transformation.

[0053] The generative network is trained based on ground panoramic images in different directions and remote sensing images after polar coordinate transformation.

[0054] The ground panoramic images in different directions are input into the (pre-trained) generative network to obtain the simulated polar coordinate transformed remote sensing images generated by the generative network.

[0055] The simulated remote sensing image after polar coordinate transformation and the remote sensing image after polar coordinate transformation are respectively input into the preset weight-sharing convolutional neural network for feature extraction; then, they are respectively input into the feature aggregation module (equivalent to the feature aggregation structure) with spatial perception ability for feature aggregation and output.

[0056] The output features of the two branches of the feature aggregation module of the spatial perception capability are matched, and a plurality of ranking results are output.

[0057] Sort multiple sorting results based on a preset sorting aggregation model to obtain a comprehensive sorting result.

[0058] Through the above steps of the embodiment of the present invention, the ground panoramic image is randomly offset to different degrees within a 360-degree range of a target number of times, and a target number of offset ground panoramic images corresponding to the target number are generated, wherein the direction of the ground panoramic image is an unknown direction, and the viewing angle of the ground panoramic image is 360 degrees; the ground panoramic image and the offset ground panoramic image of the target number are respectively input into the pre-trained generative network model to obtain the simulated polar coordinate transformed remote sensing image output by the pre-trained generative network model, wherein the number of the simulated polar coordinate transformed remote sensing images is the sum of the target number and a constant 1; polar coordinate transformation is performed on each remote sensing image in the remote sensing image sample set to obtain the polar coordinate transformed remote sensing image. Remote sensing image data set; calling a preset convolutional neural network to extract features from the simulated remote sensing image after polar coordinate transformation and the remote sensing image data set after polar coordinate transformation, respectively, to obtain a first feature vector set and a second feature vector set; calling a preset feature aggregation structure to perform feature aggregation on the first feature vector set and the second feature vector set, respectively, to obtain a first output set and a second output set, wherein the preset feature aggregation structure has spatial perception capability; performing feature matching on the first output set and the second output set to obtain a sorting result, wherein the number of sorting results is the sum of the target number and a constant 1; making a decision based on the sorting result to obtain a final sorting result for image geolocation. It can better grasp the spatial position relationship of the objects contained in the image under different perspectives. This solves the limitation that the directions of the ground panoramic images must be aligned in the cross-perspective image geolocation method in the related art.

[0059] According to a cross-view image geolocation method under direction misalignment conditions provided by the present invention, polar coordinate transformation is performed on each remote sensing image in a remote sensing image sample set to obtain a remote sensing image data set after polar coordinate transformation, including the following steps.

[0060] For each pixel point coordinate of each remote sensing image in the remote sensing image sample set, polar coordinate transformation is performed with the central pixel as the center to obtain a remote sensing image dataset after polar coordinate transformation.

[0061] In an embodiment of the present invention, each remote sensing image in the remote sensing image sample set is transformed by a polar coordinate transformation method to generate a remote sensing image after polar coordinate transformation, and a remote sensing image data set after polar coordinate transformation is constructed, and the steps are as follows.

[0062] For the pixel coordinates on a remote sensing image , the pixel coordinates after polar coordinate transformation Please refer to the following formula (1).

[0063] (1)

[0064] in, Represents the horizontal coordinate of the pixel point on the remote sensing image. Represents the width of the remote sensing image, Represents the ordinate of the pixel after polar coordinate transformation, Indicates the height of the image after polar coordinate transformation. Indicates the width of the image after polar coordinate transformation. Represents the horizontal coordinate of the pixel after polar coordinate transformation; Represents the vertical coordinate of the pixel point on the remote sensing image. Represents the height of the remote sensing image, represents the sine function, Represents the cosine function.

[0065] The scales of polar coordinate transformation are respectively × and × express.

[0066] By performing polar coordinate transformation on each remote sensing image in a remote sensing image sample set through the embodiment of the present invention, the viewing angle difference between different remote sensing images can be effectively reduced, which helps subsequent models to match images with excessive viewing angle difference.

[0067] According to a cross-viewing image geolocation method under direction misalignment conditions provided by the present invention, the ground panoramic image is randomly offset to different degrees within a 360-degree range for a target number of times to generate a target number of offset ground panoramic images corresponding to the target number, including the following steps.

[0068] Get ground panoramic images.

[0069] The pixel coordinates of the starting position of the ground panoramic image are randomly obtained through a preset random function.

[0070] The horizontal coordinate of the pixel coordinate of the starting position is used as a symmetry axis for cropping to obtain a first partial image and a second partial image.

[0071] After the first partial image is stitched with the second partial image, an offset ground panoramic image is obtained.

[0072] In an embodiment of the present invention, an input ground panoramic image with an unknown viewing angle is subjected to n random shifts of different degrees to generate n ground panoramic images with unknown directions, and the viewing angle is kept unchanged at 360° during the transformation process. The steps are as follows.

[0073] Step 1: Given a ground panoramic image with unknown direction I g, use the random function to randomly obtain the pixel coordinates of the starting position of the image after the offset (x p , 0), with x=x p The ground panoramic image is cropped as the axis, and the first part of the cropped image is spliced ​​behind the second part of the image.

[0074] Step 2: Perform multiple random shifts on the same input image to establish a set of ground panoramic images with different directions at the same geographical location.

[0075] Through the embodiments of the present invention, a set of ground panoramic images of the same geographical location but in different directions can be established by randomly shifting the same input image multiple times.

[0076] According to a cross-view image geolocation method under direction misalignment conditions provided by the present invention, before the ground panoramic image and the target number of offset ground panoramic images are respectively input into a pre-trained generative network model, the method further includes the following steps.

[0077] Based on the preset objective function and the preset loss function, a generative network is constructed. The generative network includes a generative network and a discriminative network.

[0078] Obtain the ground panoramic image training set and the remote sensing image training set after polar coordinate transformation.

[0079] Based on the generative network, the ground panoramic image training set is converted into a remote sensing image training set after simulated polar coordinate transformation.

[0080] Based on the discriminant network, the input remote sensing image is discriminated according to the preset objective function to determine the first probability that the input remote sensing image belongs to the remote sensing image training set after polar coordinate transformation and the second probability that the input remote sensing image belongs to the remote sensing image training set after simulated polar coordinate transformation.

[0081] The generative network is updated based on the first probability, the second probability and a preset loss function to obtain a pre-trained generative network model.

[0082] refer to Figure 3 , Figure 3 It is a structural schematic diagram of the generative adversarial network provided by the present invention, which includes the following specific steps: inputting random noise and ground panoramic images into the generative network respectively; the generative network outputs the simulated polar coordinate transformed remote sensing image; based on the simulated polar coordinate transformed remote sensing image and the real polar coordinate transformed remote sensing image, the remote sensing image is input into the discrimination network, and the discrimination network determines whether the input image is true data or false data.

[0083] In an embodiment of the present invention, the original input of one ground panoramic image and n ground panoramic images generated after random offset are respectively input into the generative network model to generate a total of n+1 simulated remote sensing images after polar coordinate transformation; this implementation adopts a conditional generative adversarial network as the generative network model, and the steps are as follows.

[0084] Step 1: Build a conditional generative adversarial network.

[0085] Among them, the structure of the conditional generation adversarial network is as follows Figure 3 As shown, the adopted loss function can refer to the following formula (2).

[0086] (2)

[0087] in, Representation Generator With discriminator The loss function of .

[0088] Represents the loss on real data. Specifically, it is expected that under given conditions Next, the discriminator Able to correctly convert real data The classification is true (i.e., equivalent to the first probability above, the output is close to 1), represents the distribution of real data, is a true sample drawn from this distribution, When the input is a real sample and constraints When , the discriminator outputs the real data is the true log probability; Represents the loss of generated data. Specifically, it is expected that under given conditions Next, the discriminator Ability to correctly generate data is classified as false (i.e., equivalent to the second probability above, the output is close to 0), represents the distribution of the generator input noise (usually Gaussian or uniform), is a noise vector sampled from this distribution, is the generator according to the noise and constraints Generated fake samples. When the input is a generated sample and conditions When , the discriminator outputs the log probability of being false.

[0089] Here, , , Represent the real samples, random noise and conditional constraints of user input respectively; Indicates that the constraints and real data After all are input into the discriminant network, the real data is obtained is the true probability; Indicates that the discriminant network will constrain the condition and random noise The probability that the simulated data generated after being input into the generative network is judged to be real.

[0090] The objective function used can refer to the following formula (3).

[0091] (3)

[0092] in, Describes the training objectives of GANs, specifically, Indicates that the generator (Generator, with express), Represents a fixed generator Parameters of , only train the discriminator The goal is to maximize the loss function , that is, let the discriminator Identify as accurately as possible which data is real (from the real dataset) and which data is generated (from the generator ).generally, The design of The output is close to 1 for real data (indicating "true"), and close to 0 for generated data (indicating "false").

[0093] Step 2: Training the conditional generative adversarial network. The input of the model is the ground panoramic image and the corresponding remote sensing image after polar coordinate transformation. During the training process of the model, the generative network is responsible for converting the ground panoramic image into the remote sensing image after polar coordinate transformation. The discriminative network is responsible for judging whether the input remote sensing image is real or generated by the generative network.

[0094] Step 3, in the verification phase of the model, the generation network receives the ground panoramic image as input and generates a false polar coordinate transformed remote sensing image.

[0095] Through the embodiments of the present invention, the generative network model is pre-trained to improve the robustness and accuracy of the generative network model.

[0096] According to a cross-view image geolocation method under direction misalignment conditions provided by the present invention, the preset feature aggregation structure includes multiple position embedding modules, wherein each position embedding module in the multiple position embedding modules has the same architecture and different weights; the position embedding module includes channel-directional maximum pooling and a spatial-aware importance generator.

[0097] refer to Figure 4 , Figure 4 It is a schematic diagram of the feature aggregation structure with spatial perception capability provided by the present invention, which includes multiple position embedding modules with the same architecture and different weights, and each position embedding module includes input features, channel direction maximum pooling, spatial perception importance generator, and output features.

[0098] In the embodiment of the present invention, a feature aggregation structure with spatial perception capability is used to aggregate the features extracted by the convolutional neural network. The feature aggregation structure with spatial perception capability is as follows: Figure 4 shown.

[0099] In an embodiment of the present invention, the first feature vector set and the second feature vector set obtained by feature extraction by a convolutional neural network are input into a feature aggregation structure with spatial perception capability; the feature aggregation structure with spatial perception capability is composed of multiple position embedding modules, each of which has the same architecture but different weights; the position embedding module includes two modules, namely, maximum pooling in the channel direction and a spatial perception importance generator, and finally outputs features containing spatial information. Subsequently, the output features of each position embedding module are aggregated and output.

[0100] Through the embodiment of the present invention, a feature aggregation structure with spatial perception capability is adopted to embed the spatial position information of the image, which can better grasp the spatial position relationship of the objects contained in the image under different viewing angles.

[0101] According to a cross-view image geolocation method under direction misalignment conditions provided by the present invention, a decision is made based on the sorting result to obtain a final sorting result, which includes the following steps.

[0102] The Euclidean distance between each first output feature vector in the first output set and each second output feature vector in the second output set is determined, wherein the number of the Euclidean distances is the number of simulated remote sensing images after polar coordinate transformation.

[0103] Sort based on Euclidean distance to get the sorting result.

[0104] In the embodiment of the present invention, the output features of the two branches in the feature aggregation structure with spatial perception are matched, and a total of n+1 sorting results are output. The steps are as follows:

[0105] Step 1: Calculate the feature vector set of all remote sensing images in the preset image library after feature extraction through polar coordinate transformation, convolutional neural network and feature aggregation architecture with spatial perception ability.

[0106] Step 2: For the first output set and the second output set (i.e., the ground panoramic image feature vectors output by aggregating the output features of the position embedding module), respectively calculate the Euclidean distance of the feature vectors in the remote sensing image feature vector set in step 1 above; for the two feature vectors , , the calculation formula of Euclidean distance can refer to the following formula (4):

[0107] (4)

[0108] in, Represents the feature vector and The Euclidean distance of For each dimension (from 1 to ), calculates the square of the difference between the coordinates of two points in that dimension.

[0109] Step 3, sorting in ascending order according to the distance calculation results to obtain a sorted list;

[0110] Since there are n+1 output ground panoramic image feature vectors, this step outputs n+1 groups of sorted lists.

[0111] refer to Figure 5 , Figure 5 It is a flowchart of the comprehensive decision-making of ranking output provided by the present invention, which includes multiple groups of ranking results, ranking score accumulation (including items and scores), sorting, and final ranking output.

[0112] Step 4: Use ranking aggregation technology to make a comprehensive decision on the n+1 ranking outputs in step 3 above and output the final ranking result. The specific steps are as follows.

[0113] Step 41, for each item in the sorted list, sum its ranking in each sorted list to calculate a cumulative ranking score.

[0114] The above-mentioned items may be the ground panoramic image feature vector (ie, the feature vector in the first output set) obtained by aggregating the output features of the position embedding module and outputting the feature vector.

[0115] Step 42, sort the items in reverse order according to the ranking scores.

[0116] Step 43, output the final overall ranking result.

[0117] Through the embodiments of the present invention, the final overall ranking result can be output according to the comprehensive sorting result, so as to match the remote sensing image closest to the ground panoramic image according to the overall ranking result.

[0118] The following describes embodiments of the present invention in specific application scenarios.

[0119] The remote sensing image used in the present invention can be a satellite remote sensing image with a zoom level of 18-20, and the ground image is a ground panorama. The present invention is described in detail below in conjunction with the accompanying drawings. Figure 2 This is a flow chart of the cross-view image geolocation of the directionally misaligned input ground image of the present invention, and the specific steps include:

[0120] (1) For each remote sensing image in the remote sensing image sample set, transform it by polar coordinate transformation to generate a remote sensing image after polar coordinate transformation and construct a remote sensing image dataset after polar coordinate transformation. The steps are as follows:

[0121] (1.1) For the pixel coordinates on a remote sensing image , the pixel coordinates after polar coordinate transformation It can be calculated using the following formula (5):

[0122] (5)

[0123] in, Represents the horizontal coordinate of the pixel point on the remote sensing image. Represents the width of the remote sensing image, Represents the ordinate of the pixel after polar coordinate transformation, Indicates the height of the image after polar coordinate transformation. Indicates the width of the image after polar coordinate transformation. Represents the horizontal coordinate of the pixel after polar coordinate transformation; Represents the vertical coordinate of the pixel point on the remote sensing image. Represents the height of the remote sensing image, represents the sine function, Represents the cosine function.

[0124] The scales of polar coordinate transformation are respectively × and × express.

[0125] (2) For an input ground panoramic image with unknown viewing angle, perform n random shifts of different degrees to generate n ground panoramic images with unknown directions. During the transformation process, the viewing angle remains unchanged at 360°. The steps are as follows:

[0126] (2.1) Given a ground panoramic image with unknown direction I g , use the random function to randomly obtain the pixel coordinates (x p ,0), crop the image with x=xp as the axis, and splice the first part of the cropped image to the back of the second part of the image;

[0127] (2.2) Perform multiple random shifts on the same input image to create a set of ground panoramic images with different directions at the same geographical location.

[0128] (3) The original input ground panoramic image in step (2) and the n ground panoramic images generated after random offset are respectively input into the generative network model to generate a total of n+1 simulated remote sensing images after polar coordinate transformation. This embodiment adopts a conditional generative adversarial network as the generative network model, and the steps are as follows:

[0129] (3.1) Construct a conditional generative adversarial network. The structure of the conditional generative adversarial network is as follows: Figure 3 As shown, the loss function used is:

[0130] (6)

[0131] in, Representation Generator With discriminator The loss function of .

[0132] Represents the loss on real data. Specifically, it is expected that under given conditions Next, the discriminator Able to correctly convert real data The classification is true (i.e., equivalent to the first probability above, the output is close to 1), represents the distribution of real data, is a true sample drawn from this distribution, When the input is a real sample and constraints When , the discriminator outputs the real data is the true log probability; Represents the loss of generated data. Specifically, it is expected that under given conditions Next, the discriminator Ability to correctly generate data is classified as false (i.e., equivalent to the second probability above, the output is close to 0), represents the distribution of the generator input noise (usually Gaussian or uniform), is a noise vector sampled from this distribution, is the generator according to the noise and constraints Generated fake samples. When the input is a generated sample and conditions When , the discriminator outputs the log probability of being false.

[0133] The objective function used is:

[0134] (7)

[0135] in, Describes the training objectives of GANs, specifically, Indicates that the generator (Generator, with express), Represents a fixed generator Parameters of , only train the discriminator The goal is to maximize the loss function , that is, let the discriminator Identify as accurately as possible which data is real (from the real dataset) and which data is generated (from the generator ).generally, The design of The output is close to 1 for real data (indicating "true"), and close to 0 for generated data (indicating "false").

[0136] in, , , Represent the real samples, random noise and conditional constraints of user input respectively; Indicates that the constraints and real data After all are input into the discriminant network, the real data is obtained is the true probability; Indicates that the discriminant network will constrain the condition and random noise The probability that the simulated data generated after being input into the generative network is judged to be real.

[0137] (3.2) Training conditional generative adversarial network. The input of the model is a ground panoramic image and a corresponding remote sensing image after polar coordinate transformation. During the training process of the model, the generative network is responsible for converting the ground panoramic image into a remote sensing image after polar coordinate transformation. The discriminative network is responsible for judging whether the input remote sensing image is real or generated by the generative network.

[0138] (3.3) In the verification phase of the model, the generation network receives the ground panoramic image in step (2) as input and generates a false polar coordinate transformed remote sensing image.

[0139] (4) Inputting the output result of the generator in the generative network in step (3) and the remote sensing image after polar coordinate transformation in step (1) into the convolutional neural network for feature extraction;

[0140] (5) A feature aggregation structure with spatial perception capability is used to aggregate the features extracted by the convolutional neural network in step (4). The feature aggregation structure with spatial perception capability is as follows: Figure 4 As shown, the steps are as follows:

[0141] (5.1) Input the features in step (4) into a feature aggregation structure with spatial perception capability (feature aggregation module); the feature aggregation structure with spatial perception capability is composed of multiple position embedding modules, each of which has the same architecture but different weights; the position embedding module includes two modules: channel-wise maximum pooling and spatial perception importance generator, and finally outputs features containing spatial information;

[0142] (5.2) Aggregate and output the output features of the embedding modules at each position in step (5.1).

[0143] (6) Perform feature matching on the output features of the two branches in the feature aggregation structure with spatial perception in step (5), and output a total of n+1 sorting results. The steps are as follows:

[0144] (6.1) Calculate the feature vector set of all remote sensing images in the image library after feature extraction through polar coordinate transformation, convolutional neural network and feature aggregation architecture with spatial perception capability;

[0145] (6.2) For the ground panoramic image feature vector output in step (5), calculate the Euclidean distance between it and the feature vector in the remote sensing image feature vector set in step (6.1); for two feature vectors , , the calculation formula of Euclidean distance is as follows:

[0146] (8)

[0147] in, Represents the feature vector and The Euclidean distance of For each dimension (from 1 to ), calculates the square of the difference between the coordinates of two points in that dimension.

[0148] (6.3) Sort the distance calculation results in ascending order to obtain a sorted list. Since there are n+1 ground panoramic image feature vectors output in step (5), this step outputs n+1 groups of sorted lists in total.

[0149] (7) Use the sorting aggregation technology to make a comprehensive decision on the n+1 ranking outputs in step (6) and output the final sorting result. The sorting aggregation process is as follows: Figure 5 As shown, the steps are as follows:

[0150] (7.1) For each item in the sorted list, sum its ranking in each sorted list and calculate the cumulative ranking score;

[0151] (7.2) Sort the items in reverse order according to their ranking scores;

[0152] (7.3) Output the final overall ranking result.

[0153] Through the above-mentioned embodiment provided by the present invention, multiple groups of sorting results are obtained by performing multiple random directional shifts on the input ground image, inputting them into the model together, and using a sorting aggregation model to comprehensively output different sorting results, so as to improve the robustness of the model to inputs in different directions, thereby being able to perform geolocation under conditions where the ground image directions are not aligned. A total of two operations are performed to reduce the difference between the two perspective images: the first is to perform polar coordinate transformation on the remote sensing image, and the second is to use a generative network to convert the ground panoramic image into a simulated remote sensing image after polar coordinate transformation. The two reductions in perspective differences help the model match images with excessive perspective differences (greater than 90 degrees). A feature aggregation structure with spatial perception capability is used to embed the spatial position information of the image, which can better grasp the spatial position relationship of the objects contained in the image at different perspectives.

[0154] The cross-view image geolocation device under direction misalignment conditions provided by the present invention is described below. The cross-view image geolocation device under direction misalignment conditions described below and the cross-view image geolocation method under direction misalignment conditions described above can be referenced to each other.

[0155] refer to Figure 6 , Figure 6 It is a structural schematic diagram of the cross-view image geolocation device under the condition of direction misalignment provided by the present invention, which includes a generation module 601, an output module 602, a transformation module 603, an extraction module 604, an aggregation module 605, a matching module 606 and a decision module 607.

[0156] A generating module 601 is used to perform random shifts of different degrees within a 360-degree range for a target number of times on the ground panoramic image, and generate a target number of shifted ground panoramic images corresponding to the target number of times, wherein the direction of the ground panoramic image is an unknown direction, and the viewing angle of the ground panoramic image is 360 degrees;

[0157] The output module 602 is used to input the ground panoramic image and the offset ground panoramic image of the target number into the pre-trained generative network model respectively, and obtain the simulated polar coordinate transformed remote sensing image output by the pre-trained generative network model, wherein the number of the simulated polar coordinate transformed remote sensing images is the sum of the target number and a constant 1;

[0158] The transformation module 603 is used to perform polar coordinate transformation on each remote sensing image in the remote sensing image sample set to obtain a remote sensing image data set after polar coordinate transformation;

[0159] An extraction module 604 is used to call a preset convolutional neural network to perform feature extraction on the simulated remote sensing image after polar coordinate transformation and the remote sensing image data set after polar coordinate transformation, respectively, to obtain a first feature vector set and a second feature vector set;

[0160] Aggregation module 605, used to call a preset feature aggregation structure to perform feature aggregation on the first feature vector set and the second feature vector set respectively to obtain a first output set and a second output set, wherein the preset feature aggregation structure has spatial perception capability;

[0161] A matching module 606 is used to perform feature matching on the first output set and the second output set to obtain a sorting result, wherein the number of the sorting results is a sum of a target number and a constant 1;

[0162] The decision module 607 is used to make a decision based on the sorting result to obtain a final sorting result for image geolocation.

[0163] Specifically, the cross-view image geolocation device under the above-mentioned direction misalignment conditions provided by the present invention can implement all the method steps implemented in the cross-view image geolocation method embodiment under the above-mentioned direction misalignment conditions, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.

[0164] Figure 7 is a schematic diagram of the physical structure of the electronic device provided by the present invention, such as Figure 7As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the cross-view image geolocation method under the condition of direction misalignment, the method comprising: performing random offsets of different degrees within a 360-degree range of a target number of ground panoramic images to generate a target number of offset ground panoramic images corresponding to the target number of times, wherein the direction of the ground panoramic image is an unknown direction and the viewing angle of the ground panoramic image is 360 degrees; inputting the ground panoramic image and the offset ground panoramic image of the target number into the pre-trained generative network model respectively, obtaining the simulated polar coordinate transformed remote sensing image output by the pre-trained generative network model, wherein the number of simulated polar coordinate transformed remote sensing images is the sum of the target number and a constant 1; for each remote sensing image in the remote sensing image sample set, The method comprises the following steps: performing polar coordinate transformation on the remote sensing image to obtain a remote sensing image data set after polar coordinate transformation; calling a preset convolutional neural network to perform feature extraction on the simulated remote sensing image after polar coordinate transformation and the remote sensing image data set after polar coordinate transformation, respectively obtaining a first feature vector set and a second feature vector set; calling a preset feature aggregation structure to perform feature aggregation on the first feature vector set and the second feature vector set, respectively obtaining a first output set and a second output set, wherein the preset feature aggregation structure has spatial perception capability; performing feature matching on the first output set and the second output set to obtain a sorting result, wherein the number of sorting results is the sum of the target number and a constant 1; making a decision based on the sorting result to obtain a final sorting result for image geolocation.

[0165] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0166] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the cross-view image geolocation method under the condition of direction misalignment provided by the above-mentioned methods, and the method includes: performing random offsets of different degrees within a 360-degree range of a target number of times on the ground panoramic image, generating a target number of offset ground panoramic images corresponding to the target number, wherein the direction of the ground panoramic image is an unknown direction and the viewing angle of the ground panoramic image is 360 degrees; inputting the ground panoramic image and the target number of offset ground panoramic images into a pre-trained generative network model respectively, and obtaining a simulated polar coordinate transformed remote sensing image output by the pre-trained generative network model, wherein the simulated polar coordinate transformed remote sensing image The number of remote sensing images is the sum of the target number and a constant 1; polar coordinate transformation is performed on each remote sensing image in the remote sensing image sample set to obtain a remote sensing image data set after polar coordinate transformation; a preset convolutional neural network is called to perform feature extraction on the simulated remote sensing image after polar coordinate transformation and the remote sensing image data set after polar coordinate transformation, respectively, to obtain a first feature vector set and a second feature vector set; a preset feature aggregation structure is called to perform feature aggregation on the first feature vector set and the second feature vector set, respectively, to obtain a first output set and a second output set, wherein the preset feature aggregation structure has spatial perception capability; feature matching is performed on the first output set and the second output set to obtain a sorting result, wherein the number of sorting results is the sum of the target number and a constant 1; a decision is made based on the sorting result to obtain a final sorting result for image geolocation.

[0167] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the cross-viewing image geolocation method under the condition of direction misalignment provided by the above-mentioned methods, the method comprising: performing random offsets of different degrees within a 360-degree range of a target number of ground panoramic images to generate a target number of offset ground panoramic images corresponding to the target number, wherein the direction of the ground panoramic image is an unknown direction and the viewing angle of the ground panoramic image is 360 degrees; inputting the ground panoramic image and the target number of offset ground panoramic images into a pre-trained generative network model respectively, to obtain a simulated polar coordinate transformed remote sensing image output by the pre-trained generative network model, wherein the number of simulated polar coordinate transformed remote sensing images is the product of the target number and a constant 1. The sum of the values ​​of the vectors is obtained by performing polar coordinate transformation on each remote sensing image in the remote sensing image sample set to obtain a remote sensing image data set after polar coordinate transformation; calling a preset convolutional neural network to perform feature extraction on the simulated remote sensing image after polar coordinate transformation and the remote sensing image data set after polar coordinate transformation to obtain a first feature vector set and a second feature vector set respectively; calling a preset feature aggregation structure to perform feature aggregation on the first feature vector set and the second feature vector set to obtain a first output set and a second output set, wherein the preset feature aggregation structure has spatial perception capability; performing feature matching on the first output set and the second output set to obtain a sorting result, wherein the number of sorting results is the sum of the target number and a constant 1; making a decision based on the sorting result to obtain a final sorting result for image geolocation.

[0168] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0169] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0170] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cross-view image geolocation method under direction misalignment conditions, characterized in that: include: Performing random shifts of different degrees within a 360-degree range for a target number of times on the ground panoramic image, generating a target number of shifted ground panoramic images corresponding to the target number of times, wherein the direction of the ground panoramic image is an unknown direction, and the viewing angle of the ground panoramic image is 360 degrees; Inputting the ground panoramic image and the offset ground panoramic image of the target number into a pre-trained generative network model respectively, obtaining a simulated polar coordinate transformed remote sensing image output by the pre-trained generative network model, wherein the number of the simulated polar coordinate transformed remote sensing images is the sum of the target number and a constant 1; Perform polar coordinate transformation on each remote sensing image in the remote sensing image sample set to obtain a remote sensing image dataset after polar coordinate transformation; Calling a preset convolutional neural network to perform feature extraction on the simulated remote sensing image after polar coordinate transformation and the remote sensing image data set after polar coordinate transformation, respectively, to obtain a first feature vector set and a second feature vector set; Calling a preset feature aggregation structure to perform feature aggregation on the first feature vector set and the second feature vector set respectively to obtain a first output set and a second output set, wherein the preset feature aggregation structure has spatial perception capability; Perform feature matching on the first output set and the second output set to obtain a sorting result, wherein the number of the sorting results is the sum of the target number and a constant 1; Making a decision based on the ranking result to obtain a final ranking result for image geolocation; The step of performing polar coordinate transformation on each remote sensing image in the remote sensing image sample set to obtain a remote sensing image dataset after polar coordinate transformation includes: For the pixel coordinates on the remote sensing image , the pixel coordinates on the remote sensing image after polar coordinate transformation are determined by the following formula : ; Among them, the scales of the polar coordinate transformation are respectively × and × express; in, represents the horizontal coordinate of the pixel point on the remote sensing image, represents the width of the remote sensing image, Represents the vertical coordinate of the pixel after polar coordinate transformation, represents the height of the remote sensing image after polar coordinate transformation, represents the width of the remote sensing image after polar coordinate transformation, Represents the horizontal coordinate of the pixel after polar coordinate transformation; represents the vertical coordinate of the pixel point on the remote sensing image, represents the height of the remote sensing image, represents the sine function, Represents the cosine function.

2. The cross-view image geolocation method under direction misalignment conditions according to claim 1, characterized in that: The step of performing random shifting of the ground panoramic image within a 360-degree range for a target number of times to generate a target number of shifted ground panoramic images corresponding to the target number of times includes: Obtain ground panoramic images; The pixel coordinates of the starting position of the ground panoramic image are randomly obtained by a preset random function; Cutting is performed using the horizontal coordinate of the pixel coordinate of the starting position as a symmetry axis to obtain a first partial image and a second partial image; After the first partial image is stitched with the second partial image, an offset ground panoramic image is obtained.

3. The cross-view image geolocation method under direction misalignment conditions according to claim 1, characterized in that: Before respectively inputting the ground panoramic image and the target number of offset ground panoramic images into a pre-trained generative network model, the method further includes: Based on a preset objective function and a preset loss function, a generative network is constructed, wherein the generative network includes a generative network and a discriminative network; Obtain ground panoramic image training set and remote sensing image training set after polar coordinate transformation; Based on the generative network, the ground panoramic image training set is converted into a remote sensing image training set after simulated polar coordinate transformation; Based on the discrimination network, the input remote sensing image is discriminated according to the preset objective function, and a first probability that the input remote sensing image belongs to the remote sensing image training set after polar coordinate transformation and a second probability that the input remote sensing image belongs to the remote sensing image training set after polar coordinate transformation of the simulation are determined; The generative network is updated based on the first probability, the second probability and the preset loss function to obtain a pre-trained generative network model.

4. The cross-view image geolocation method under direction misalignment conditions according to claim 1, characterized in that: The preset feature aggregation structure includes multiple position embedding modules, wherein each position embedding module in the multiple position embedding modules has the same architecture and different weights; the position embedding module includes channel-wise maximum pooling and a spatially-aware importance generator.

5. The cross-view image geolocation method under direction misalignment conditions according to claim 1, characterized in that: The making a decision based on the sorting result to obtain a final sorting result includes: Determine the Euclidean distance between each first output feature vector in the first output set and each second output feature vector in the second output set, wherein the number of the Euclidean distances is the number of the simulated remote sensing images after polar coordinate transformation; Sorting is performed based on the Euclidean distance to obtain a sorting result.

6. A cross-view image geolocation device under direction misalignment conditions, characterized in that: include: A generating module, configured to perform random shifts of different degrees within a 360-degree range for a target number of times on the ground panoramic image, and generate a target number of shifted ground panoramic images corresponding to the target number of times, wherein the direction of the ground panoramic image is an unknown direction, and the viewing angle of the ground panoramic image is 360 degrees; An output module, used for inputting the ground panoramic image and the offset ground panoramic image of the target number into a pre-trained generative network model respectively, to obtain a simulated polar coordinate transformed remote sensing image output by the pre-trained generative network model, wherein the number of the simulated polar coordinate transformed remote sensing images is the sum of the target number and a constant 1; A transformation module is used to perform polar coordinate transformation on each remote sensing image in the remote sensing image sample set to obtain a remote sensing image data set after polar coordinate transformation; An extraction module, used for calling a preset convolutional neural network to perform feature extraction on the simulated remote sensing image after polar coordinate transformation and the remote sensing image data set after polar coordinate transformation, respectively, to obtain a first feature vector set and a second feature vector set; an aggregation module, configured to call a preset feature aggregation structure to perform feature aggregation on the first feature vector set and the second feature vector set respectively, to obtain a first output set and a second output set, wherein the preset feature aggregation structure has spatial perception capability; a matching module, configured to perform feature matching on the first output set and the second output set to obtain a sorting result, wherein the number of the sorting results is a sum of the target number and a constant 1; A decision module, used to make a decision based on the sorting result to obtain a final sorting result for image geolocation; The transformation module is also used to transform the pixel coordinates on the remote sensing image. , the pixel coordinates on the remote sensing image after polar coordinate transformation are determined by the following formula : ; Among them, the scales of the polar coordinate transformation are respectively × and × express; in, represents the horizontal coordinate of the pixel point on the remote sensing image, represents the width of the remote sensing image, Represents the vertical coordinate of the pixel after polar coordinate transformation, represents the height of the remote sensing image after polar coordinate transformation, represents the width of the remote sensing image after polar coordinate transformation, Represents the horizontal coordinate of the pixel after polar coordinate transformation; represents the vertical coordinate of the pixel point on the remote sensing image, represents the height of the remote sensing image, represents the sine function, Represents the cosine function.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the cross-view image geolocation method under the direction misalignment condition as described in any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the cross-view image geolocation method under direction misalignment conditions as described in any one of claims 1 to 5 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the cross-view image geolocation method under direction misalignment conditions as described in any one of claims 1 to 5 is implemented.

Citation Information

Patent Citations

  • Cross-view-angle image real-time matching geographic positioning method and system based on deep learning

    CN114241464A

  • Geographic positioning method based on ground-air cross view

    CN114757999A

  • Cross-view image geographic positioning method and device based on attention efficiency network

    CN116310701A

  • Four-branch cross-view-angle image geographic positioning system and method of collaborative perspective view angle and polar coordinate view angle

    CN116310813A

  • Remote sensing image generation method and device based on ground-air visual angle geometric transformation

    CN117274514A