A cross-view target geolocation method based on view generation

Abstract

The application discloses a cross-view target geographic positioning method based on view generation, adopts a polar coordinate conversion method for cross-view pictures of satellite remote sensing and unmanned aerial vehicle aerial photography, and preliminarily eliminates the difference between two view domains; a conditional generative adversarial network Pix2Pix GAN is used to train satellite-unmanned aerial vehicle image pairs, a generative network is used to generate unmanned aerial vehicle view pictures from satellite remote sensing images, and a satellite view image library is constructed; the generated pictures are one-to-one corresponding to initial unmanned aerial vehicle view pictures, and are divided into a training set and a verification set for subsequent training and evaluation of geographic positioning matching; a deep learning model ConvNeXtV2 pre-trained through an ImageNet data set is used for feature extraction and feature similarity matching, so that cross-view geographic positioning is realized. The application still has strong robustness even in the case that the view angle is large (unmanned aerial vehicle 45-degree oblique shooting view angle and satellite bird's eye view angle) and the unmanned aerial vehicle shooting angle is variable.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of geographic positioning, and particularly relates to a cross-view target geographic positioning method based on view generation. BACKGROUND

[0002] Cross-view target geographic positioning can improve the accuracy and robustness of target positioning by combining information from different views. The data from different views provides multi-dimensional information of the target, reduces the errors that may be caused by a single view, and improves the accuracy of positioning. It can enhance the target detection and tracking ability in a dynamically changing complex environment through the cooperation of different views.

[0003] Cross-view geographic positioning aims to determine the target location through cross-platform image matching, specifically including matching ground orthographic view images, unmanned aerial vehicle near 45-degree oblique view angle images, and satellite overhead view images with geographic labels. The core lies in bridging the view difference and establishing the association between these images and the actual geographic space. Cross-view geographic positioning usually involves multiple technologies, including feature extraction, deep learning modeling, large-scale image retrieval, and spatial geometric reasoning. It not only focuses on low-level features such as texture, shape, and color of image content itself, but also needs to combine context information and semantic understanding to capture potential associated features in cross-view images. Existing technologies have been mainly applied to unmanned aerial vehicle positioning, disaster assessment, smart city construction, and military reconnaissance scenes, i.e., by providing unmanned aerial vehicle image queries to match candidate satellite images, and then locating the current geographic position. At the same time, this technology also provides important support for improving the cross-domain cognitive ability in the field of vision.

[0004] The research on cross-view geographic positioning mainly divides into two methods, one is based on traditional hand-crafted features for cross-view matching, and the other is based on deep learning methods. In the method of cross-view matching based on traditional hand-crafted features, initially, hand-crafted features such as direction gradient histogram and color histogram are used for matching. Later, there are studies that distort street panoramic images into overhead view images and use SIFT, SURF, and FREAK feature detection algorithms for matching to locate the geographic position of the query image. However, the image feature extraction method of hand-crafted image features has too large differences in extracted image features, and the retrieval accuracy is not high.

[0005] In the cross-view matching method based on deep learning, some methods try to use a convolutional neural network (CNN) to establish a network model, pre-train the cross-view image pair, use the trained network to extract the features of the ground query image and the database aerial view image respectively, and perform feature retrieval to obtain the geographic location of the query image. For example, LPN (Local Pattern Network) realizes end-to-end learning of context information through a feature-level division strategy. SAIG-D (Simple Attention Image Geolocation Network) adopts a full-connection form of MLP-mixer. This method can effectively represent the long-distance interaction between image blocks and the corresponding relationship between cross-view images through a multi-head self-attention layer. Some recent researches try to use a vision transformer (ViT) for cross-view image retrieval. However, these methods try to directly extract image features between different domains for comparison through a more powerful neural network model. They only learn feature representation based on image content, but ignore the spatial correspondence between unmanned aerial vehicle and satellite images, so the positioning capability is difficult to improve effectively.

[0006] In order to reduce the difference between cross-views, many researches are devoted to converting cross-view images into the same view before image retrieval. The current methods for view conversion mainly include perspective projection transformation, polar coordinate conversion, and generative adversarial network (GAN). The PCL method uses conditional GAN to generate corresponding satellite view images from unmanned aerial vehicle view images, and then uses a ResNet network for image matching and geographic positioning. However, this method needs to first input the unmanned aerial vehicle view image into the GAN to generate the corresponding satellite view, and then input the result into the feature extraction network. The extracted features are used for feature retrieval and comparison with the satellite image library. This process is serial, and there is computational redundancy. For some real-time scenarios, there may be problems.

[0007] Therefore, in view of the above problems, there is an urgent need to provide a new cross-view geographic positioning method. SUMMARY

[0008] The present application aims to provide a cross-view target geographic positioning method based on view generation, which aims to overcome the shortcomings of the existing geographic positioning method that only relies on feature extraction network to effectively realize cross-view image feature matching and network calculation redundancy, resulting in insufficient real-time performance.

[0009] The technical solution for achieving the purpose of the present application is a cross-view target geographic positioning method based on view generation, comprising the following steps:

[0010] Step 1: Obtain satellite and unmanned aerial vehicle cross-view geographic positioning images of the region to be queried, construct image pairs matched at different views, the image pairs include unmanned aerial vehicle view images and satellite view images of the same geographic location, and divide the unmanned aerial vehicle view images into a training set and a verification set;

[0011] Step 2: For each image in the image pair in step 1, re-map each pixel of the image to a polar coordinate system with the image center point as the origin by polar coordinate conversion to generate a corresponding panoramic view image, wherein the panoramic view image generated from the unmanned aerial vehicle view image is uniformly denoted as D, the panoramic view image generated from the satellite view image is uniformly denoted as S, and D is constructed as an unmanned aerial vehicle image query library, denoted as Q;

[0012] Step 3: Construct a conditional generative adversarial network Pix2Pix GAN, input S and D in step 2 at the same time, output a pseudo unmanned aerial vehicle view image generated from S, train the conditional generative adversarial network Pix2Pix GAN, and construct a satellite image retrieval library from the pseudo unmanned aerial vehicle view image generated from S, denoted as R;

[0013] Step 4: Match Q in step 2 with R in step 3 one by one, and take the matched image pairs as positive sample pairs, and the rest as negative sample pairs;

[0014] Step 5: Based on the pre-trained ConvNeXtV2 network, construct a geographic positioning matching model, input S and D in step 2 at the same time, input each image into the ConvNeXtV2 network in turn, output the feature vector corresponding to each image, use InfoNCE contrastive loss for back propagation, reduce the feature similarity distance between the feature vectors corresponding to the positive sample pairs, and expand the distance between the feature vectors corresponding to the negative sample pairs, train the geographic positioning matching model, and construct a satellite image feature retrieval library from the satellite image features of the positive sample pairs;

[0015] Step 6: Use the unmanned aerial vehicle image features in the verification set to perform feature matching and sorting with the satellite image feature retrieval library, match the K satellite image features with the highest feature similarity, obtain the position of the target in the unmanned aerial vehicle view image through a target detection model, cross-check the satellite remote sensing image corresponding to the matched satellite image feature for the identified target region, compare and analyze the spatial geometric relationship and pixel distribution characteristics of the target, and recalculate the pixel geometric center of the target in the satellite view image to realize cross-view target positioning.

[0016] Further, step 2: for each image in the image pair in step 1, using polar coordinate conversion, taking the image center point as the origin of the polar coordinate system, remapping each pixel of the image to the polar coordinate system to generate the corresponding panoramic view image, wherein the panoramic view image generated by the UAV view image is uniformly denoted as D, and the panoramic view image generated by the satellite view image is uniformly denoted as S, and D is constructed as a UAV image query library, denoted as Q, wherein the conversion formula is as follows:

[0017]

[0018] wherein and represent the pixel coordinates of the original view image and the panoramic view image respectively, and their image sizes correspond to (W s ,H s ) and (W ps ,H ps ) respectively.

[0019] Further, step 3: constructing a conditional generative adversarial network Pix2Pix GAN, taking S and D in step 2 as input, outputting the pseudo-UAV view image generated by S, training the conditional generative adversarial network Pix2Pix GAN, and constructing the pseudo-UAV view image generated by S as a satellite image retrieval library, denoted as R, wherein:

[0020] The conditional generative adversarial network Pix2Pix GAN includes a generator and a discriminator, wherein the generator network model adopts a U-Net architecture, which includes an encoder and a decoder for feature extraction by downsampling and feature restoration by upsampling, specifically, given a satellite view image In the encoder, the input is converted into and then down-sampled to get a feature map By analogy, the feature map size is reduced by half and the channel number is doubled every round of down-sampling, and the feature map X2, X3, X4 is obtained by passing through the second, third and fourth encoding blocks respectively; in the decoder, the feature map X4 is input to the convolution layer after being concatenated with the feature map X3 by the nearest neighbor up-sampling layer to obtain the feature map Y4, Y4 is input to the convolution layer after being concatenated with X2 by the up-sampling layer to obtain the feature map Y3, and so on, to obtain Y2, Y1; finally, the feature map Y1 is input to the 1x1 convolution layer to obtain the final output Y0 is the final generated pseudo-UAV view image;

[0021] The discriminator network model uses a PatchGAN structure composed of multiple convolutional operations, and the real UAV view image and the pseudo-UAV view image generated by the generator are concatenated to obtain Then, the feature map is down-sampled three times, each time doubling the channel and reducing the size by half, and then sequentially padded, 1x1 convolution, BN normalization, LeakyReLU activation function, padding, 1x1 convolution, and finally an output matrix is obtained For judging the authenticity of each block region of the pseudo-unmanned aerial vehicle perspective image.

[0022] Further, step 3: build a conditional generative adversarial network Pix2Pix GAN, input S and D in step 2 at the same time, output the pseudo-unmanned aerial vehicle perspective image generated by S, train the conditional generative adversarial network Pix2Pix GAN, and build the pseudo-unmanned aerial vehicle perspective image generated by S as a satellite image retrieval library, denoted as R, wherein the total loss function formula of training the conditional generative adversarial network Pix2Pix GAN is:

[0023]

[0024] L L1 =E[||y-G(x)||1]#

[0025] L total =λL L1 +L GAN

[0026] Wherein, y is the real target image, G(x) is the image generated by the generator, D(x,y) is the discrimination value of the discriminator for the real image, D(x,G(x)) is the discrimination value of the discriminator for the generated image, L L1 is the L1 loss between the generated image and the target image, L GAN is the adversarial loss of the generator and the discriminator, L total is the final total loss function, and the parameter λ is used to balance the importance of L1 loss and adversarial loss.

[0027] Further, step 3: build a conditional generative adversarial network Pix2Pix GAN, input S and D in step 2 at the same time, output the pseudo-unmanned aerial vehicle perspective image generated by S, train the conditional generative adversarial network Pix2Pix GAN, and build the pseudo-unmanned aerial vehicle perspective image generated by S as a satellite image retrieval library, denoted as R, wherein the process of training the conditional generative adversarial network Pix2Pix GAN:

[0028] The optimizer adopts the Adam method, the learning rate of the generator and the discriminator is 0.0001, the momentum parameters β1 and β2 are set to 0.5 and 0.999 respectively, and the batch size is set to 8.

[0029] Further, step 5: based on the pre-trained ConvNeXtV2 network, a geographic positioning matching model is constructed, S and D in step 2 are simultaneously input, each image is input into the ConvNeXtV2 network in turn, the feature vector corresponding to each image is output, the InfoNCE contrast loss is used for back propagation, the feature similarity distance between the feature vectors corresponding to the positive sample pairs is reduced, the distance between the feature vectors corresponding to the negative sample pairs is expanded, the geographic positioning matching model is trained, and the satellite image features of the positive sample pairs are used to construct a satellite image feature retrieval library, wherein:

[0030] The ConvNeXtV2 network is composed of 4 stages, each stage respectively stacks a plurality of ConvNeXtV2 blocks and a convolution downsampling block, the number of ConvNeXtV2 blocks in the 4 stages is respectively 3, 3, 27 and 3, each ConvNeXtV2 block is connected in turn with a depth separable convolution DW-Conv, a pointwise convolution Conv and a GELU activation function, which are respectively used for extracting spatial features, recombining feature channels and improving nonlinear expression ability; the downsampling block uses convolution, and except that the downsampling block in the first stage adopts a 4-fold downsampling rate, the downsampling blocks in the remaining stages all adopt a 2-fold downsampling rate.

[0031] The initial input image is After each stage, the number of feature channels gradually increases, which is respectively 128, 256, 512 and 1024, and the image size is reduced by half in each stage. After 4 stages, the feature vector output with a length of 1000 is obtained Finally, a global average pooling GAP and a fully connected layer Linear are used to obtain the feature vector output with a length of 1000

[0032] Further, step 5: based on the pre-trained ConvNeXtV2 network, a geographic positioning matching model is constructed, S and D in step 2 are simultaneously input, each image is input into the ConvNeXtV2 network in turn, the feature vector corresponding to each image is output, the InfoNCE contrast loss is used for back propagation, the feature similarity distance between the feature vectors corresponding to the positive sample pairs is reduced, the distance between the feature vectors corresponding to the negative sample pairs is expanded, the geographic positioning matching model is trained, and the satellite image features of the positive sample pairs are used to construct a satellite image feature retrieval library, wherein, the calculation formula of the InfoNCE loss function for training the geographic positioning matching model is:

[0033]

[0034] Wherein N represents the size of each batch, A + represents the unique query positive sample under each batch, B -represents (N-1) negative samples under each batch. During the training process, the query vector continuously reduces the distance between the positive sample and increases the distance between the negative sample, thereby reducing the loss value.

[0035] Further, step 5: based on the pre-trained ConvNeXtV2 network, a geographic positioning matching model is constructed, S and D in step 2 are simultaneously input, each image is input in turn through the ConvNeXtV2 network, and the feature vector corresponding to each image is output. The InfoNCE contrastive loss is used for back propagation to reduce the feature similarity distance between the feature vectors corresponding to the positive sample pairs, and to expand the distance between the feature vectors corresponding to the negative sample pairs. The geographic positioning matching model is trained, and the satellite image features of the positive sample pairs are used to construct a satellite image feature retrieval library. During the training of the geographic positioning matching model:

[0036] The AdamW optimizer is used for training, the initial learning rate is set to 0.001, the learning rate decay adopts the cosine annealing algorithm, the minimum learning rate is set to 0.0001, the batch size is set to 64, and the maximum training epoch number is 10.

[0037] Further, the target detection model adopts the YOLOv5 network.

[0038] A cross-view target geographic positioning system based on the cross-view target geographic positioning method, realizes cross-view target geographic positioning, and is divided into 6 modules, which respectively execute steps 1-6.

[0039] A computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the cross-view target geographic positioning method is realized based on the cross-view target geographic positioning method.

[0040] A computer readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the cross-view target geographic positioning method is realized based on the cross-view target geographic positioning method.

[0041] Compared with the prior art, the present application has the following advantages: 1) The spatial correspondence between the UAV aerial image and the satellite remote sensing image is fully utilized. Firstly, the polar coordinate conversion method is used to restore the scene distribution and object arrangement between the cross-view images, and the domain gap between the satellite view and the UAV view images is preliminarily narrowed. Secondly, the Pix2Pix GAN network is used to train the image pair generated by the polar coordinate conversion, so as to input the satellite view image and generate the UAV view image by learning the detailed feature mapping relationship. After the two steps, the domain gap between the cross-view image pairs is further reduced. 2) The ConvNeXtV2 model is used, which has stronger multi-scale feature extraction capability and can capture long-distance dependence and fine-grained features. The features of the UAV view query set are matched with the features in the satellite view retrieval library, so as to achieve the purpose of accurate geographic positioning. 3) The UAV view image is input into the GAN to generate the corresponding satellite view, and then the result is input into the feature extraction network. This process is serial, and there is calculation redundancy. Our method is to input the satellite remote sensing image into the GAN to generate a pseudo-UAV view image, and then build a satellite image retrieval database based on the generated image. This step can be parallel with the input UAV aerial image for retrieval positioning, and it no longer needs to generate the input UAV aerial image, which greatly reduces the calculation redundancy and improves the real-time performance of cross-view geographic positioning. 4) Overall, the present application is superior to the past work and method in the accuracy of cross-view geographic positioning matching, and opens up a new direction for the construction of future intelligent combat system. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 The overall flowchart of the cross-view target geographic positioning in the embodiment of the present application.

[0043] Figure 2 The structure diagram of the network model generated in the embodiment of the present application.

[0044] Figure 3 The structure diagram of the network model generated in the embodiment of the present application.

[0045] Figure 4 The structure diagram of the feature extraction network model in the embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0047] As Figure 1As shown, a cross-view target geolocation method based on view generation, the whole process mainly includes the following steps: 1. Constructing a UAV-satellite image pair; 2. Polar coordinate conversion; 3. Constructing conditional generative adversarial network Pix2PixGAN; 4. Constructing matching data required by the geolocation model; 5. Constructing the geolocation matching model ConvNeXtV2; 6. Load the trained ConvNeXtV2 model for inference prediction.

[0048] Step 1, constructing a UAV-satellite image pair:

[0049] Collect images from multiple public datasets (including CVUSA, CUACT, University-1652), a total of 3000 pairs of images, each pair of images containing 1 satellite view image and 1 UAV view image of the same geographic location, and divide the 3000 UAV view images into training set and validation set according to 8:2, that is, the training set and the validation set contain 2400 and 600 UAV view images respectively.

[0050] Step 2, polar coordinate conversion:

[0051] For each image in the image pair in step 1, polar coordinate conversion is used, taking the image center point as the origin of the polar coordinate system, and remapping each pixel of the image to the polar coordinate system to generate the corresponding panoramic view image. The panoramic view image generated from the UAV view image is uniformly denoted as D, and the panoramic view image generated from the satellite view image is uniformly denoted as S. D is constructed as a UAV image query library, denoted as Q. The polar coordinate conversion formula is as follows:

[0052]

[0053] Where and represent the pixel coordinates of the original view image and the panoramic view image respectively, and their image sizes correspond to (W s ,H s ) and (W ps ,H ps ) respectively. In the experiment, the image size is set to 512x512. Through polar coordinate conversion, the scene distribution and object arrangement of the original image are basically restored. For satellite images, polar coordinate transformation can map the parallel characteristics in the overhead view to a circular distribution, preserving the global information of the image; for UAV images, polar coordinate transformation adjusts the trapezoidal distortion to a radial distribution, close to the characteristics of satellite images. However, relying solely on this cannot completely eliminate the domain gap between cross-view views, and more detailed features cannot be restored through this method.

[0054] Step 3, constructing conditional generative adversarial network Pix2Pix GAN:

[0055] The conditional generative adversarial network Pix2Pix GAN is constructed, S and D in step 2 are simultaneously input, the output is a pseudo-unmanned aerial vehicle perspective image generated by S, a trained conditional generative adversarial network Pix2Pix GAN is obtained, and the pseudo-unmanned aerial vehicle perspective image generated by S is constructed into a satellite image retrieval library, denoted as R.

[0056] The conditional generative adversarial network Pix2Pix GAN includes a generator and a discriminator, wherein the generator network model adopts a U-Net architecture, which includes an encoder and a decoder, respectively used for feature extraction by downsampling and recovery by upsampling, specifically, given a satellite perspective image The input is converted into Then, the first encoding block is downsampled to obtain a feature map By analogy, the size of the feature map is reduced by half and the number of channels is doubled every round of downsampling, and the feature maps X2, X3, and X4 are obtained by passing through the second, third, and fourth encoding blocks, respectively; in the decoder, the feature map X4 is input to the convolution layer after being concatenated with the feature map X3 by the nearest neighbor upsampling layer to obtain the feature map Y4, Y4 is input to the convolution layer after being concatenated with X2 by the upsampling layer to obtain the feature map Y3, and so on, to obtain Y2 and Y1; finally, the feature map Y1 is input to the 1x1 convolution to obtain the final output Y0 is the final generated pseudo-unmanned aerial vehicle perspective image.

[0057] The discriminator network model uses a PatchGAN structure composed of multiple convolution operations, and the input real unmanned aerial vehicle perspective image and the pseudo-unmanned aerial vehicle perspective image generated by the generator are concatenated to obtain Then, the feature map is downsampled by three convolution layers, the number of channels is doubled and the size is reduced by half every time, and then padding, 1x1 convolution, BN normalization, LeakyReLU activation function, padding, and 1x1 convolution are sequentially performed to finally obtain The output matrix is used to judge the authenticity of each block area of the pseudo-unmanned aerial vehicle perspective image.

[0058] The total loss function formula of training the conditional generative adversarial network Pix2Pix GAN is:

[0059]

[0060] L L1 =E[||y-G(x)||1]#

[0061] L total =λL L1 +L GAN

[0062] where y is the real target image, G(x) is the image generated by the generator, D(x, y) is the discrimination value of the discriminator for the real image, D(x, G(x)) is the discrimination value of the discriminator for the generated image, L L1 is the L1 loss between the generated image and the target image, L GAN is the adversarial loss of the generator and the discriminator, L total is the final total loss function, and the parameter λ is used to balance the importance of the L1 loss and the adversarial loss.

[0063] In the process of training the conditional generative adversarial network Pix2Pix GAN, the Adam method is used as the optimizer, the learning rate of the generator and the discriminator is 0.0001, the momentum parameters β1 and β2 are set to 0.5 and 0.999 respectively, and the batch size is set to 8.

[0064] Step 4, construct the matching data required by the geolocation model:

[0065] Match Q in step 2 with R in step 3 one by one, which is used for training (denoted as T, numbered T Q1 ,T Q2 ,…T Q2400 and T R1 ,T R2 ,…T R3000 ), which constitute positive sample pairs (such as T Q1 and T R1 ) in training, and the rest are negative sample pairs (such as T Q1 and T R2 );

[0066] Step 5, construct the geolocation matching model ConvNeXtV2:

[0067] Using T of step 4, each image is sequentially input into the ConvNeXtV2 network, and the feature vector corresponding to each image is output. The InfoNCE contrastive loss is used for backpropagation to reduce the feature similarity distance between the feature vectors corresponding to the positive sample pairs, and to expand the distance between the feature vectors corresponding to the negative sample pairs, to obtain the trained geolocation matching model, where:

[0068] The ConvNeXtV2 network is composed of 4 stages, each of which stacks several ConvNeXtV2 blocks (the number of blocks in the 4 stages is 3, 3, 27, and 3, respectively) and a convolution downsampling block. Each ConvNeXtV2 block is connected in turn with a depthwise separable convolution DW-Conv, a pointwise convolution Conv, and a GELU activation function, which are used to extract spatial features, reorganize feature channels, and improve non-linear expression capabilities, respectively. The downsampling block is implemented using convolution. Except for the first stage downsampling block, which uses a 4-fold downsampling rate, the remaining stages use a 2-fold downsampling rate.

[0069] The initial input image is After each stage, the number of feature channels gradually increases, and the image size is reduced by half each time. After 4 stages, the feature vector output is Finally, a global average pooling GAP and a fully connected layer Linear are used to obtain a feature vector output with a length of 1000

[0070] The calculation formula of the InfoNCE loss function for training the geographic positioning matching model is:

[0071]

[0072] where N represents the size of each batch, A + represents the unique query positive sample under each batch, B - represents (N-1) negative samples under each batch. During the training process, the distance between the query vector and the positive sample is gradually reduced, and the distance between the query vector and the negative sample is gradually increased, thereby reducing the loss value.

[0073] During the training of the geographic positioning matching model, the AdamW optimizer is used for training, the initial learning rate is set to 0.001, the learning rate decay uses the cosine annealing algorithm, the minimum learning rate is set to 0.0001, the batch size is set to 64, and the maximum training epoch number is 10.

[0074] Step 6, load the trained ConvNeXtV2 model for inference prediction:

[0075] Input T in T R1 ,T R2 ,…T R3000, obtain corresponding satellite image features, construct a satellite image feature retrieval library, input the unmanned aerial vehicle perspective image to be tested in the verification set, first convert through polar coordinates, then pass through the trained ConvNeXtV2 model to obtain the unmanned aerial vehicle image feature, and use the unmanned aerial vehicle image feature to perform feature matching and sorting from the satellite image feature retrieval library, match K satellite image features with the highest feature similarity, obtain the position of the target in the unmanned aerial vehicle perspective image through the target detection model, cross-check the satellite remote sensing image corresponding to the matched satellite image feature for the identified target area, compare and analyze the spatial geometric relationship and pixel distribution characteristics of the target, recalculate the pixel geometric center of the target in the satellite perspective image, and realize cross-perspective target positioning.

[0076] Top-K (denoted as R@K) is used as an evaluation index for measuring the cross-perspective geographic target positioning model. R@K calculates the proportion of correct positions in the top K positions with the highest probability in the predicted positions, and its calculation formula is as follows:

[0077]

[0078] wherein, represents 1 when the real category y i exists in the top K categories predicted, otherwise 0. K takes 1, 5, 10 or the top 1%.

[0079] As can be seen from Table 1, compared with other methods (LPN is a local pattern network, which uses a feature level division strategy to learn context information, SAIG-D is a simple image geographic positioning network based on attention mechanism, and PCL is a method combining conditional GAN and ResNet network), the method of the present application has certain improvement in the matching accuracy of the top 1, 5, 10 and 1%, and the average precision, among which R@1 reaches 86.31%, R@5 and R@10 are 95.30% and 96.72% respectively, and in R@1% retrieval, it is further improved to 96.88%. These results prove the feasibility and efficiency of the strategy of first narrowing the cross-domain perspective gap and then using a powerful feature extraction network for matching in improving the adaptability of the model to the cross-perspective target geographic positioning task.

[0080] Table 1: Unmanned aerial vehicle image query results

[0081]

[0082] Any technical features in the above embodiments can be combined, and for the sake of brevity, not all possible combinations are described above, however, as long as the combinations of technical features do not contradict each other, they shall be considered within the scope of the present disclosure.

[0083] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it shall not be understood as a limitation on the scope of the present application. It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and these shall be within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A cross-view target geolocation method based on view generation, characterized in that, Includes the following steps: Step 1: Obtain satellite and UAV cross-view geolocation images of the area to be queried, and at the same time construct image pairs that match each other from different perspectives. Each image pair contains UAV view images and satellite view images of the same geographical location, and divide the UAV view images into training set and validation set. Step 2: For each image in the image pair in Step 1, use polar coordinate transformation, with the image center point as the origin of the polar coordinate system, and remap each pixel of the image to the polar coordinate system to generate the corresponding panoramic view image. The panoramic view image generated from the UAV view image is uniformly denoted as D, and the panoramic view image generated from the satellite view image is uniformly denoted as S. D is used to build a UAV image query library, denoted as Q. Step 3: Construct a conditional generative adversarial network Pix2Pix GAN, taking S and D from Step 2 as inputs, outputting the pseudo UAV view image generated by S, training the conditional generative adversarial network Pix2Pix GAN, and constructing a satellite image retrieval library from the pseudo UAV view image generated by S, denoted as R. Step 4: Match Q from Step 2 with R from Step 3 one by one. The successfully matched image pairs are taken as positive sample pairs, and the rest are taken as negative sample pairs. Step 5: Construct a geolocation matching model based on the pre-trained ConvNeXtV2 network. Take S and D from Step 2 as inputs simultaneously. Each image is passed through the ConvNeXtV2 network as input in turn. The output is the feature vector corresponding to each image. Backpropagation is performed using InfoNCE contrastive loss to reduce the feature similarity distance between the feature vectors corresponding to positive sample pairs and increase the distance between the feature vectors corresponding to negative sample pairs. Train the geolocation matching model and construct a satellite image feature retrieval library using the satellite image features of positive sample pairs. Step 6: Using the image features of the UAV under test in the validation set, perform feature matching and ranking with the satellite image feature retrieval library, match the K satellite image features with the highest feature similarity, obtain the location of the target in the UAV view image through the target detection model, and perform cross-checking on the identified target area in combination with the satellite remote sensing image corresponding to the matched satellite image features, compare and analyze the spatial geometric relationship and pixel distribution characteristics of the target, recalculate the pixel geometric center of the target in the satellite view image, and realize cross-view target localization.

2. The cross-view target geolocation method according to claim 1, characterized in that, Step 2: For each image in the image pair from Step 1, perform polar coordinate transformation, using the image center point as the origin of the polar coordinate system, and remap each pixel of the image to the polar coordinate system to generate the corresponding panoramic view image. The panoramic view image generated from the UAV perspective image is denoted as D, and the panoramic view image generated from the satellite perspective image is denoted as S. D is then used to construct a UAV image lookup library, denoted as Q. The transformation formula is as follows: in and These represent the pixel coordinates of the original view image and the panoramic view image, respectively, and their image sizes correspond to (W... s H s ) and (W ps H ps ).

3. The cross-view target geolocation method according to claim 1, characterized in that, Step 3: Construct a conditional generative adversarial network (Pix2Pix GAN), taking S and D from Step 2 as inputs, outputting the pseudo-UAV view image generated by S, training the Pix2Pix GAN, and constructing a satellite image retrieval library, denoted as R, from the pseudo-UAV view images generated by S, where: The conditional generative adversarial network Pix2Pix GAN consists of a generator and a discriminator. The generator network model adopts the U-Net architecture, which includes an encoder and a decoder, used for downsampling feature extraction and upsampling feature recovery, respectively. Specifically, given a satellite view image... In the encoder, the input is transformed through a convolutional layer. Then, the feature map is obtained by downsampling the first coding block. Following this pattern, the size of the downsampled feature map is halved and the number of channels is doubled in each round. After passing through the second, third, and fourth coding blocks respectively, the downsampled feature maps X2, X3, and X4 are obtained. In the decoder, feature map X4 is concatenated with feature map X3 after passing through a nearest-neighbor upsampling layer and then input into a convolutional layer to obtain feature map Y4. Y4 is then concatenated with X2 after passing through an upsampling layer and then input into a convolutional layer to obtain feature map Y3. This process continues until Y2 and Y1 are obtained. Finally, feature map Y1 undergoes a 1×1 convolution to obtain the final output. Y0 is the final generated pseudo-drone view image; The discriminator network model uses a PatchGAN structure, consisting of multiple convolutional operations, which concatenates the input real drone view image with the pseudo drone view image generated by the generator. Then, after three convolutional downsampling layers, the feature map channels are doubled and the size is halved in each downsampling. This is followed by padding, 1×1 convolution, BN normalization, LeakyReLU activation, padding, and another 1×1 convolution, finally yielding the output matrix. Used to determine the authenticity of each region in a fake drone-view image.

4. The cross-view target geolocation method according to claim 3, characterized in that, Step 3: Construct a Conditional Generative Adversarial Network (Pix2Pix GAN). Using S and D from Step 2 as inputs, output the pseudo-UAV view image generated by S. Train the Pix2Pix GAN and construct a satellite image retrieval library, denoted as R, from the pseudo-UAV view images generated by S. The total loss function formula for training the Pix2Pix GAN is: L L1 =E[||y-G(x)||1]# THE total =λL L1 +L GAN Where y is the real target image, G(x) is the image generated by the generator, D(x, y) is the discriminant value of the discriminator for the real image, D(x, G(x)) is the discriminant value of the discriminator for the generated image, and L L1 It is the L1 loss between the generated image and the target image, L GAN It is the adversarial loss between the generator and the discriminator, L total It is the final total loss function, and the parameter λ is used to balance the importance of L1 loss and adversarial loss.

5. The cross-view target geolocation method according to claim 4, characterized in that, Step 3: Construct a Conditional Generative Adversarial Network (Pix2Pix GAN). Take S and D from Step 2 as inputs, and output the pseudo-UAV view image generated by S. Train the Pix2Pix GAN and construct a satellite image retrieval library, denoted as R, from the pseudo-UAV view images generated by S. During the training of the Pix2Pix GAN: The optimizer uses the Adam method, with a learning rate of 0.0001 for both the generator and discriminator, momentum parameters β1 and β2 set to 0.5 and 0.999 respectively, and a batch size of 8.

6. The cross-view target geolocation method according to claim 1, characterized in that, Step 5: Construct a geolocation matching model based on the pre-trained ConvNeXtV2 network. Using S and D from Step 2 as inputs, each image is sequentially passed through the ConvNeXtV2 network, outputting the feature vector corresponding to each image. Backpropagation is performed using InfoNCE contrastive loss to reduce the feature similarity distance between feature vectors corresponding to positive sample pairs and increase the distance between feature vectors corresponding to negative sample pairs. The geolocation matching model is then trained, and satellite image features of positive sample pairs are used to construct a satellite image feature retrieval database. The ConvNeXtV2 network consists of four stages, each stacking several ConvNeXtV2 Blocks and one convolutional downsampling block. The number of ConvNeXtV2 Blocks in the four stages are 3, 3, 27, and 3, respectively. Each ConvNeXtV2 Block is sequentially connected to a depthwise separable convolutional layer (DW-Con). v Pointwise convolution (Conv) and GELU activation functions are used to extract spatial features, reconstruct feature channels, and enhance nonlinear expression capabilities, respectively. The downsampling blocks are implemented using convolution. Except for the first stage downsampling block which uses a 4x downsampling rate, all other stages use a 2x downsampling rate. The initial input image is In each stage, the number of feature channels gradually increases, to 128, 256, 512, and 1024 respectively, while the image size is halved in each stage. After four stages, the image is obtained... Finally, a feature vector of length 1000 is obtained by applying a global average pooling (GAP) layer and a fully connected linear layer.

7. The cross-view target geolocation method according to claim 6, characterized in that, Step 5: Construct a geolocation matching model based on the pre-trained ConvNeXtV2 network. Using S and D from Step 2 as inputs, each image is sequentially passed through the ConvNeXtV2 network, outputting the feature vector corresponding to each image. Backpropagation is performed using the InfoNCE contrastive loss to reduce the feature similarity distance between feature vectors corresponding to positive sample pairs and increase the distance between feature vectors corresponding to negative sample pairs. The geolocation matching model is then trained, and satellite image features of positive sample pairs are used to construct a satellite image feature retrieval database. The formula for calculating the InfoNCE loss function for training the geolocation matching model is as follows: Where N represents the size of each batch, A + B represents a unique positive query sample within each batch. - This represents (N-1) negative samples in each batch. During training, the query vector continuously narrows the distance between itself and the positive samples and widens the distance between itself and the negative samples, thereby reducing the loss value.

8. The cross-view target geolocation method according to claim 6, characterized in that, Step 5: Construct a geolocation matching model based on the pre-trained ConvNeXtV2 network. Using S and D from Step 2 as inputs, each image is sequentially passed through the ConvNeXtV2 network, outputting the feature vector corresponding to each image. Backpropagation is performed using InfoNCE contrastive loss to reduce the feature similarity distance between feature vectors corresponding to positive sample pairs and increase the distance between feature vectors corresponding to negative sample pairs. The geolocation matching model is then trained, and satellite image features of positive sample pairs are used to construct a satellite image feature retrieval database. During the training of the geolocation matching model: Training was performed using the AdamW optimizer, with an initial learning rate of 0.001, learning rate decay using cosine annealing, a minimum learning rate of 0.0001, a batch size of 64, and a maximum number of training epochs of 10.

9. The cross-view target geolocation method according to claim 1, characterized in that, The target detection model uses the YOLOv5 network.

10. A cross-view target geolocation system, characterized in that, Based on the cross-view target geolocation method according to any one of claims 1-9, cross-view target geolocation is achieved, which is divided into 6 modules, and steps 1 to 6 are executed respectively.

Images