A high-precision UAV-remote sensing image cross-viewpoint geolocation method

Abstract

The present invention discloses a high-precision cross-view geolocation method for drone-remote sensing images. By constructing a cross-view geolocation model for drone-remote sensing images enhanced by feature space grouping, multi-scale semantic features of drones and remote sensing images are extracted based on ConvNeXt to obtain global features and local features. The global features and local features spatially enhanced by the feature space grouping enhancement module are subjected to feature fusion and quality regression. The model is trained using a hybrid function measured by triplet loss and cross-entropy loss, and drone images and satellite images are sent to the trained model for matching to achieve cross-view geolocation of drone-remote sensing. The present invention utilizes a spatial grouping enhancement module to enrich the spatial information of features, and utilizes a hybrid function measured by triplet loss and cross-entropy loss to improve the effective information utilization and feature matching effectiveness between non-aligned image pairs, thereby improving the accuracy of cross-view matching.

Description

Technical Field

[0001] The present invention belongs to the field of photogrammetry and remote sensing technology, and in particular relates to a high-precision UAV-remote sensing image cross-viewing angle geographic positioning method. Background Art

[0002] Cross-perspective geolocation uses methods such as feature matching and deep learning to match multi-source, multi-perspective images to achieve accurate geolocation. This can serve applications such as drone positioning and drone navigation based on cross-perspective image matching. Solving the cross-perspective matching problem between drone-perspective images and satellite remote sensing images is the core challenge in achieving image-based geolocation. Drone images and remote sensing images differ significantly in perspective, spatial resolution, and imaging methods: first, the spatial resolution of remote sensing images is generally lower than that of drone images; second, remote sensing images are mostly orthophotos, while drones provide oblique perspective images from multiple perspectives, depicting image information in the vertical direction of the ground object, which helps to bridge the visual gaps between different perspectives, but also brings difficulties to image matching. Therefore, cross-perspective geolocation based on drone-remote sensing image matching is challenging.

[0003] Most drone images are oblique-view images, while most remote sensing images are orthophoto images. Existing matching methods mainly include: (1) traditional feature point matching methods, which have poor robustness; (2) deep learning-based methods, such as methods based on unit subtraction convolution operators to generate key point masks, methods based on local pattern networks (LPNs), and methods based on thermal value segmentation to achieve feature alignment at different regional levels. However, existing methods ignore the geometric feature space enhancement of drone-remote sensing images, and the positioning accuracy of traditional feature matching methods is poor.

[0004] In summary, there is an urgent need for a UAV-remote sensing cross-view image matching method that considers geometric feature space enhancement to achieve cross-view matching with low time cost and high positioning accuracy. Summary of the Invention

[0005] In response to the problem of cross-perspective matching between existing drone oblique images and remote sensing orthophotos, the present invention proposes a high-precision drone-remote sensing image cross-perspective geolocation method. By introducing spatial enhancement and convolutional mixed attention mechanism, it solves the problems of the existing technology such as lack of geometric feature spatial enhancement, limited cross-perspective geolocation performance, and high time complexity.

[0006] The specific steps of the present invention are as follows:

[0007] S1: Constructing a cross-view geolocation model for UAV-remote sensing images based on feature space grouping enhancement;

[0008] The cross-view geolocation model for UAV-remote sensing images based on feature spatial grouping enhancement includes a dual-branch ConvNeXt Siamese neural network that shares the weights of the UAV and remote sensing images. Each Siamese neural network branch includes: a ConvNeXt semantic feature extraction module, a spatial grouping enhancement module, a multi-scale feature fusion module, and a multi-classifier module.

[0009] S2: Multi-scale semantic feature extraction is completed through the ConvNeXt semantic feature extraction module, including global features and local features;

[0010] S3: Complete the spatial enhancement of global features through the spatial grouping enhancement module;

[0011] S4: Feature fusion and quality regression of global and local features are completed through the multi-scale feature fusion module and the multi-classifier module;

[0012] S5: The constructed UAV-remote sensing image cross-view geolocation model based on feature space grouping enhancement is trained using a hybrid function of triplet loss and cross entropy loss metrics.

[0013] S6: Feed the UAV imagery and satellite imagery into the trained model for matching to achieve cross-viewpoint geolocation of UAV-remote sensing.

[0014] Furthermore, the semantic feature extraction module in step S2 is specifically implemented as follows: a pre-trained convolutional neural network is used as the semantic feature extraction module; the pre-trained convolutional neural network is a network constructed by using a ConvNeXt network pre-trained on the ImageNet21k dataset to obtain an image classification model; specifically, as follows:

[0015]

[0016] Among them, L j ∈T b×N×C Represents the j features obtained, where j∈{1,2}; represents the ConvNeXt layer, T represents the feature; B represents the batch size; N represents the number of feature map elements; C represents the number of channels; the input image is denoted by x j , j represents the collection of x j The platform, j = {1, 2}. x1 represents the remote sensing image, and x2 represents the drone image.

[0017] Furthermore, the spatial grouping enhancement module in step S3 can adjust the importance of each sub-feature by generating an attention factor for each spatial position in each semantic group so that each individual group can autonomously enhance its learned representation and suppress noise.

[0018] Furthermore, the specific method of step S4 is as follows:

[0019] The global feature classifier in the multi-classifier module is used to obtain the global feature classification result; the multi-scale feature fusion module is used to integrate the processed multiple local features into a feature vector, which is put into the local feature classifier in the multi-classifier module to obtain the local feature classification result; finally, the global feature classification result and the local feature classification result are fused.

[0020] Furthermore, the specific method for training the model in step S5 is as follows:

[0021] The model is trained using the SGD optimizer with a weight decay of 5×10 -4 , the initial learning rate is 3×10 -3 , then at 40 and 80 epochs, the learning rate is reduced by 0.1, the model weights are initialized using the pre-trained ConvNext-tiny network, and the cross entropy loss function is used in training, as follows:

[0022]

[0023] Where R represents the number of classes; p(x jr ) and q(x jr ) represent the ground truth probability and estimated probability respectively; x jr Represents the characteristics of the corresponding target, if x jr =1, then p(x jr )=1, otherwise p(x jr )=0;

[0024] The triplet loss is used to reduce the Euclidean distance between feature vectors of the same category from different views as follows:

[0025] TL=max(||F a -F1||2-||F a -F n ||2+M,0);

[0026] where ||·||2 represents the L2-normal form; F a is the feature vector of remote sensing or drone image; F1 is the feature vector of image with the same category as a; F n is the feature vector of the image of a different category; M is a constant 0.3;

[0027] The total loss is composed of the cross entropy loss of multiple feature representations and the triplet loss, which is calculated as follows:

[0028]

[0029] Where k represents the kth feature in multiple feature representations; CL j: and TL j: x j The cross entropy loss and triplet loss of the k-th feature.

[0030] The present invention adopts the above technical solution and has the following technical effects:

[0031] 1) The present invention constructs a ConvNeXt twin neural network for geolocation, which improves the model convergence speed by sharing the weights of drones and remote sensing images;

[0032] 2) The present invention applies the feature space grouping enhancement method to cross-view geolocation of UAV-remote sensing images. By extracting and matching the spatial features of UAV images and remote sensing images, the accuracy of geolocation is improved.

[0033] 3) The present invention adopts a hybrid function of cross entropy loss and triplet loss loss measurement to improve the effective information utilization between image pairs when measuring the similarity between images. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a flow chart of the present invention;

[0035] Figure 2 This is the overall framework diagram of the cross-viewpoint geolocation model of UAV-remote sensing images based on feature space grouping enhancement of the present invention;

[0036] Figure 3 This is a heat map enhanced by feature space grouping in an embodiment of the present invention;

[0037] Figure 4 3 is a schematic diagram of retrieval of drones and remote sensing images in an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The present invention will be fully described below in detail with reference to the accompanying drawings.

[0039] like Figure 1 As shown in FIG, a high-precision UAV-remote sensing image cross-viewing geolocation method includes the following steps:

[0040] S1: Constructing a cross-view geolocation model for UAV-remote sensing images based on feature space grouping enhancement;

[0041] like Figure 2As shown in the figure, the cross-view geolocation model of UAV-remote sensing images based on feature space grouping enhancement includes a two-branch ConvNeXt Siamese neural network that shares the weights of UAV and remote sensing images. Each Siamese neural network branch includes: ConvNeXt semantic feature extraction module, spatial grouping enhancement module, multi-scale feature fusion module, and multi-classifier module.

[0042] S2: Multi-scale semantic feature extraction is completed through the ConvNeXt semantic feature extraction module, including global features and local features; specifically:

[0043] A pre-trained convolutional neural network is used as the semantic feature extraction module. The pre-trained convolutional neural network is a network constructed by pre-training the image classification model on the ImageNet21k dataset using the ConvNeXt network. The details are as follows:

[0044]

[0045] Among them, L j ∈T B×N×C represents the j features obtained, where j∈{1,2}; B represents the batch size; N represents the number of feature map elements; C represents the number of channels;

[0046] In the University-1652 geolocation dataset, the input image is denoted as x j , j represents the collection of x j The platform, j = {1, 2}. x1 represents the remote sensing image, and x2 represents the drone image. Then the remote sensing image x1 and the drone image x2 are input into the dual-branch ConvNeXt semantic feature extraction module to obtain the global features of the satellite remote sensing image. and local features Global features of drone images and local features

[0047] S3: The spatial enhancement of global features is completed through the spatial grouping enhancement module; the details are as follows:

[0048] The spatial grouping enhancement module adjusts the importance of each sub-feature by generating an attention factor for each spatial position in each semantic group, so that each individual group can autonomously enhance its learned representation and suppress possible noise. and global features of drone images Input spatial grouping enhancement module to obtain and

[0049] S4: Based on the obtained global features and local features, feature fusion and quality regression are realized; the details are as follows:

[0050] The global feature classifier in the multi-classifier module is used to obtain the global feature classification result; the multi-scale feature fusion module is used to integrate the processed multiple local features into a feature vector, which is put into the local feature classifier in the multi-classifier module to obtain the local feature classification result; finally, the global feature classification result and the local feature classification result are fused.

[0051] The global features of remote sensing images and local features of remote sensing images The classification module and feature fusion module in the multi-classifier module are integrated into a feature vector to integrate the global features of the drone image. and local features of drone images The classification module and feature fusion module in the multi-classifier module are integrated into another feature vector, and these two feature vectors will be used for cross-view matching.

[0052] S5: Train the constructed cross-view geolocation model of UAV imagery and satellite remote sensing imagery based on feature space grouping enhancement. The details are as follows:

[0053] The model is trained using the SGD optimizer with a weight decay of 5×10 -4 , the initial learning rate is 3×10- 3 , then at 80 and 120 epochs, the learning rate is reduced by 0.1, the model weights are initialized using the pre-trained ConvNext-tiny network, and the cross entropy loss function is used in training, as follows:

[0054]

[0055] Where R represents the number of classes; p(x jr ) and q(x jr ) represent the ground truth probability and estimated probability respectively; x jr Represents the characteristics of the corresponding target, if x jr =1, then p(x jr )=1, otherwise p(x jr )=0;

[0056] The triplet loss is used to reduce the Euclidean distance between feature vectors of the same category from different views as follows:

[0057] TL=max(||F a -F p ||2-||F a -F n ||2+M,0);

[0058] where ||·||2 represents the L2-normal form; F a is the feature vector of remote sensing or drone image; F1 is the feature vector of image with the same category as a; F n is the feature vector of the image of a different category; M is a constant 0.3;

[0059] The total loss is composed of the cross entropy loss of multiple feature representations and the triplet loss, which is calculated as follows:

[0060]

[0061] Where k represents the kth feature in multiple feature representations; CL j: and TL j: x j The cross entropy loss and triplet loss of the k-th feature.

[0062] S6: Send the drone image and satellite image to the trained model for matching to achieve cross-perspective geolocation of drone and remote sensing. Specifically as follows: given a geolocation dataset, it contains latitude and longitude coordinates, matched drone-remote sensing image equivalent data, and label information such as buildings, roads, and trees (this embodiment uses the University-1652 geolocation dataset as an example to illustrate the method. A single building has multiple drone images and only one remote sensing image). The remote sensing image and the drone perspective image are input into the dual-branch ConvNeXt semantic feature extraction module to obtain two global features and two local features. Then, the drone image global features and the remote sensing image global features are respectively input into the spatial grouping enhancement module (such as Figure 3 As shown in Figure 2). Then, the multi-classifier module is used to implement the classification and fusion of multi-scale features. Finally, the cross entropy loss and triple loss of the quality regression module are used to aggregate the matching scores of the feature maps. Output the matching Top-5 map (as shown in Figure 2). Figure 4 As shown in the figure, it should be noted that because the dataset contains multiple drone images and a single remote sensing image, when inputting drone images and searching for remote sensing images, there will inevitably be mismatched images in the output top-5. Conversely, this situation will not occur when inputting remote sensing images and searching for drone images.

[0063] As shown in Tables 1 and 2, this example demonstrates the effectiveness of the trained model based on the University-1652 geolocation dataset. Table 1 shows the results of searching for remote sensing image slices matching drone images, while Table 2 shows the results of searching for drone images corresponding to remote sensing image slices. As can be seen from Tables 1 and 2, the network learning method described in this invention achieves improved recall and average precision.

[0064] Table 1

[0065]

[0066] Table 2

[0067]

Claims

1. A high-precision UAV-remote sensing image cross-viewpoint geolocation method, characterized by: The specific steps are as follows: S1: Constructing a cross-view geolocation model for UAV-remote sensing images based on feature space grouping enhancement; The UAV-remote sensing image cross-view geolocation model based on feature spatial grouping enhancement includes a two-branch ConvNeXt Siamese neural network that shares the weights of the UAV and remote sensing images. Each Siamese neural network branch includes: a ConvNeXt semantic feature extraction module, a spatial grouping enhancement module, a multi-scale feature fusion module, and a multi-classifier module; S2: Multi-scale semantic feature extraction is completed through the ConvNeXt semantic feature extraction module, including global features and local features; S3: Complete the spatial enhancement of global features through the spatial grouping enhancement module; S4: Feature fusion and quality regression of global and local features are completed through the multi-scale feature fusion module and the multi-classifier module; S5: The constructed cross-view geolocation model for UAV-remote sensing images based on feature space grouping enhancement is trained using a hybrid function of triplet loss and cross entropy loss metrics; The model training method is specifically as follows: The whole model is trained using SGD optimizer with weight decay of 5×10 -4 , the initial learning rate is 3×10 -3 , then at 40 and 80 epochs, the learning rate is reduced by 0.1, the model weights are initialized using the pre-trained ConvNext-tiny network, and the cross entropy loss function is used in training, as follows: Where R represents the number of classes; p(x jr ) and q(x ir ) represent the ground truth probability and estimated probability respectively; x jr Represents the characteristics of the corresponding target, if x jr =1, then p(x jr )=1, otherwise p(x jr )=0; The triplet loss is used to reduce the Euclidean distance between feature vectors of the same category from different views as follows: TL=max(||F a -F p ||2-||F a -F n ||2+M,0); where ||·||2 represents the L2-normal form; F a is the feature vector of remote sensing or drone image; F p is the feature vector of the image with the same category as a; F n is the feature vector of the image of a different category; M is a constant 0.3; The total loss is composed of the cross entropy loss of multiple feature representations and the triplet loss, which is calculated as follows: Where k represents the kth feature in multiple feature representations; CL jk and TL jk x j The cross entropy loss and triplet loss of the k-th feature; S6: Feed the UAV imagery and satellite imagery into the trained model for matching to achieve cross-viewpoint geolocation of UAV-remote sensing.

2. The high-precision UAV-remote sensing image cross-viewing geolocation method according to claim 1, characterized in that: The semantic feature extraction module described in step S2 is specifically: using a pre-trained convolutional neural network as a semantic feature extraction module; the pre-trained convolutional neural network is a network constructed by using the image classification model pre-trained on the ImageNet21k dataset using the ConvNeXt network; the details are as follows: Among them, L j ∈T B×N×C Represents the j features obtained, where j∈{1,2}; represents the ConvNeXt layer, T represents the feature; B represents the batch size; N represents the number of feature map elements; C represents the number of channels; the input image is denoted by x j , j represents the collection of x j The platform, j = {1, 2}; x1 represents the remote sensing image, and x2 represents the UAV image.

3. The high-precision UAV-remote sensing image cross-viewing angle geolocation method according to claim 1, characterized in that: The spatial grouping enhancement module described in step S3 adjusts the importance of each sub-feature by generating an attention factor for each spatial position in each semantic group so that each individual group can autonomously enhance its learned representation and suppress noise.

4. The high-precision UAV-remote sensing image cross-viewing geolocation method according to claim 1, characterized in that: Step S4 is specifically as follows: using the global feature classifier in the multi-classifier module to obtain the global feature classification result; using the multi-scale feature fusion module to integrate the processed multiple local features into a feature vector, and putting it into the local feature classifier in the multi-classifier module to obtain the local feature classification result; finally, the global feature classification result and the local feature classification result are fused.

Images