Air-to-ground geographic location method and device with unknown azimuth angle and computer equipment

Abstract

The application relates to an unknown azimuth angle air-to-ground geographic positioning method, device and computer equipment. The method extracts local features by using a convolution feature extraction network, reduces the size of a model under the condition of ensuring geographic positioning performance, improves network reasoning speed and reduces network storage resource occupation, and improves the deployability of the method; an incremental difficult sample mining training strategy is used; with the continuous improvement of network performance, the difficulty of training is gradually increased, the problems that a model collapses due to too high difficulty in a fixed difficulty training strategy and performance is poor due to too low difficulty can be avoided, the difficulty automatically changes with performance, and the influence of human experience when manually designing difficulty levels is avoided; an alignment-correlation hybrid loss is constructed, azimuth information in training data is used to enhance the perception ability of the direction, the dependence on true values is avoided, and the adaptability of the method to the unknown ground azimuth angle condition is improved.

Description

Technical Field

[0001] This application relates to the field of cross-view image retrieval technology, and in particular to a ground-to-air geolocation method, apparatus and computer equipment for unknown azimuth angles. Background Technology

[0002] Image retrieval geolocation refers to retrieving the most relevant image from a reference image library containing geographic coordinate information, given an image to be located, and using the geographic coordinates of the matching image as the location result. When there is a significant parallax between the image to be located and the reference image in geolocation, it is called cross-view image retrieval geolocation. In ground-to-air cross-view geolocation tasks, the image to be located comes from a ground view while the reference image comes from an aerial view; this dramatic difference in perspective presents a significant challenge to the task.

[0003] In the early stages of ground-to-air geolocation research, researchers used hand-designed features to represent and match images. However, hand-designed features rely heavily on human experience, making them very difficult to implement, and their low semantic depth severely limits matching effectiveness. Due to the significant differences between ground and aerial perspectives, even pseudo-images created by distorting one perspective into another through perspective transformation still exhibit considerable visual differences, making it difficult for hand-designed features to achieve good results and hindering the improvement of image retrieval accuracy.

[0004] With the development of deep learning technology, the ability of deep neural networks to extract high-level semantic features has led researchers to gradually abandon the meticulous design of handcrafted features and instead focus on training networks with more powerful performance. Deep learning methods mainly utilize the Siamese network structure in contrastive learning, using branches with identical structures to extract and match features from images viewed from different perspectives. Existing technologies address this from three aspects: network structure, perspective transformation, and training strategies. Examples include: 1) Layer-to-Layer Transformer (L2LTR) based on a convolutional neural network + Transformer network structure and the pure Transformer method TransGeo, which leverage the global encoding capability of the Transformer to achieve better performance; 2) Decoupled Hierarchical (DeHi), a decoupled multi-level method using polar coordinate perspective transformation, which eliminates differences between perspectives through perspective transformation; and 3) Sample for Geo-localization (S4G), a method proposing two global sample mining strategies, which utilize more effective samples for training to improve network performance.

[0005] Existing methods do not consider automatically and progressively adjusting the training difficulty of the network during the training process, and they also make unreasonable use of the orientation alignment prior information in the original dataset, either excessively or insufficiently, which limits further improvement of network performance. Summary of the Invention

[0006] Therefore, it is necessary to provide a ground-to-air geolocation method, apparatus, and computer equipment that can provide high-accuracy geolocation for unknown azimuth angles when the azimuth angle of the ground image is unknown, in order to address the above-mentioned technical problems.

[0007] A ground-to-air geolocation method for unknown azimuth angles, the method comprising:

[0008] The acquired ground and satellite images are preprocessed to obtain the true azimuth angle and the preprocessed ground and satellite images.

[0009] The preprocessed ground and satellite images are used to extract local features using a convolutional feature extraction network to obtain the local features of the ground and satellite images.

[0010] Based on the local features of the ground image and the local features of the satellite image, the sliding window method is used to estimate the azimuth information, and the estimated azimuth information of the ground image is obtained.

[0011] Based on the true azimuth information and the estimated azimuth information of the ground image, the corresponding local features of the ground image are cut from the local features of the satellite image to obtain the true azimuth cutting features and the estimated azimuth cutting features.

[0012] Based on the current highest accuracy, the local features of the ground image are progressively mined for hard samples by combining the ground truth azimuth cutting features and the estimated azimuth cutting features. A loss function is constructed based on the obtained hard samples to complete this round of training and testing. The current highest accuracy is then compared and judged, and the current highest accuracy is updated. The next round of training continues until the preset conditions are met, resulting in a well-trained network model.

[0013] The trained network model is used to extract features from real-time acquired ground and satellite images, and geolocation is performed based on the extraction results to obtain ground-to-air geolocation results.

[0014] In one embodiment, the acquired ground and satellite images are preprocessed to obtain true azimuth angles and preprocessed ground and satellite images, including:

[0015] The ground image is randomly horizontally shifted and cropped proportionally according to the field of view, and the amount of random horizontal shift is used as the true azimuth information.

[0016] By transforming satellite images to a pseudo-ground view using polar coordinates, the representation of azimuth angles is changed from rotation in the satellite image to translation in the pseudo-ground view image, resulting in a satellite image after polar coordinate transformation.

[0017] In one embodiment, the convolutional feature extraction network is a GonvNext network model, which includes four stages. The first stage includes convolutional layers, layer normalization, and several residual modules. The second stage includes a downsampling module and several residual modules. The third stage includes a downsampling module and several residual modules. The fourth stage includes an upsampling module, several residual modules, and an average pooling layer. The downsampling module includes layer normalization and convolutional layers. The upsampling module includes layer normalization and upsampling convolutional layers.

[0018] The residual module includes convolutional layers, layer normalization, convolutional layers, GELU activation function, convolutional layers, layer scale, and drop path, as well as residual connections from the input to the output.

[0019] In one embodiment, based on local features of the ground image and local features of the satellite image, a sliding window method is used to estimate the azimuth information to obtain the estimated azimuth information of the ground image, including:

[0020] Using local features of the ground image as a sliding window, the image is translated along the horizontal axis of the local features of the satellite image. The correlation between the local features of the satellite image and the local features of the ground image at each different location is calculated using the inner product. The location with the highest correlation is used as the estimated azimuth information. The correlation calculation expression is:

[0021]

[0022] Among them, F gc F s These are local features of the ground image and local features of the satellite image, respectively. gc and W s F respectively gc and F s The width, H and C are F gc and F s The common height and number of channels, % is the modulo operation, h, w, and c are the independent variables for traversing along the three dimensions of height, width, and channels, respectively, and i is the i-th position of the local feature on the horizontal axis of the satellite image.

[0023] In one embodiment, based on the true azimuth information and the estimated azimuth information of the ground image, portions of corresponding local features of the ground image are cut from the local features of the satellite image to obtain true azimuth cut features and estimated azimuth cut features, including:

[0024] The true azimuth information is scaled to the true azimuth information in the local features according to the ratio of image downsampling to feature downsampling.

[0025] Based on the true azimuth information and estimated azimuth information in the local features, corresponding portions of the local features of the ground image are cut from the local features of the satellite image. These are the true azimuth cutting features and the estimated azimuth cutting features. The method for using the azimuth information is to start from the leftmost side of the local feature of the satellite map, with the azimuth information serving as the starting point for horizontal cutting. The width and W of the cut portion are... gc same.

[0026] In one embodiment, the specific steps of progressive hard sample mining include:

[0027] Based on the current highest accuracy rate, the number of difficult samples to be mined is determined to be:

[0028]

[0029] Among them, A b For the current highest accuracy, N h Difficult sample size, N n A represents the number of negative samples in the training batch. l The lower threshold for starting sample mining is γ, which is used to adjust the training difficulty as A increases. b The linear parameter of the rate of change.

[0030] N is selected from the local features and ground truth azimuth cut features of the ground images in the training batch. h The set of the most difficult-to-distinguish negative samples n is obtained as follows:

[0031]

[0032] in, For the first set of difficult samples, N h Let n be a positive integer, n be a negative sample, and d be a negative integer. q,n Let q be the distance between a negative sample n and a local feature q of the ground image in the feature metric space.

[0033] N is selected from the local features and estimated azimuth cut features of the ground images in the training batch. h The set of the second most difficult negative samples is obtained by identifying the n most difficult negative samples to distinguish.

[0034] In one embodiment, the step of constructing a loss function based on the obtained hard samples includes:

[0035] The PHSM loss function is constructed based on the two hard sample sets as follows:

[0036]

[0037]

[0038] in, Let be the PHSM loss functions corresponding to the two hard sample sets, and q be the local features of the ground image. The training batch is N, where N is the total number of samples in the training batch; p1 and n1 are samples from the ground truth azimuth cut feature and the first hard sample set, respectively; p2 and n2 are samples from the estimated azimuth cut feature and the second hard sample set, respectively. These are the first and second most difficult sample sets, respectively. The distance between local features of the ground image and the ground truth azimuth cutting features in the feature metric space. The distance between local features of the ground image and samples in the first hard sample set in the feature metric space. The distance between local features of the image and the estimated azimuth cut features in the feature metric space. Let α be the distance between the local features of the ground image and the samples in the second hard sample set in the feature metric space, and let α be a hyperparameter used to adjust the convergence speed and performance.

[0039] Based on the current training epoch, the two PHSM loss functions are automatically weighted and summed to obtain the alignment-correlation hybrid loss:

[0040]

[0041] in, The alignment-correlation hybrid loss is used, where β is a hyperparameter that gradually decays from 1 to 0 as the training steps progress.

[0042] In one embodiment, a trained network model is used to extract features from real-time acquired ground and satellite images, and geolocation is performed based on the extraction results to obtain ground-to-air geolocation results, including:

[0043] Based on real-time acquired ground and satellite images, a trained network model is used to extract features, resulting in ground truth azimuth cut features and estimated azimuth cut features.

[0044] Calculate the 2-norm distance between the local features of each ground image and each ground truth azimuth cut feature and each estimated azimuth cut feature.

[0045] Sort all L2 distances in ascending order, and use the geographic coordinates of the satellite image with the highest L2 distance as the location information.

[0046] A ground-to-air geolocation device with an unknown azimuth angle, the device comprising:

[0047] The image preprocessing module is used to preprocess the acquired ground and satellite images to obtain the true azimuth angle and the preprocessed ground and satellite images.

[0048] The local feature extraction module is used to extract local features from the preprocessed ground and satellite images using a convolutional feature extraction network, thereby obtaining the local features of the ground and satellite images.

[0049] The azimuth information estimation module is used to estimate the azimuth information of the ground image by using a sliding window method based on the local features of the ground image and the local features of the satellite image.

[0050] The automatic progressive learning module is used to cut corresponding local features of the ground image from the local features of the satellite image based on the ground azimuth information and the estimated azimuth information of the ground image, respectively, to obtain the ground azimuth cut features and the estimated azimuth cut features. Based on the current highest accuracy, the local features of the ground image are progressively mined with the ground azimuth cut features and the estimated azimuth cut features, respectively. The loss function is constructed based on the obtained hard samples to complete the current round of training and testing. The current highest accuracy is compared and judged, and the current highest accuracy is updated to enter the next round of training until the preset conditions are met, and the trained network model is obtained.

[0051] The ground-to-air geolocation module is used to extract features from real-time acquired ground and satellite images using a trained network model, and perform geolocation based on the extraction results to obtain the ground-to-air geolocation result.

[0052] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the above-described method.

[0053] The aforementioned ground-to-air geolocation method, apparatus, and computer equipment for unknown azimuth angles employ a convolutional feature extraction network to extract local features, reducing the model size while maintaining geolocation performance. This simultaneously improves network inference speed and reduces storage resource consumption, enhancing the deployability of the method. A progressively difficult sample mining training strategy is used, gradually increasing the training difficulty as network performance improves. This avoids the problems of model collapse due to excessive difficulty or poor performance due to excessive ease, which are common with fixed-difficulty training strategies. Furthermore, the difficulty automatically changes with performance, avoiding the influence of human experience when manually designing difficulty levels. The constructed alignment-correlation hybrid loss utilizes azimuth information from the training data to enhance directional perception. Simultaneously, the weight of utilizing the ground truth value decreases continuously as training progresses, while the weight of correlation estimation increases accordingly, avoiding dependence on the ground truth and improving the method's adaptability to unknown ground azimuth angles. Attached Figure Description

[0054] Figure 1 This is a flowchart illustrating a ground-to-air geolocation method for an unknown azimuth angle in one embodiment.

[0055] Figure 2 This is a framework diagram of a ground-to-air geolocation method with unknown azimuth angle in one embodiment;

[0056] Figure 3 This is a schematic diagram illustrating the effect of polar coordinate transformation in one embodiment;

[0057] Figure 4 This is a schematic diagram of the structure of GonvNext in one embodiment;

[0058] Figure 5 This is a structural block diagram of a ground-to-air geolocation device with an unknown azimuth angle in one embodiment.

[0059] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0061] Abbreviations of names involved in this application:

[0062] Progressive Hard Sample Mining (PHSM) is a technique that extracts hard samples from progressively harder samples.

[0063] Alignment-Correlation Hybrid Loss, abbreviated as ACH Loss.

[0064] This method introduces an AutoMatic Progressive Learning (AMPLE) ground-to-air geolocation method for unknown azimuth angles.

[0065] In one embodiment, such as Figure 1 As shown, a ground-to-air geolocation method for unknown azimuth angles is provided, which includes the following steps:

[0066] Step 100: Preprocess the acquired ground and satellite images to obtain the true azimuth angle and the preprocessed ground and satellite images.

[0067] Specifically, the preprocessing input data includes: ground images, satellite image data, and the field of view of the ground images.

[0068] Output: Ground image after random horizontal translation and cropping according to the field of view, true azimuth information of the ground image, and satellite image after polar coordinate transformation.

[0069] Step 102: Use a convolutional feature extraction network to extract local features from the preprocessed ground image and satellite image to obtain the local features of the ground image and satellite image.

[0070] Specifically, in this stage, the convolutional feature extraction network extracts features from the image after step 100 to obtain the corresponding local features.

[0071] Convolutional feature extraction networks can be, but are not limited to, the backbone network of the ConvNeXt network model or other types of convolutional neural network models. The model is required to directly output local features (cube features height × width × number of channels, similar to an abstract image) without adding a final fully connected layer to convert local features into vector descriptors describing the entire image, and then perform subsequent sliding window estimation.

[0072] Step 104: Based on the local features of the ground image and the local features of the satellite image, the azimuth information is estimated using the sliding window method to obtain the estimated azimuth information of the ground image.

[0073] Specifically, the local features of the ground image are used as a sliding window to translate along the horizontal axis of the local features of the satellite image (related to the azimuth angle), and the correlation is calculated using the inner product. The position with the highest correlation is taken as the estimated azimuth angle information.

[0074] Step 106: Based on the true azimuth information and the estimated azimuth information of the ground image, cut the corresponding local features of the ground image from the local features of the satellite image to obtain the true azimuth cut features and the estimated azimuth cut features.

[0075] Specifically, the local features of the satellite image are segmented based on the estimated azimuth information and field of view of the ground image, and segmented based on the true azimuth information and field of view of the satellite image, resulting in local features of the satellite image segmented based on the true azimuth information and field of view (true azimuth segmentation features) and local features of the satellite image segmented based on the estimated azimuth information and field of view (estimated azimuth segmentation features).

[0076] Step 108: Based on the current highest accuracy, the local features of the ground image are progressively mined with the ground truth azimuth cutting features and the estimated azimuth cutting features. A loss function is constructed based on the obtained hard samples to complete this round of training and testing. The current highest accuracy is compared and judged, and the current highest accuracy is updated. The next round of training is then carried out until the preset conditions are met, and the trained network model is obtained.

[0077] Specifically, based on the current highest accuracy, the two types of satellite local features obtained in step 106 and the local features of the ground image extracted in step 102 are subjected to progressive hard sample mining, a loss function is constructed to complete this round of training and testing, and the current highest accuracy is compared and judged to update the current highest accuracy and the updated network model.

[0078] The trained network model is saved as a file containing the various parameters of the network model. When using the trained model for inference, the parameters are read first. The network and parameters correspond to each other, including names and dimensions.

[0079] The trained network model includes a convolutional feature extraction network. During inference, feature extraction is performed first, followed by operations to extract satellite image feature parts that correspond to local features of the ground image. Then, the 2-norm is used to calculate distances and sort the data.

[0080] During training, the localization results are known in the training set, specifically in the form of matching image pairs between ground images and satellite images. During training, features extracted from a ground image, along with features from both matching and non-matching satellite images, are used to construct a loss function. The network parameters are optimized by minimizing this loss function.

[0081] Step 110: Use a trained network model to extract features from the real-time acquired ground and satellite images, and perform geolocation based on the extraction results to obtain the ground-to-air geolocation result.

[0082] Specifically, a trained network model is used to extract features from real-time acquired ground and satellite images. The extracted features are then normalized using the 2-norm and sorted from smallest to largest. The geographic coordinates of the top-ranked satellite image are used as the location information.

[0083] The aforementioned ground-to-air geolocation method, apparatus, and computer equipment for unknown azimuth angles employ a convolutional feature extraction network to extract local features, reducing the model size while maintaining geolocation performance. This simultaneously improves network inference speed and reduces storage resource consumption, enhancing the deployability of the method. A progressively difficult sample mining training strategy is used, gradually increasing the training difficulty as network performance improves. This avoids the problems of model collapse due to excessive difficulty or poor performance due to excessive ease, which are common with fixed-difficulty training strategies. Furthermore, the difficulty automatically changes with performance, avoiding the influence of human experience when manually designing difficulty levels. The constructed alignment-correlation hybrid loss utilizes azimuth information from the training data to enhance directional perception. Simultaneously, the weight of utilizing the ground truth value decreases continuously as training progresses, while the weight of correlation estimation increases accordingly, avoiding dependence on the ground truth and improving the method's adaptability to unknown ground azimuth angles.

[0084] In one embodiment, step 100 includes: randomly horizontally translating the ground image and cropping it proportionally according to the field of view, using the random horizontal translation amount as the true azimuth information; converting the satellite image to a pseudo-ground view through polar coordinate transformation, transforming the representation of the azimuth from rotation in the satellite image to translation in the pseudo-ground view image, and obtaining the satellite image after polar coordinate transformation.

[0085] Specifically, (1) preprocessing is performed on the ground image and satellite image to obtain the true azimuth angle and the preprocessed ground image and satellite image:

[0086] In this stage, the ground image is randomly horizontally shifted and the corresponding portion is cropped proportionally according to the field of view, such as... Figure 2 Middle I G To I GC As shown, the two solid-line boxes represent the periodicity in the ground image, specifically the connection between -180° and 180°. Horizontal translation in this step is a data augmentation technique to improve the network's adaptability to unknown azimuth angles. The original ground image is a 360° panoramic image, and the cropped ground image's width is proportional to the original image's field of view's 360°. The random horizontal translation amount is recorded as 'a'. G This serves as the true azimuth information.

[0087] Meanwhile, by transforming satellite images to a pseudo-terrestrial view using polar coordinates, the viewing angle difference between the two perspectives can be reduced, thereby improving the accuracy of subsequent geolocation. Furthermore, the representation of azimuth angles is transformed from rotation in the satellite image to translation in the pseudo-terrestrial view image, reducing the difficulty of representing azimuth angles and thus improving the performance of the positioning method in scenarios where the azimuth angle is unknown. Specifically, let S×S be the size of the satellite image, H... S ×W S Let the size of the rectangular pseudo-ground image obtained after polar coordinate transformation be the target, then the number of pixels in the satellite image is... and the pixels of the target image The mapping relationship between the coordinates can be expressed as:

[0088]

[0089] The effect of this transformation and its comparison with the corresponding ground image are as follows: Figure 3 As shown.

[0090] In one embodiment, the convolutional feature extraction network in step 102 is a GonvNext network model, which includes four stages. The first stage includes convolutional layers, layer normalization, and several residual modules. The second stage includes a downsampling module and several residual modules. The third stage includes a downsampling module and several residual modules. The fourth stage includes an upsampling module, several residual modules, and an average pooling layer. The downsampling module includes layer normalization and convolutional layers. The upsampling module includes layer normalization and upsampling convolutional layers. The residual module includes convolutional layers, layer normalization, convolutional layers, GELU activation function, convolutional layers, layer scale, and drop path, with residual connections from the input to the output.

[0091] Specifically, GonvNeXt networks, such as Figure 4 As shown, the GonvNeXt network consists of four stages, each primarily composed of a downsampling / upsampling module and a residual module. In the downsampling and upsampling modules, k represents the kernel size, s represents the stride, and c represents the number of channels. In the residual module, k represents the kernel size for depthwise separable convolutions. The four stages are as follows:

[0092] S1: (k=4, s=4, c=C1) convolution, layer normalization, B1 (k=7) blocks. After S1, the width and height decrease to 1 / 4, and the number of channels changes from 3 to C1;

[0093] S2: Layer normalization, (k=2, s=2, c=C2) convolution, B2 (k=7) blocks. After S2, the width and height decrease to 1 / 2, and the number of channels changes from C1 to C2;

[0094] S3: Layer normalization, (k=2, s=2, c=C3) convolution, B3 (k=7) blocks. After S3, the width and height decrease to 1 / 2, and the number of channels changes from C2 to C3;

[0095] S4: Layer normalization, (k=2, s=2, c=C4) deconvolution, B4 (k=3) blocks, one (k=(4,1), s=(4,1)) average pooling layer. After S4, the length decreases to 1 / 2, the width increases to twice, and the number of channels changes from C3 to C4.

[0096] More specifically, for GonvNeXt-Tiny, which is a modification of ConvNeXt-Tiny, then (B1,B2,B3,B4) = (3,3,9,3), and (C1,C2,C3,C4) = (96,192,384,16). For other versions, the number of blocks in GonvNeXt remains the same as in ConvNeXt, while the number of channels in the first three stages of GonvNext remains the same as in ConvNeXt, and the number of channels in the fourth stage is set to 16. For different versions of GonvNext, the size of the local features extracted from images of the same size is the same. For example, for a 128×512×3 image, the feature size after four stages is 4×64×16.

[0097] In addition, the local features extracted in this step will eventually be normalized to the 2-norm. For example, the 4×64×16=4096-dimensional features are treated as a vector, and the vector length is normalized to 1.

[0098] In one embodiment, step 104 includes: using local features of the ground image as a sliding window, translating along the horizontal axis of the local features of the satellite image, and calculating the correlation between the local features of the satellite image and the local features of the ground image at each different location using the inner product, and taking the location with the highest correlation as the estimated azimuth information; the correlation calculation expression is:

[0099]

[0100] Among them, F gc F s These are local features of the ground image and local features of the satellite image, respectively. gc and W s F respectively gc and F s The width, H and C are F gc and F sThe common height and number of channels, % is the modulo operation, h, w, and c are the independent variables for traversing along the three dimensions of height, width, and channels, respectively, and i is the i-th position of the local feature on the horizontal axis of the satellite image.

[0101] Specifically, such as Figure 2 As shown, the local features of the ground image and the local features of the satellite image are F gc and F s F gc As the sliding window edge F s The horizontal axis is translated, and the inner product is calculated at each different position i according to formula (2).

[0102] Furthermore, the position with the largest inner product is used as the estimated azimuth information, denoted as a. sc :

[0103]

[0104] In one embodiment, step 106 includes: scaling the true azimuth information to the true azimuth information in the local features according to the downsampling ratio from the image to the features; cutting the corresponding local features of the ground image in the local features of the satellite image according to the true azimuth information and the estimated azimuth information in the local features, respectively, with the true azimuth cutting feature and the estimated azimuth cutting feature; wherein the azimuth information is used in such a way that, starting from the leftmost side of the local features of the satellite map, the azimuth information is the starting point for horizontal cutting, and the width of the cut part and W gc same.

[0105] Specifically, the true azimuth information a G First, the image is scaled to the ground truth azimuth information 'a' in the local features based on the downsampling ratio from the image to the features. sa W GC Width of the cropped ground image:

[0106] a sa =a G ×W gc / W GC (4)

[0107] Furthermore, based on the true azimuth information a in the features... sa Estimate azimuth information a sc Local features F in satellite images s The parts corresponding to local features of the ground image are cut into F. sa and F sc The method for using azimuth information is as follows: starting from the leftmost side of the local satellite feature, the azimuth information serves as the starting point for horizontal cutting, and the width of the cut portion and W... gcSame, such as Figure 2 As shown in the text "Featured field of view cut".

[0108] In one embodiment, the specific steps of progressive hard sample mining in step 108 include:

[0109] Based on the current highest accuracy rate A b Determine the number N of difficult samples to be mined. h for:

[0110]

[0111] Among them, A b For the current highest accuracy, N h Difficult sample size, N n A represents the number of negative samples in the training batch. l The lower threshold for starting sample mining is γ, which is used to adjust the training difficulty as A increases. b The linear parameter of the rate of change.

[0112] N is selected from the local features and ground truth azimuth cut features of the ground images in the training batch. h The set of the most difficult-to-distinguish negative samples n is obtained as follows:

[0113]

[0114] in, For the first set of difficult samples, N h d is an integer greater than 0. q,n Let n be the distance between the negative sample n and the local feature q of the ground image in the feature metric space; Let be the first set of difficult samples, and n be negative samples, where n is the feature of the satellite image that does not match q.

[0115] N is selected from the local features and estimated azimuth cut features of the ground images in the training batch. h The set of the second most difficult negative samples is obtained by identifying the n most difficult negative samples to distinguish.

[0116] In one embodiment, step 108, which involves constructing a loss function based on the obtained hard samples, includes: constructing PHSM loss functions for the two hard sample sets respectively:

[0117]

[0118]

[0119] in, Let be the PHSM loss functions corresponding to the two hard sample sets, and q be the local features of the ground image. The training batch is N, where N is the total number of samples in the training batch; p1 and n1 are samples from the ground truth azimuth cut feature and the first hard sample set, respectively; p2 and n2 are samples from the estimated azimuth cut feature and the second hard sample set, respectively. These are the first and second most difficult sample sets, respectively. The distance between local features of the ground image and the ground truth azimuth cutting features in the feature metric space. The distance between local features of the ground image and samples in the first hard sample set in the feature metric space. The distance between local features of the image and the estimated azimuth cut features in the feature metric space. Let α be the distance between the local features of the ground image and the samples in the second hard sample set in the feature metric space, and let α be a hyperparameter used to adjust the convergence speed and performance.

[0120] Based on the current training epoch, the two PHSM loss functions are automatically weighted and summed to obtain the alignment-correlation hybrid loss:

[0121]

[0122] in, The alignment-correlation hybrid loss is used, where β is a hyperparameter that gradually decays from 1 to 0 as the training steps progress.

[0123] In this stage, progressive hard sample mining is first performed based on the features obtained in previous steps and the highest recorded accuracy; (F gc ,F sa ) and (F gc ,F sc The two branches of the network perform hard sample mining, each with comparable status. The asymptotic hard sample mining selects samples deemed to have high training value. Using these samples, each branch constructs its own loss function. These two loss functions are then weighted and summed to obtain the final loss, which is used to train the network. After one round of training and updating the network model, a test is performed. If the accuracy of this training round is higher than the current highest accuracy A, the network is considered successful. b If the highest accuracy is reached, then the accuracy for this round will be updated; otherwise, it will remain unchanged.

[0124] Specifically, progressive hard sample mining incorporates network performance into the mining strategy. As network performance improves, the number of hard samples mined is continuously reduced to increase training difficulty, thus smoothly increasing the network's training difficulty. Based on the current best accuracy A... b The number of difficult samples N mined h The value of N is determined using formula (5). hThen, for a specific ground image feature q to be matched, in the training batch Choose N h The most difficult-to-distinguish negative samples n (the smaller the distance between n and q in the metric space, the more difficult it is to distinguish), construct a selected set S as shown in formula (6) (p is the feature of satellite image (positive sample) matching q, and n is the feature of satellite image (negative sample) not matching q).

[0125] Furthermore, for For each negative sample n, a weighted soft-boundary triplet loss is constructed, and the PHSM loss for a single image to be matched is obtained by averaging:

[0126]

[0127] Here, α is a hyperparameter used to adjust the convergence speed and performance. d represents the distance between features in the feature metric space. The goal of optimizing this loss function is to make positive sample pairs (q, p) as close as possible and negative sample pairs (q, n) as far apart as possible. Furthermore, finally... The final PHSM loss is obtained by averaging the losses corresponding to each q in the table.

[0128]

[0129] At the same time, it should be noted that (F gc ,F sa ) and (F gc ,F sc (The two groups are of similar status and each performs hard sample mining separately, resulting in two branches.) gc Acting as q, F sa It can be p, n, and F sc Similarly, it can also be p,n. Thus, through two similar branches, we can obtain losses of the form of formulas (7) and (8), denoted as L respectively. A and L C .

[0130] Furthermore, based on the current training round, L is automatically adjusted. A and L C The weighted summation yields the alignment-correlation hybrid loss as shown in formula (9), which is used to optimize the network model. In this embodiment, β specifically decays to 0 at the training step corresponding to the end of the tenth training round.

[0131] In one embodiment, step 110 includes: extracting features from real-time acquired ground images and satellite images using a trained network model to obtain ground truth azimuth cut features and estimated azimuth cut features; calculating the 2-norm distance between the local features of each ground image and each ground truth azimuth cut feature and each estimated azimuth cut feature; sorting all 2-norm distances in ascending order, and using the geographic coordinates of the satellite image with the highest ranking as the positioning information.

[0132] Specifically, this stage involves testing the model trained in the previous stage. Given a ground image to be located from a satellite image, the trained network model extracts local features. The local features of the ground image are used as a sliding window to estimate the correlation of local features in each satellite image and segment the corresponding parts. After segmenting all satellite image features, the 2-norm distance between the ground features and each satellite feature is calculated. The satellite features are then sorted from smallest to largest based on this distance, with smaller distances indicating a higher probability of a correct match. Simply put, the geographic coordinates of the top-ranked satellite image can be used as the location information. However, considering that the top-ranked image is not always a correct match, the probability of a correct match is modeled based on the distance, and post-processing methods are used to eliminate uncertainty.

[0133] In a specific embodiment, a ground-to-air geolocation method for unknown azimuth angles is proposed. This method simultaneously incorporates a lighter network structure, a more effective training strategy, and more rational information utilization into geolocation. Since this method automatically and progressively adjusts the training strategy difficulty and the utilization of ground-value azimuth information during training, it introduces AutoMatic Progressive Learning (AMPLE), which is named as such. This method achieves state-of-the-art performance in scenarios with unknown azimuth angles, exhibiting high matching accuracy, as shown in Tables 1 and 2. Table 1 compares the method with other methods on the CVUSA dataset, and Table 2 compares it with other methods on the CVACT dataset. The performance metric used for comparison is the recall rate R@K for finding correct matches in the first K images. Simultaneously, the method's R@1 and parameter count are compared, demonstrating that this invention achieves good performance while maintaining a lightweight network, facilitating deployment.

[0134] Table 1 shows the performance comparison on the CVUSA dataset.

[0135]

[0136]

[0137] Table 2 shows the performance comparison on the CVACT dataset.

[0138] L2LTR 68.82 86.44 90.38 96.43 TransGeo 67.74 85.34 89.46 96.67 DeHi 83.16 90.62 93.26 97.65 S4G 83.69 90.18 91.78 95.53 AMPLE (This invention) 85.94 93.72 94.86 97.69

[0139] Table 3. Comprehensive comparison of methods

[0140] L2LTR 195.90M 76.42 68.82 TransGeo 44.92M 78.59 67.74 DeHi 29.52M 82.38 83.16 S4G 88.00M 92.40 83.69 AMPLE (This invention) 24.80M 93.22 85.94

[0141] In summary, this method addresses the ground-to-air geolocation problem under unknown azimuth angles. It constructs a ground-to-air geolocation method based on progressive hard sample mining and alignment-correlation hybrid loss, applying the concept of automatic progressive learning to this problem. Through reasonable training strategies and information utilization methods, it improves network performance at low cost and lightweights the feature extraction network, enabling the algorithm to achieve superior performance with fewer model parameters and smaller size. The algorithm provided by this invention can well adapt to real-world environments with unknown azimuth angles in ground images, exhibiting higher performance compared to other existing algorithms, and consuming fewer resources, making it suitable for ground-to-air geolocation and possessing strong practical value.

[0142] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0143] In one embodiment, such as Figure 5 As shown, a ground-to-air geolocation device with unknown azimuth angle is provided, comprising: an image preprocessing module, a local feature extraction module, an azimuth angle information estimation module, an automatic progressive learning module, and a ground-to-air geolocation module, wherein:

[0144] The image preprocessing module is used to preprocess the acquired ground and satellite images to obtain the true azimuth angle and the preprocessed ground and satellite images.

[0145] The local feature extraction module is used to extract local features from the preprocessed ground and satellite images using an improved ConvNeXt network model, thereby obtaining the local features of the ground and satellite images.

[0146] The azimuth information estimation module is used to estimate the azimuth information of the ground image by using a sliding window method based on the local features of the ground image and the local features of the satellite image.

[0147] The automatic progressive learning module is used to cut corresponding local features of the ground image from the local features of the satellite image based on the ground azimuth information and the estimated azimuth information of the ground image, respectively, to obtain the ground azimuth cut features and the estimated azimuth cut features. Based on the current highest accuracy, the local features of the ground image are progressively mined with the ground azimuth cut features and the estimated azimuth cut features, respectively. The loss function is constructed based on the obtained hard samples to complete the current round of training and testing. The current highest accuracy is compared and judged, and the current highest accuracy is updated to enter the next round of training until the preset conditions are met, and the trained network model is obtained.

[0148] The ground-to-air geolocation module is used to extract features from real-time acquired ground and satellite images using a trained network model, and perform geolocation based on the extraction results to obtain the ground-to-air geolocation result.

[0149] In one embodiment, the image preprocessing module is further configured to perform random horizontal translation on the ground image and crop it proportionally according to the field of view, using the random horizontal translation amount as the true azimuth information; and to convert the satellite image to a pseudo-ground view through polar coordinate transformation, transforming the representation of the azimuth angle from rotation in the satellite image to translation in the pseudo-ground view image, thereby obtaining the satellite image after polar coordinate transformation.

[0150] In one embodiment, the convolutional feature extraction network in the local feature extraction module is a GonvNext network model, which includes four stages. The first stage includes convolutional layers, layer normalization, and several residual modules. The second stage includes a downsampling module and several residual modules. The third stage includes a downsampling module and several residual modules. The fourth stage includes an upsampling module, several residual modules, and an average pooling layer. The downsampling module includes layer normalization and convolutional layers. The upsampling module includes layer normalization and upsampling convolutional layers. The residual module includes convolutional layers, layer normalization, convolutional layers, GELU activation function, convolutional layers, layer scale, and drop path, with residual connections from the input to the output.

[0151] In one embodiment, the azimuth information estimation module is further configured to use the local features of the ground image as a sliding window to translate along the horizontal axis of the local features of the satellite image, and use the inner product to calculate the correlation between the local features of the satellite image and the local features of the ground image at each different location, and take the location with the highest correlation as the estimated azimuth information; the correlation calculation expression is shown in formula (2).

[0152] In one embodiment, the automatic progressive learning module is further configured to scale the true azimuth information to the true azimuth information in the local features according to the downsampling ratio from the image to the features; and to cut the corresponding local features of the ground image in the local features of the satellite image based on the true azimuth information and the estimated azimuth information in the local features, respectively, with the true azimuth cutting feature and the estimated azimuth cutting feature; wherein the azimuth information is used in such a way that, starting from the leftmost side of the local features of the satellite map, the azimuth information is the starting point for horizontal cutting, and the width of the cut portion and W gc same.

[0153] In one embodiment, the specific steps of progressive hard sample mining in the automatic progressive learning module include: determining the number of hard samples to be mined using the expression shown in formula (5) based on the current highest accuracy.

[0154] N is selected from the local features and ground truth azimuth cut features of the ground images in the training batch. h The most difficult negative samples n are obtained, and the first difficult sample set is shown in formula (6).

[0155] N is selected from the local features and estimated azimuth cut features of the ground images in the training batch. h The set of the second most difficult negative samples is obtained by identifying the n most difficult negative samples to distinguish.

[0156] In one embodiment, the step of constructing a loss function based on the obtained hard samples in the automatic progressive learning module includes: constructing PHSM loss functions as shown in formulas (7) and (8) based on the two hard sample sets respectively.

[0157] Based on the current training round, the two PHSM loss functions are automatically weighted and summed to obtain the alignment-correlation hybrid loss as shown in formula (9).

[0158] In one embodiment, the ground-to-air geolocation module is further configured to perform feature extraction using a trained network model based on real-time acquired ground images and satellite images to obtain ground truth azimuth cut features and estimated azimuth cut features; calculate the 2-norm distance between the local features of each ground image and each ground truth azimuth cut feature and each estimated azimuth cut feature; sort all 2-norm distances in ascending order, and use the geographic coordinates of the satellite image ranked first as the positioning information.

[0159] Specific limitations regarding ground-to-air geolocation devices with unknown azimuth angles can be found in the limitations of ground-to-air geolocation methods with unknown azimuth angles mentioned above, and will not be repeated here. Each module in the aforementioned ground-to-air geolocation device with unknown azimuth angles can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0160] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements a ground-to-air geolocation method with an unknown azimuth angle. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0161] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0162] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps in the above method embodiment.

[0163] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0164] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A ground-to-air geographic positioning method for unknown azimuth angles, characterized in that, The method includes: Preprocessing of ground and satellite images yields true azimuth angles and preprocessed ground and satellite images; The preprocessed ground and satellite images are used to extract local features using a convolutional feature extraction network to obtain the local features of the ground and satellite images; Based on the local features of the ground image and the local features of the satellite image, the azimuth information is estimated using the sliding window method to obtain the estimated azimuth information of the ground image; Based on the true azimuth information and the estimated azimuth information of the ground image, the corresponding local features of the ground image are cut from the local features of the satellite image to obtain the true azimuth cut features and the estimated azimuth cut features. Based on the current highest accuracy, progressive hard sample mining is performed using local features of the ground image and ground truth azimuth cutting features, and progressive hard sample mining is performed using local features of the ground image and estimated azimuth cutting features. A loss function is constructed based on the obtained hard samples to complete this round of training and testing. The results are compared and judged with the current highest accuracy, the current highest accuracy is updated, and the next round of training is entered until the preset conditions are met, resulting in a trained network model. The trained network model is used to extract features from the real-time acquired ground and satellite images, and geolocation is performed based on the extraction results to obtain the ground-to-air geolocation results. Based on the current highest accuracy rate, the number of difficult samples to be mined is determined to be: in, This represents the highest accuracy rate currently available. The difficulty lies in the number of samples. For training batches The number of negative samples. The lower threshold for starting sample mining. To adjust the training difficulty The linear parameter of the rate of change; Select from the local features and ground truth azimuth cut features of the ground images in the training batch. The most difficult negative sample to distinguish The first set of difficult samples is obtained as follows: in, For the first set of difficult samples, Integers greater than 0 For negative samples, Local features of negative sample n to ground image q Distance in the feature metric space; Select from the local features and estimated azimuth cut features of the ground images in the training batch. The most difficult negative sample to distinguish Thus, the second difficult sample set is obtained; The PHSM loss function is constructed based on the two hard sample sets as follows: in, , These are the PHSM loss functions corresponding to the two hard sample sets, respectively. For local features of ground images, For training batches, This represents the total number of samples in the training batch. These are the true azimuth angle cutting features and samples from the first difficult sample set, respectively. These are samples from the estimated azimuth cutting features and the second difficult sample set, respectively. , These are the first and second most difficult sample sets, respectively. The distance between local features of the ground image and the ground truth azimuth cutting features in the feature metric space. The distance between local features of the ground image and samples in the first hard sample set in the feature metric space. The distance between local features of the image and the estimated azimuth cut features in the feature metric space. The distance between local features of the ground image and samples in the second hard sample set in the feature metric space. It is a hyperparameter used to adjust the convergence speed and performance; Based on the current training epoch, the two PHSM loss functions are automatically weighted and summed to obtain the alignment-correlation hybrid loss: in, For alignment-correlation mixed loss, This is a hyperparameter that gradually decays from 1 to 0 as the training steps progress.

2. The method according to claim 1, characterized in that, The acquired ground and satellite images are preprocessed to obtain the true azimuth angle and the preprocessed ground and satellite images, including: The ground image is randomly horizontally shifted and cropped proportionally according to the field of view, and the random horizontal shift amount is used as the true azimuth information. By transforming satellite images to a pseudo-ground view using polar coordinates, the representation of azimuth angles is changed from rotation in the satellite image to translation in the pseudo-ground view image, resulting in a satellite image after polar coordinate transformation.

3. The method according to claim 1, characterized in that, The convolutional feature extraction network is a GonvNext network model, comprising four stages. The first stage includes convolutional layers, layer normalization, and several residual modules. The second stage includes a downsampling module and several residual modules. The third stage includes a downsampling module and several residual modules. The fourth stage includes an upsampling module, several residual modules, and an average pooling layer. The downsampling module includes layer normalization and convolutional layers. The upsampling module includes layer normalization and upsampling convolutional layers. The residual module includes convolutional layers, layer normalization, convolutional layers, GELU activation function, convolutional layers, layer scale, and drop path, as well as residual connections from the input to the output.

4. The method according to claim 1, characterized in that, The local features of the ground image and the local features of the satellite image are used to estimate the azimuth information using a sliding window method, resulting in estimated azimuth information for the ground image, including: The local features of the ground image are used as a sliding window, translated along the horizontal axis of the local features of the satellite image, and the correlation between the local features of the satellite image and the local features of the ground image at each different location is calculated using the inner product. The location with the highest correlation is used as the estimated azimuth information. The correlation calculation expression is: in, , These are local features of ground images and local features of satellite images, respectively. and They are respectively and width, and for and Common height and number of channels, % is the modulo operation. h, w, c These are the independent variables used for traversal along the three dimensions of height, width, and channel. i The first local feature on the horizontal axis of the satellite image i One position.

5. The method according to claim 1, characterized in that, Based on the true azimuth information and the estimated azimuth information of the ground image, portions of corresponding local features of the ground image are cut from the local features of the satellite image to obtain true azimuth cut features and estimated azimuth cut features, including: The true azimuth information is scaled to the true azimuth information in the local features according to the ratio of image downsampling to feature downsampling; Based on the true azimuth information and estimated azimuth information in the local features, corresponding portions of the local features of the ground image are cut from the local features of the satellite image. These are the true azimuth cutting features and the estimated azimuth cutting features. The method of using the azimuth information is as follows: starting from the leftmost side of the local feature of the satellite map, the azimuth information serves as the starting point for horizontal cutting, and the width of the cut portion and... same.

6. The method according to claim 1, characterized in that, A trained network model is used to extract features from real-time acquired ground and satellite images, and geolocation is performed based on the extraction results to obtain ground-to-air geolocation results, including: Based on real-time acquired ground and satellite images, a trained network model is used to extract features to obtain ground truth azimuth cut features and estimated azimuth cut features. Calculate the 2-norm distance between the local features of each ground image and each ground truth azimuth cut feature and each estimated azimuth cut feature; Sort all L2 distances in ascending order, and use the geographic coordinates of the satellite image with the highest L2 distance as the location information.

7. A ground-to-air geographic positioning device with an unknown azimuth angle, characterized in that, The device is used to execute the ground-to-air geolocation method for unknown azimuth angles as described in claim 1 to achieve ground-to-air geolocation for unknown azimuth angles; the device includes: The image preprocessing module is used to preprocess the acquired ground and satellite images to obtain the true azimuth angle and the preprocessed ground and satellite images; The local feature extraction module is used to extract local features from the preprocessed ground and satellite images using a convolutional feature extraction network to obtain the local features of the ground and satellite images. The azimuth information estimation module is used to estimate the azimuth information of the ground image by using a sliding window method to combine the local features of the ground image and the local features of the satellite image. An automatic progressive learning module is used to cut corresponding parts of the local features of the ground image from the local features of the satellite image based on the ground azimuth information and the estimated azimuth information of the ground image, respectively, to obtain ground azimuth cut features and estimated azimuth cut features. Based on the current highest accuracy, progressive hard sample mining is performed using the local features of the ground image and the ground azimuth cut features, and progressive hard sample mining is performed using the local features of the ground image and the estimated azimuth cut features. A loss function is constructed based on the obtained hard samples to complete the current round of training and testing. The current highest accuracy is compared and judged, and the current highest accuracy is updated. The next round of training is then carried out until the preset conditions are met, and a trained network model is obtained. The ground-to-air geolocation module is used to extract features from real-time acquired ground and satellite images using a trained network model, and perform geolocation based on the extraction results to obtain the ground-to-air geolocation result.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 6.

Images