Small-view-field-angle cross-view-angle image geographic positioning method and system

Abstract

The invention provides a small-field-angle cross-view-angle image geographic positioning method and system, and the method comprises the steps: carrying out the feature extraction and weighting processing of a small-field-angle training set obtained through preprocessing through employing a pre-trained OfficientNet network and a triple attention network, and obtaining a ground image feature descriptor set and a satellite image feature descriptor set; calculating the loss by adopting a comparison loss function or a loss function based on a multi-difficulty negative sample weighting strategy, and performing iterative training on the pre-trained OfficientNet network and the triple attention network according to a loss minimization target; and positioning a ground image to be positioned by adopting the trained OfficientNet network and the triple attention network. According to the method, the positioning precision of the to-be-positioned ground image is high, and the recall rates of the trained OfficientNet network and the triple attention network are averagely improved by 10% to 20%.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a method and system for geo-positioning images with a small field of view and a cross-viewing angle. Background Art

[0002] Image geolocation technology is a method to determine the geographic location of the scene or photographer based on the image content or additional data. Traditional image geolocation methods mainly match query images with reference images with labels from the ground perspective, but images with GPS locations on the Internet are mostly distributed in densely populated urban areas. This method has limited applicability in sparsely populated or remote areas.

[0003] In recent years, with the rapid development of aerospace industries such as drones and satellites, there are more and more aerial images with geographic location information. Since satellite images are easy to obtain, have a wide coverage, and are highly dense, matching ground images to be queried with satellite images has gradually become an increasingly popular geolocation technology. This technology is called cross-view image geolocation technology (CVIG). CVIG technology matches the ground view image to be queried with satellite images with a wide coverage to determine the geographic location of the image. However, due to the huge difference in field of view between the satellite view and the ground view, as well as the incomplete features of ground images with small field of view, traditional cross-view image geolocation technology cannot fully utilize the rich information brought by this difference, resulting in difficulties in cross-view image geolocation.

[0004] Therefore, how to use the information brought by the difference in field of view angle to study the matching relationship between ground images and satellite images is one of the current problems facing cross-view image geolocation. Summary of the invention

[0005] In order to solve the technical problem that the traditional cross-viewing angle image geolocation is difficult due to the huge difference in viewing angle between the satellite viewing angle and the ground viewing angle and the incomplete features brought by the small viewing angle ground image, the present invention discloses a small viewing angle cross-viewing angle image geolocation method, the method comprising the following steps:

[0006] S1, preprocessing the matching ground image and satellite image pairs in the panoramic image training set to obtain a small field of view training set;

[0007] S2. Perform feature extraction and weighted processing on the small field of view training set using a pre-trained EfficientNet network and a triple attention network to obtain a ground image feature descriptor set and a satellite image feature descriptor set;

[0008] S3, according to the ground image feature descriptor set and the satellite image feature descriptor set, using a contrast loss function or a loss function based on a multi-difficult negative sample weighting strategy to calculate the loss, and iteratively training the pre-trained EfficientNet network and the triple attention network according to the goal of minimizing the loss;

[0009] S5. Use the trained EfficientNet network and triple attention network to perform feature extraction and feature weighting processing on the real-time collected ground images and satellite images to be located in turn, obtain ground image feature descriptors and satellite image feature descriptors, perform geolocation according to the ground image feature descriptors and the satellite image feature descriptors, and obtain the positioning result of the ground image to be located.

[0010] In an optional embodiment, the above-mentioned small field of view cross-viewing angle image geolocation method further includes:

[0011] S4. Based on the panoramic image test set, the recall rate is used as the indicator to evaluate the trained EfficientNet network and triple attention network.

[0012] The embodiment of the present invention further provides a small-viewing-angle cross-viewing-angle image geographic positioning system, comprising an image preprocessing module, a network training module and a positioning module.

[0013] The image preprocessing module is used to preprocess the matching ground image and satellite image pairs in the panoramic image training set to obtain a small field of view training set;

[0014] The network training module is used to perform feature extraction and weighted processing on the small field of view training set using a pre-trained EfficientNet network and a triple attention network to obtain a ground image feature descriptor set and a satellite image feature descriptor set; according to the ground image feature descriptor set and the satellite image feature descriptor set, a contrast loss function or a loss function based on a multi-difficult negative sample weighting strategy is used to calculate the loss, and the pre-trained EfficientNet network and the triple attention network are iteratively trained according to the goal of minimizing the loss to obtain a trained EfficientNet network and a triple attention network;

[0015] The positioning module is used to use the trained EfficientNet network and triple attention network to perform feature extraction and feature weighting processing on the ground images and satellite images to be positioned that are collected in real time, obtain ground image feature descriptors and satellite image feature descriptors, perform geographic positioning according to the ground image feature descriptors and the satellite image feature descriptors, and obtain the positioning result of the ground image to be positioned.

[0016] An embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-mentioned small field of view cross-view image geolocation methods when executing the computer program, so as to solve the technical problem that the traditional cross-view image geolocation is difficult due to the huge field of view difference between the satellite perspective and the ground perspective and the incomplete features brought by the small field of view ground image.

[0017] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program for executing any of the above-mentioned small field of view cross-view image geolocation methods to solve the technical problem of difficulty in geolocation of traditional cross-view images caused by the huge field of view difference between the satellite perspective and the ground perspective and the incomplete features brought by the small field of view ground images.

[0018] Compared with the prior art, the beneficial effects achieved by at least one of the above technical solutions adopted in the embodiments of this specification include at least: using the EfficientNet network as a feature extraction network of the model, extracting image features with rich information and introducing a triple attention network as a feature processing module, enhancing the model's understanding and processing capabilities of image features, and solving the problem of incomplete ground image features. At the same time, by using a loss function based on a multi-difficult negative sample weighting strategy to calculate the loss, the decision boundary of the network is optimized through the target pair that minimizes the loss, the classification accuracy is improved, and the learning depth of the network for difficult negative samples is enhanced.

[0019] At the same time, experimental comparisons were conducted with the ResNet18 network method on the public standard dataset CVUSA. The results showed that at various field of view angles, the EfficientNet network and triple attention network of the present invention have high positioning accuracy for ground images to be positioned, and the recall rates of the trained EfficientNet network and triple attention network are improved by an average of 10% to 20%. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 A flowchart of a method for geo-positioning an image with a small field of view and across viewing angles is disclosed in an embodiment of the present invention;

[0022] Figure 2A technical roadmap for a method for geo-locating images with small field of view and across viewing angles is disclosed for an embodiment of the present invention;

[0023] Figure 3 The present invention discloses a panoramic image training set preprocessing process;

[0024] Figure 4 A schematic diagram of an MBConv block in an EfficientNet network is disclosed in an embodiment of the present invention;

[0025] Figure 5 A schematic diagram of a triple attention network in an EfficientNet network is disclosed in an embodiment of the present invention;

[0026] Figure 6 It is an architecture diagram of a small-viewing-angle cross-viewing-angle image geolocation system disclosed in an embodiment of the present invention;

[0027] Figure 7 A schematic diagram of a computer device disclosed in an embodiment of the present invention;

[0028] Among them, 601 is an image preprocessing module; 602 is a network training module; 603 is a positioning module; 701 is a memory; and 702 is a processor. DETAILED DESCRIPTION

[0029] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0030] The following describes the implementation methods of the present application through specific specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and the features of the embodiments can be combined with each other in the absence of conflict. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0031] The embodiment of the present invention discloses a method for geo-positioning of small field-of-view cross-viewing angle images, see Figure 1 and Figure 2 As shown, the geolocation method comprises the following steps:

[0032] S1, preprocessing the matching ground image and satellite image pairs in the panoramic image training set to obtain a small field of view training set;

[0033] S2. Perform feature extraction and weighted processing on the small field of view training set using a pre-trained EfficientNet network and a triple attention network to obtain a ground image feature descriptor set and a satellite image feature descriptor set;

[0034] S3, according to the ground image feature descriptor set and the satellite image feature descriptor set, using a contrast loss function or a loss function based on a multi-difficult negative sample weighting strategy to calculate the loss, and iteratively training the pre-trained EfficientNet network and the triple attention network according to the goal of minimizing the loss;

[0035] S5. Use the trained EfficientNet network and triple attention network to perform feature extraction and feature weighting processing on the real-time collected ground images and satellite images to be located in turn, obtain ground image feature descriptors and satellite image feature descriptors, perform geolocation according to the ground image feature descriptors and the satellite image feature descriptors, and obtain the positioning result of the ground image to be located.

[0036] In an optional embodiment of step S1, preprocessing the matched ground image and satellite image pairs in the panoramic image training set to obtain a small field of view training set includes:

[0037] S11. Under multiple field of view angles, perform fixed center cropping on each panoramic ground image in the panoramic image training set, and reconstruct the panoramic ground image and the cropped image into a small field of view angle ground image with the same resolution through a resize function, wherein the multiple field of view angles include 70°, 120° and 180°.

[0038] S12. Under multiple field of view angles, fixed center cropping is performed on the satellite images matching each of the panoramic ground images in the panoramic image training set, and the satellite images and the cropped images are reconstructed into small field of view satellite images with the same resolution as the small field of view ground images through a resize function to obtain a small field of view training set, wherein the small field of view training set includes multiple matching pairs of small field of view ground images and small field of view satellite images.

[0039] In the specific implementation, the panoramic image training set comes from the standard dataset CVUSA, which consists of matching pairs of ground panoramas and satellite images taken by satellites, with standardized directions and aligned GPS positions. The main scenes of image acquisition are from the suburbs of the city. The ground-view images in CVUSA are 360-degree panoramas with a resolution of 1232*224, while the resolution of the satellite images from the aerial perspective is 750*750.

[0040] To improve the efficiency, speed, and accuracy of network training, ensure data diversity, and reduce the computing resources required for training, see Figure 3 As shown, in the embodiment of the present invention, each panoramic ground image and satellite image in the panoramic image training set is cropped at multiple viewing angles of 70°, 120° and 180° by using a fixed center cropping method, and then the cropped images are reconstructed with the same resolution. Among them, fixed center cropping refers to cropping with the center of the image as the fixed center point, which can ensure that the cropped images have similar areas and the integrity of key information, thereby facilitating the network to better learn the detailed information in the image, and also helping to reduce the impact of data set noise on model training, so that the network can better learn the characteristics of ground images.

[0041] In the cross-view image matching task, the convolutional neural network (CNN) used in the feature extraction module is generally improved to improve the retrieval accuracy, and the depth, width and resolution of the network and the input image are the key factors affecting the performance of CNN. By increasing the depth and width of the network and improving the resolution of the input image, richer and more complex features can be obtained. However, if these factors are increased blindly, problems such as network degradation, increased training difficulty and huge amount of calculation may occur. Therefore, in the embodiment of the present invention, the EfficientNet network is selected as a new type of convolutional neural network (CNN) to extract the semantic features of the image.

[0042] In an embodiment of the present invention, the EfficientNet network selected in step S2 includes nine stages, the first stage includes a 3×3 convolution layer, a SiLU activation function layer, and a BN batch normalization layer. The second to eighth stages all include MBConv blocks. Specifically, the second stage is an MBConv block with an expansion ratio of 1 and a convolution kernel size of 3×3. The third to eighth stages are respectively composed of MBConv blocks with an expansion ratio of 6, a convolution kernel size of 3×3, and 5×5, which are stacked alternately in the numbers of 2, 2, 3, 3, 4, and 1. The ninth stage includes a 1×1 convolution layer, a pooling layer, and a fully connected layer, and the image features extracted after the 1×1 convolution in the ninth stage are used as the final image feature representation.

[0043] See also Figure 4As shown in the figure, the MBConv block includes a 1×1 convolution layer, a Depthwise convolution layer, a channel attention module SE, a 1×1 convolution layer and a residual block connected in sequence. The first 1×1 convolution layer is used to increase the feature dimension, in which batch normalization BN and Swish activation function are integrated; the Depthwise convolution layer is a kxk depth-separable convolution, where the specific value of k is set according to the EfficientNet-B0 network framework, usually 3x3 or 5x5; the channel attention module SE is used to recalibrate the features; the second 1×1 convolution layer is used to reduce the feature dimension; BN is also integrated, and the residual block Dropout layer is used to prevent overfitting.

[0044] Among them, Depthwise convolution can use independent convolution kernels to perform convolution operations on each channel of the image, thereby retaining the information of each channel to the greatest extent. It helps to fully retain the channel feature information such as the outline, shape, color, etc. of each object in the image, which facilitates subsequent feature recognition and screening. In addition, the SE module will output the feature map f of the Depthwise convolution. H×W×C (where H represents height, W represents width, and C represents the number of channels) performs a global average pooling operation to obtain a feature map f of size 1×1×C 1×1×C , which can compress the feature space dimension and capture the energy of each feature channel dimension, thereby extracting high-level semantic feature maps. 1×1×C The correlation between channel features is modeled through two fully connected layers, and the weights of each channel are determined. The weight mask Mask is generated using the Sigmoid activation function, in which the weights of irrelevant features are small, even close to 0, while useful features that help geolocation of cross-view images have larger weights, making them close to 1. The weight mask Mask is applied to the input Depthwise convolution feature map f through a multiplication operation. H×W×C , get the weighted feature map f H×W×C , which is the following high-level semantic feature map.

[0045] The EfficientNet network selected in the embodiment of the present invention can learn richer and more complex features while maintaining efficient computing, capture the semantic object characteristics in the image, and provide strong support for small field of view cross-view image matching tasks.

[0046] When performing geolocation of images with small field of view and cross-viewing angle, in addition to capturing the characteristics of semantic objects in the image, it is also necessary to fully consider the spatial layout relationship between semantic objects in the image. For example, when judging whether a ground image and a satellite image belong to the same geographical location, it is usually first judged whether the two images contain geographically significant content objects with similar colors and shapes, such as buildings, roads, and water systems, while ignoring content objects such as vehicles and pedestrians that are not geographically significant; then, it is judged whether the spatial layout relationship between the same content objects is consistent (for example, whether the roads in the two images are both located in the middle of the grass), and then accurately judge whether the two images belong to the same geographical location. In the embodiment of the present invention, the extracted high-level semantic feature map is refined by a triple attention network, the purpose of which is to determine the weights of different semantic objects at each position of the feature map, and to ensure that the descriptor finally generated can fully reflect the importance of each semantic object and its spatial distribution relationship.

[0047] Specifically, see Figure 5 As shown, the triple attention network includes a joint attention channel, a first spatial attention channel and a second spatial attention channel from top to bottom.

[0048] The first spatial attention channel is used to sequentially perform horizontal spatial transformation, attention weighting processing and inverse transformation on the high-level semantic feature map obtained after feature extraction to obtain a horizontal spatial attention weighted feature map;

[0049] The second spatial attention channel is used to sequentially perform vertical spatial transformation, attention weighting processing and inverse transformation on the high-level semantic feature map to obtain a vertical spatial attention weighted feature map;

[0050] The joint attention channel is used to perform attention weighted processing on the high-level semantic feature map to obtain an attention weighted feature map, and to perform weighted averaging on the attention weighted feature map, the horizontal space attention weighted feature map, and the vertical space attention weighted feature map to obtain an image feature descriptor. The high-level semantic feature map includes a high-level semantic feature map of a ground image and a high-level semantic feature map of a satellite image, and the image feature descriptor includes a ground image feature descriptor and a satellite image feature descriptor.

[0051] In the specific implementation, for the high-level semantic feature map f with a size of 1280×4×16 extracted by the EfficientNet network C×H×W First, the first spatial attention channel and the second spatial attention channel are used to obtain two other feature maps f through permutation operation. W×H×C and f H×C×W Then, the original high-level semantic feature map and the converted feature map are processed by three branches with the same structure to generate an attention-weighted feature map. and The specific processing flow is as follows: In the first branch (joint attention channel), the feature map f C×H×W After passing through the maximum pooling layer and the average pooling layer, the two are connected in series to form a feature map f with a shape of 2×H×W 2×H×W The maximum pooling layer helps to highlight the significant features in each feature map and reduce the interference of irrelevant information, thereby improving the effectiveness of feature expression. The average pooling layer smoothes the feature map by calculating the average value of the values ​​in each pooling window, thereby improving the robustness and generalization ability of the model. 2×H×W The weight mask is obtained through 7×7 convolution, BN batch normalization layer and Sigmoid activation function layer. The mask is compared with the input feature map f C×H×W Multiply them together to achieve weighted attention processing, and get The processing flow of the other two branches is similar to that of the first branch, and finally the corresponding attention weighted feature maps are obtained. and Finally, and The original shape C×H×W is restored through the inverse permutation operation, and the three attention-weighted feature maps are added and averaged to obtain the final image feature descriptor.

[0052] By introducing the triple attention network, the intrinsic dependencies between different semantic objects in the image can be captured more accurately, and the key target objects can be highlighted, which helps to improve the performance of the model in tasks such as image classification and object detection. In addition, the triple attention network has a small number of parameters and high computational efficiency, and will not significantly increase the complexity and computational cost of the model.

[0053] In an optional embodiment of step S3, the loss is calculated using a contrast loss function or a loss function based on a multi-hard negative sample weighting strategy according to the ground image feature descriptor set and the satellite image feature descriptor set, including:

[0054] S31. For the ground image feature descriptor in the ground image feature descriptor set, use the satellite image feature descriptor in the satellite image feature descriptor set that matches the ground image feature descriptor as a positive sample, and use the rest of the satellite image feature descriptors as negative samples.

[0055] In the specific implementation, negative samples can be divided into easy negatives, semi-hard negatives and hard negatives according to the similarity with the ground image feature descriptor (reference sample). The Euclidean distance d between the easy negatives and the reference samples is a,n Significantly greater than the Euclidean distance d between the reference sample and the positive sample a,p In other words, these negative samples are the satellite images that are most visually dissimilar to the benchmark samples, with significant differences, so it is very easy to identify the non-correspondence between them. The Euclidean distance d between the semi-hard negatives and the benchmark samples a,n Although it is close to the Euclidean distance d between the benchmark sample and the positive sample a,p , but still slightly larger than d a,p These samples show a high visual similarity with the anchor image, but subtle differences can still be found after careful comparison. For the network model, it is relatively difficult to recognize such samples. The Euclidean distance d between hard negatives and the benchmark sample a,n Less than the Euclidean distance d between the reference sample and the positive sample a,p Although these samples are visually very similar to the benchmark samples, they do not actually correspond to the benchmark samples. For the network model, it is extremely difficult to identify such samples.

[0056] S32, calculating the first Euclidean distance d between the ground image feature descriptor and the positive sample a,p , calculate the second Euclidean distance d between the ground image feature descriptor and each of the negative samples a,ni .

[0057] S33. Calculate the loss according to the first Euclidean distance and the second Euclidean distance by comparing the loss function or the loss function based on the multi-difficult negative sample weighting strategy.

[0058] In the above step S33, the loss is calculated according to the first Euclidean distance and the second Euclidean distance by using a comparison loss function or a loss function based on a multi-hard negative sample weighting strategy, including:

[0059] S331, the first Euclidean distance d a,p With all the second Euclidean distance d a,ni If there are ≤1 negative samples whose second Euclidean distance is smaller than the first Euclidean distance, the loss L is calculated using the contrast loss function according to the smallest second Euclidean distance. Hu, the specific calculation process is shown in the following formula:

[0060]

[0061] Among them, d a,p means the Euclidean distance between the features of each small field of view ground image a and its corresponding positive sample p in the current batch, It represents the minimum Euclidean distance between the features of the small field of view ground image a and all negative samples in this batch B, and α represents the weight coefficient set to improve the convergence of the network.

[0062] S332: If there are ≥2 negative samples whose second Euclidean distance is smaller than the first Euclidean distance, use a loss function based on a multi-difficult negative sample weighting strategy to calculate the loss.

[0063] When there are more than or equal to 2 negative samples whose Euclidean distance is greater than the Euclidean distance of the positive sample, it means that there are multiple negative samples (i.e., hard negative samples) that are difficult to identify in the current training batch. Therefore, the embodiment of the present invention designs a loss function based on a multi-hard negative sample weighted strategy, i.e., MHNW (Multiple Hard Negatives Weighted). When multiple hard negative samples emerge in a batch, this strategy will comprehensively examine and make full use of these samples. The specific calculation process is as follows:

[0064] First, for each benchmark sample a in batch B, the first Euclidean distance between the benchmark sample a and the positive sample p is calculated; the second Euclidean distance between the benchmark sample a and all negative samples n is calculated. Through this systematic process, a B×B distance matrix D can be constructed. B×B It should be emphasized here that the batch B mentioned here refers specifically to the input B pairs of small-viewing angle ground images and small-viewing angle satellite images. For any small-viewing angle ground image in this batch, in addition to its corresponding small-viewing angle satellite image being regarded as a positive sample, all the remaining small-viewing angle satellite images in the current batch are regarded as its negative samples. Therefore, in each processing batch, the number of positive samples is fixed to 1, and the number of negative samples is correspondingly B-1.

[0065] Secondly, the distance matrix D B×B The horizontal axis of the matrix has B ground image features arranged in order, while the vertical axis has B satellite image features arranged in order. Each column of the matrix accurately represents the Euclidean distance between the small-view ground image feature and all small-view satellite image features. The diagonal elements of the matrix reflect the similarity distance between the ground panoramic image feature and the matching satellite image feature. The second Euclidean distance d between the benchmark sample a and the negative sample n is compared one by one from each column. a,ni(where i is an integer within B-1), and the first Euclidean distance d between the reference sample a and the positive sample p a,p If after comparison, it is found that there is no or only one negative sample that makes d a,ni Less than d a,p , then the method of step S331 will be used to process and calculate the value of the loss function accordingly.

[0066] Again, if there are multiple negative samples satisfying d a,ni Less than d a,p If the condition is met, further processing will be performed according to the method of step S332, specifically:

[0067] S3321, extract all the negative samples whose second Euclidean distance is less than the first Euclidean distance as hard negative samples, the number of hard negative samples is N, and calculate their respective difficulty metrics. Here, the difficulty metric is set to D i =d a,p -d a,ni (i∈N), where d a,ni The smaller the difficulty, the higher the i The bigger;

[0068] S3322. Calculate the difference between the first Euclidean distance and the second Euclidean distance of each of the hard negative samples, and calculate the weight of each of the hard negative samples according to the difference, as shown in the following formula:

[0069]

[0070] S3323: Calculate the loss of each of the difficult negative samples using a contrast loss function according to the smallest second Euclidean distance among the difficult negative samples. The calculation formula is:

[0071] S3324, calculating the product of the weight and the loss of each difficult negative sample, and summing the products of all the difficult negative samples to obtain the final loss L Proposed , the calculation formula is

[0072] Through the loss function based on the multi-difficult negative sample weighting strategy, the constructed model is more robust in complex and changeable data processing, improves the global classification accuracy, and can optimize the decision boundary at a fine-grained level, further improving the reliability when dealing with boundary conditions.

[0073] After calculating the loss value of the loss function, the pre-trained EfficientNet network and the triple attention network can be iteratively trained according to the goal of minimizing the loss. By reducing the loss value, the loss rate can be reduced and the accuracy of the positioning model can be improved.

[0074] In the specific implementation of step S3, the parameters of the pre-trained EfficientNet network and the triple attention network can be iteratively trained through the AdamW optimizer. During the training process, the AdamW optimizer is used to ensure that the network can converge quickly and achieve good performance. The batch size is set to 32 during training; the learning rate is set to 0.00001, and gradually decreases with the increase in the number of training rounds to help the model better converge to the optimal solution; the weight decay is set to 0.0005; and L2 regularization is performed on the ground image and satellite image feature descriptors finally extracted by the Aenet network.

[0075] In an optional embodiment of the above-mentioned geolocation method, the above-mentioned small field of view cross-viewing angle image geolocation method further includes:

[0076] S4. Based on the panoramic image test set, the recall rate is used as the indicator to evaluate the trained EfficientNet network and triple attention network.

[0077] Specifically, the panoramic image test set also comes from the standard dataset CVUSA, and its processing process is the same as that of the panoramic image training set. The division ratio of the panoramic image training set and the panoramic image test set in the standard dataset CVUSA is 80:20.

[0078] When evaluating the trained network, the top K recall rate is used as the evaluation index. When K is an integer, the top K set refers to the set consisting of the top K satellite images closest to the feature descriptor of a ground image to be queried; when K is a percentage, the top K set refers to the set consisting of the top K×N satellite images closest to the feature descriptor of a ground image to be queried, where N represents the number of reference satellite images. The top K recall rate refers to the ratio of the actual matching satellite images in the top K set of corresponding ground images in the ground image to be queried. The top K recall rates commonly used in traditional methods include four evaluation indicators: R@1, R@5, R@10, and R@1%. In the embodiment of the present invention, the top K recall rate is selected as Recall@1% as the evaluation index. Specifically, let the total number of images in the test sample be Np. After executing the query, the number of correct samples ranked in the top 1% of the samples retrieved from the test sample is np. Then Recall@1% can be expressed by the formula When Recall@1% reaches a higher value, it can be regarded as a valid basis for evaluation.

[0079] The small field of view cross-view image geolocation method disclosed in the embodiment of the present invention uses the EfficientNet network as the feature extraction network of the model, extracts image features with rich information and introduces a triple attention network as a feature processing module, which enhances the model's understanding and processing capabilities of image features and solves the problem of incomplete ground image features. At the same time, by using a loss function based on a multi-difficult negative sample weighting strategy to calculate the loss, the decision boundary of the network is optimized through the target pair that minimizes the loss, the classification accuracy is improved, and the learning depth of the network for difficult negative samples is enhanced.

[0080] At the same time, experimental comparisons were conducted with the ResNet18 network method on the public standard dataset CVUSA. The results showed that at various field of view angles, the EfficientNet network and triple attention network of the present invention have high positioning accuracy for ground images to be positioned, and the recall rates of the trained EfficientNet network and triple attention network are improved by an average of 10% to 20%.

[0081] Based on the same inventive concept, a small field of view cross-view image geolocation system is also provided in an embodiment of the present invention, as described in the following embodiments. Since the principle of solving the problem by the cross-view image geolocation system is similar to the above-mentioned small field of view cross-view image geolocation method, the implementation of the geolocation system can refer to the implementation of the above-mentioned geolocation method, and the repeated parts will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements predetermined functions. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0082] Figure 6 is a structural block diagram of a small field of view cross-viewing angle image geolocation system disclosed in an embodiment of the present invention, such as Figure 6 As shown, the geographic positioning system includes an image preprocessing module 601, a network training module 602 and a positioning module 603, and the structure is described below.

[0083] The image preprocessing module 601 is used to preprocess the ground image and satellite image pairs matched in the panoramic image training set to obtain a small field of view training set;

[0084] The network training module 602 is used to perform feature extraction and weighted processing on the small field of view training set using a pre-trained EfficientNet network and a triple attention network to obtain a ground image feature descriptor set and a satellite image feature descriptor set; based on the ground image feature descriptor set and the satellite image feature descriptor set, a contrast loss function or a loss function based on a multi-difficult negative sample weighting strategy is used to calculate the loss, and the pre-trained EfficientNet network and the triple attention network are iteratively trained according to the goal of minimizing the loss to obtain the trained EfficientNet network and the triple attention network; wherein, the structures of the EfficientNet network and the triple attention network are described in the above method, and the present invention will not repeat them here.

[0085] The positioning module 603 is used to use the trained EfficientNet network and triple attention network to perform feature extraction and feature weighting processing on the ground images and satellite images to be positioned that are collected in real time, obtain ground image feature descriptors and satellite image feature descriptors, perform geographic positioning according to the ground image feature descriptors and the satellite image feature descriptors, and obtain the positioning result of the ground image to be positioned.

[0086] The embodiment of the present invention uses the ResNet18 network to compare with the geolocation model constructed by the present invention through the EfficientNet network and the triple attention network. ResNet18 is a classic deep learning network that solves the gradient vanishing and representation bottleneck problems in deep neural network training by introducing residual connections, making the network deeper and the training more effective. It has a simple structure and stable performance, so it can achieve good results in many applications.

[0087] The present invention is an embodiment that uses the public standard dataset CVUSA as the input image, compares the ResNet18 network method with the method of the present invention, and compares the specific recall rates of the ResNet18 network method and this article under different field of view angles. The test results are shown in Tables 1 and 2 below:

[0088] Table 1: Performance comparison results of the proposed method and the ResNet18 model on the CVUSA dataset at 360° and 180° viewing angles

[0089]

[0090] Table 2: Performance comparison results of the proposed method and the ResNet18 model on the CVUSA dataset at 120° and 70° viewing angles

[0091]

[0092] It can be seen from Table 1 and Table 2 that the method proposed in the present invention shows higher recall accuracy than the ResNet18 method. The recall accuracy has been improved in the four key indicators of R@1, R@5, R@10 and R@1%, especially in the R@1% indicator, the recall accuracy has reached 93.25%. When the field of view is a panoramic 360°, the model in this paper improves by 24.08% on the key evaluation indicator R@1, 15.04% on R@5, 12.96% on R@10, and 17.27% on R@1% compared with the ResNet18 method; when the field of view is 180°, it improves by 12.60% on R@1, 6.85% on R@5, 10.09% on R@10, and 17.27% on R@1%. When the small field of view angle is 120°, it is improved by 16.86% on R@1, 12.68% on R@5, 7.62% on R@10, and 21.23% on R@1%. When the small field of view angle is 70°, it is improved by 10.75% on R@1, 18.48% on R@5, 19.82% on R@10, and 14.7% on R@1%. Under different field of view angles, the present invention improves the recall rate by about 10% to 20%.

[0093] At the same time, it can be found that the smaller the small field of view angle, including the ResNet18 method, the less ideal the results of the model test are, the more difficult it is to geolocate images across viewing angles with a small field of view, the lower the accuracy of the results measured by the model, and the limited performance improvement.

[0094] It can be seen from the experimental results that the small field of view cross-view image geolocation model based on EfficientNet proposed in the present invention can further improve the small field of view cross-view image geolocation.

[0095] In this embodiment, a computer device is provided, such as Figure 7 As shown, it includes a memory 701, a processor 702, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any of the above-mentioned small field of view cross-viewing angle image geolocation methods is implemented.

[0096] Specifically, the computer device may be a computer terminal, a server or a similar computing device.

[0097] In this embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program for executing any of the above-mentioned small-field-of-view cross-viewing angle image geolocation methods.

[0098] Specifically, computer-readable storage media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable storage media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0099] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0100] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for geolocation of small field-of-view cross-viewing images, characterized in that: include: Preprocess the matching ground image and satellite image pairs in the panoramic image training set to obtain a small field of view training set; The small field of view training set is subjected to feature extraction and weighted processing using a pre-trained EfficientNet network and a triple attention network to obtain a ground image feature descriptor set and a satellite image feature descriptor set; According to the ground image feature descriptor set and the satellite image feature descriptor set, a contrast loss function or a loss function based on a multi-difficult negative sample weighting strategy is used to calculate the loss, and the pre-trained EfficientNet network and the triple attention network are iteratively trained according to the goal of minimizing the loss; The trained EfficientNet network and triple attention network are used to perform feature extraction and feature weighting processing on the real-time collected ground images and satellite images to be located, so as to obtain ground image feature descriptors and satellite image feature descriptors. Geolocation is performed according to the ground image feature descriptors and the satellite image feature descriptors to obtain the positioning result of the ground image to be located.

2. The small field of view cross-viewing angle image geolocation method according to claim 1, characterized in that: The preprocessing of the matched ground image and satellite image pairs in the panoramic image training set to obtain a small field of view training set includes: Under multiple field of view angles, each panoramic ground image in the panoramic image training set is cropped at a fixed center, and the panoramic ground image and the cropped image are reconstructed into a small field of view angle ground image with the same resolution through a resize function; Under multiple field of view angles, fixed center cropping is performed on the satellite images matching each of the panoramic ground images in the panoramic image training set, and the satellite images and the cropped images are reconstructed into small field of view satellite images with the same resolution as the small field of view ground images through a resize function to obtain a small field of view training set, wherein the small field of view training set includes multiple matched pairs of small field of view ground images and small field of view satellite images, and the multiple field of view angles include 70°, 120° and 180°.

3. The small field of view cross-viewing angle image geolocation method according to claim 1, characterized in that: The EfficientNet network includes nine stages. The first stage includes a 3×3 convolutional layer, a SiLU activation function layer and a BN batch normalization layer. The second to eighth stages all include MBConv blocks. The MBConv blocks include a 1×1 convolutional layer, a Depthwise convolutional layer, a channel attention module SE, a 1×1 convolutional layer and a residual block connected in sequence. The ninth stage includes a 1×1 convolutional layer, a pooling layer and a fully connected layer.

4. The small field of view cross-viewing angle image geolocation method according to claim 1, characterized in that: The triple attention network includes a first spatial attention channel, a second spatial attention channel and a joint attention channel; The first spatial attention channel is used to sequentially perform horizontal spatial transformation, attention weighting processing and inverse transformation on the high-level semantic feature map obtained after feature extraction to obtain a horizontal spatial attention weighted feature map; The second spatial attention channel is used to sequentially perform vertical spatial transformation, attention weighting processing and inverse transformation on the high-level semantic feature map to obtain a vertical spatial attention weighted feature map; The joint attention channel is used to perform attention weighted processing on the high-level semantic feature map to obtain an attention weighted feature map, and to perform weighted averaging on the attention weighted feature map, the horizontal space attention weighted feature map, and the vertical space attention weighted feature map to obtain an image feature descriptor. The high-level semantic feature map includes a high-level semantic feature map of a ground image and a high-level semantic feature map of a satellite image, and the image feature descriptor includes a ground image feature descriptor and a satellite image feature descriptor.

5. The small field of view cross-viewing angle image geolocation method according to claim 1, characterized in that: The loss is calculated by using a contrast loss function or a loss function based on a multi-difficult negative sample weighting strategy according to the ground image feature descriptor set and the satellite image feature descriptor set, including: For the ground image feature descriptor in the ground image feature descriptor set, the satellite image feature descriptor in the satellite image feature descriptor set that matches the ground image feature descriptor is used as a positive sample, and the rest of the satellite image feature descriptors are used as negative samples; Calculating a first Euclidean distance between the ground image feature descriptor and the positive sample, and calculating a second Euclidean distance between the ground image feature descriptor and each of the negative samples; According to the first Euclidean distance and the second Euclidean distance, the loss is calculated by comparing the loss function or the loss function based on the multi-difficult negative sample weighting strategy.

6. The small field of view cross-viewing angle image geolocation method according to claim 5, characterized in that: The calculating the loss by comparing the first Euclidean distance and the second Euclidean distance through a loss function or a loss function based on a multi-difficult negative sample weighting strategy includes: Compare the first Euclidean distance with all the second Euclidean distances, if there are ≤1 of the negative samples whose second Euclidean distance is smaller than the first Euclidean distance, calculate the loss using the contrast loss function according to the smallest second Euclidean distance; If there are ≥2 of the negative samples whose second Euclidean distance is smaller than the first Euclidean distance, a loss function based on a weighted strategy of multiple difficult negative samples is used to calculate the loss.

7. The small field of view cross-viewing angle image geolocation method according to claim 6, characterized in that: The loss function based on the multi-difficult negative sample weighting strategy is used to calculate the loss, including: Extract all the negative samples whose second Euclidean distance is smaller than the first Euclidean distance as hard negative samples; Calculating a difference between the first Euclidean distance and the second Euclidean distance of each of the hard negative samples, and calculating a weight of each of the hard negative samples according to the difference; Calculating the loss of each of the hard negative samples using a contrast loss function according to the smallest second Euclidean distance among the hard negative samples; The product of the weight and the loss of each hard negative sample is calculated, and the products of all the hard negative samples are summed to obtain a final loss.

8. A small field of view cross-viewing angle image geolocation system, characterized in that: include: An image preprocessing module, wherein the image preprocessing module is used to preprocess the matching ground image and satellite image pairs in the panoramic image training set to obtain a small field of view training set; A network training module, wherein the network training module is used to perform feature extraction and weighted processing on the small field of view training set using a pre-trained EfficientNet network and a triple attention network to obtain a ground image feature descriptor set and a satellite image feature descriptor set; based on the ground image feature descriptor set and the satellite image feature descriptor set, a contrast loss function or a loss function based on a multi-difficult negative sample weighting strategy is used to calculate the loss, and the pre-trained EfficientNet network and the triple attention network are iteratively trained according to the goal of minimizing the loss to obtain a trained EfficientNet network and a triple attention network; A positioning module, wherein the positioning module is used to use the trained EfficientNet network and triple attention network to perform feature extraction and feature weighting processing on the ground images and satellite images to be positioned that are collected in real time, obtain ground image feature descriptors and satellite image feature descriptors, perform geographic positioning according to the ground image feature descriptors and the satellite image feature descriptors, and obtain the positioning result of the ground image to be positioned.

9. The small-viewing-angle cross-viewing-angle image geolocation system according to claim 8, characterized in that: The EfficientNet network includes nine stages. The first stage includes a 3×3 convolutional layer, a SiLU activation function layer and a BN batch normalization layer. The second to eighth stages all include MBConv blocks. The MBConv blocks include a 1×1 convolutional layer, a Depthwise convolutional layer, a channel attention module SE, a 1×1 convolutional layer and a residual block connected in sequence. The ninth stage includes a 1×1 convolutional layer, a pooling layer and a fully connected layer.

10. The small-viewing-angle cross-viewing-angle image geolocation system according to claim 8, characterized in that: The triple attention network includes a first spatial attention channel, a second spatial attention channel and a joint attention channel; The first spatial attention channel is used to sequentially perform horizontal spatial transformation, attention weighting processing and inverse transformation on the high-level semantic feature map obtained after feature extraction to obtain a horizontal spatial attention weighted feature map; The second spatial attention channel is used to sequentially perform vertical spatial transformation, attention weighting processing and inverse transformation on the high-level semantic feature map to obtain a vertical spatial attention weighted feature map; The joint attention channel is used to perform attention weighted processing on the high-level semantic feature map to obtain an attention weighted feature map, and to perform weighted averaging on the attention weighted feature map, the horizontal space attention weighted feature map, and the vertical space attention weighted feature map to obtain an image feature descriptor. The high-level semantic feature map includes a high-level semantic feature map of a ground image and a high-level semantic feature map of a satellite image, and the image feature descriptor includes a ground image feature descriptor and a satellite image feature descriptor.

Images