A cross-view geoview positioning method based on dynamic observation

Abstract

The application relates to a cross-view geographical view positioning method based on dynamic observation, which comprises the following steps: acquiring satellite image data, acquiring image style information of the satellite image based on the satellite image data; acquiring unmanned aerial vehicle image data to be positioned, performing style alignment processing on the unmanned aerial vehicle image data based on the image style information, and acquiring aligned image information; inputting the aligned image information and the satellite image data into a pre-trained dynamic observation model, acquiring an output feature vector, and realizing geographical view positioning based on the output feature vector. Compared with the prior art, the visual style of images in two domains is normalized by adopting a style alignment strategy, the noise of the surrounding environment is reduced, the context consistency between two views is ensured, the possibility that the surrounding environment is divided into different parts due to the height and angle changes in the unmanned aerial vehicle view image is reduced by adopting a square ring segmentation strategy, and the positioning accuracy is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of geographic positioning, in particular to a cross-view geographic view positioning method based on dynamic observation. BACKGROUND

[0002] View geo-localization is the task of matching geo-images from different views, such as drone and satellite view images. With the popularity of drones, cross-view geo-localization of drone and satellite view images has attracted a lot of attention, including autonomous vehicles, drone navigation, precision delivery, etc.

[0003] Great progress has been made in this field recently, including methods that utilize environmental information of landmark buildings and spatial feature alignment between different views. Some methods propose a square ring segmentation strategy to utilize neighborhood context and cope with the rotation changes brought by drone shooting, and some methods propose perspective projection transformation and conditional generative adversarial networks to realize the conversion between different views, which synthesizes drone views into satellite views to narrow the gap between the two views and makes full use of contextual information. Some methods propose a transformer-based structure and a heat map segmentation module to divide the image into two parts of buildings and backgrounds to align spatial features.

[0004] In the process of positioning, the noise of the surrounding environment will seriously hinder the contextual consistency between the two views, in addition, various differences such as weather, lighting conditions and camera color difference are easy to cause domain differences between different views. Therefore, there is currently a lack of a geographic view positioning method to solve the problem of unsatisfactory positioning accuracy of existing geographic view positioning methods. SUMMARY

[0005] The purpose of the present application is to overcome the defects of the prior art and provide a cross-view geographic view positioning method based on dynamic observation to solve or partially solve the problem of unsatisfactory positioning accuracy of existing geographic view positioning methods.

[0006] The purpose of the present application can be achieved by the following technical solutions:

[0007] The present application provides a cross-view geographic view positioning method based on dynamic observation, comprising the following steps:

[0008] Obtain satellite image data, and based on the satellite image data, obtain image style information of the satellite image;

[0009] Obtain drone image data to be positioned, perform style alignment processing on the drone image data based on the image style information, and obtain aligned image information;

[0010] Input the aligned image information and the satellite image data into a pre-trained dynamic observation model to obtain an output feature vector, and realize geographic view positioning based on the output feature vector,

[0011] The dynamic observation model comprises:

[0012] An image segmentation network is configured to obtain corresponding central building image information and surrounding environment image information based on the input satellite image data and the aligned image information.

[0013] A feature extraction network based on hierarchical attention is configured to dynamically balance the central building image information and the surrounding environment image information and obtain fused feature information.

[0014] A generalized pooling network is configured to obtain the output feature vector based on the fused feature information.

[0015] As a preferred technical solution, the image style information is obtained by the following steps:

[0016] The color distribution of each satellite image is calculated in an offline manner, and the image style information is obtained by averaging the color distribution corresponding to each satellite image.

[0017] As a preferred technical solution, the color distribution is obtained by the following formula:

[0018] E c (x) = H c (x) / sum(H c (x))

[0019]

[0020] M(x) = [P R (x), P G (x), P B (x)]

[0021] In the formula, E c (x), H c (x), and P c (x) are the probability distribution, histogram, and cumulative distribution of the color channel C of the satellite image, respectively, wherein the color channel C is any one of red, green, and blue, and M(x) is the color distribution.

[0022] As a preferred technical solution, the image style information is obtained by the following formula:

[0023] I' = M(I) = [P R (I R ), PG (I G ), P B (I B )]

[0024] In the formula, I represents an input unmanned aerial vehicle image, I' represents a mapping output, I R , I G , I B respectively represent images of the unmanned aerial vehicle image in red, green and blue channels.

[0025] As a preferred technical solution, the acquisition of the fused feature information comprises the following steps:

[0026] Based on the central building image information, a first attention map is acquired by performing a maximum operation in the channel dimension;

[0027] Based on the central building image information and the surrounding environment image information, a second attention map is acquired by performing a mean operation in the channel dimension;

[0028] Based on the first attention map and the second attention map, the fused feature information is acquired.

[0029] As a preferred technical solution, the output feature vector is acquired by the following formula:

[0030]

[0031] In the formula, f k , k∈(1, 2,.., K) represents an output feature of a generalized pooling layer of channel k, F k is an output feature of channel K of the feature extraction network, θ is a feature of each position of F k , f Gem is the output feature vector, and p is a hyperparameter.

[0032] As a preferred technical solution, the dynamic observation model further comprises a classification network, which is used to calculate a loss function value based on the output feature vector corresponding to the aligned image information and the output feature vector corresponding to the satellite image data in a training process, and the pre-trained dynamic observation model is obtained after the loss function value reaches a preset convergence condition.

[0033] As a preferred technical solution, the acquisition of the loss function value comprises the following steps:

[0034] Based on the output feature vector corresponding to the aligned image information and the output feature vector corresponding to the satellite image data, a spliced vector is acquired by connection and normalization, and a center loss is calculated;

[0035] Based on the spliced vector, cross entropy loss and deconstruction loss are obtained through label prediction;

[0036] Based on the center loss, cross entropy loss and deconstruction loss, the loss function value is obtained.

[0037] As a preferred technical solution, the deconstruction loss is obtained by the following formula:

[0038]

[0039] In the formula, S is the covariance matrix of the prediction result, S ij is the element of the i-th row and j-th column of S, and lambda is used to control the contribution of the non-diagonal element.

[0040] As a preferred technical solution, the center building image information is rectangular, and the surrounding environment image information is annular.

[0041] Compared with the prior art, the present application has the following advantages:

[0042] (1) Before feature extraction, the visual style of images in two domains is normalized by adopting a style alignment strategy, the noise of the surrounding environment is reduced, the context consistency between two perspectives is ensured, a dynamic observation model is provided, feature extraction is realized in a human way, the segmented image is segmented in a novel way, a light-weight structure design is used to design a hierarchical attention module to simulate human observation, an image segmentation network is set to reduce the possibility of segmenting the surrounding environment into different parts due to the height and angle changes in the unmanned aerial vehicle view image, an attention mechanism is used to dynamically balance the center building image and the surrounding environment image, a generalized average pooling is introduced to capture the discriminative features of the two square ring partitions, thereby effectively improving the recognition accuracy, and solving or partially solving the problem that the positioning accuracy of the existing geographic view positioning method is not ideal.

[0043] (2) A deconstruction loss function is provided to reduce the correlation of geographic label features and fully utilize the output vector of the classifier for supervision, thereby effectively accelerating the model training efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 It is a framework schematic diagram of the cross-perspective geographic view positioning method based on dynamic observation in embodiment 1;

[0045] Figure 2 It is a schematic diagram of the satellite image style based on statistics in the style alignment strategy;

[0046] Figure 3 It is a schematic diagram of the hierarchical attention module;

[0047] Figure 4 It is a schematic diagram of the two square ring partition strategy;

[0048] Figure 5 Attention visualization diagram for models of different heights and different degrees of architectural prominence. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work should fall within the protection scope of the present application.

[0050] In the embodiments, the terms "first" and "second" are only used for description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first" and "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more than two, unless otherwise specifically limited.

[0051] Embodiment 1

[0052] As Figures 1-5 described, the present embodiment provides a cross-view geographical view positioning method based on dynamic observation. Before feature extraction, the present method adopts a style alignment strategy to normalize the visual style of images in two domains. The specific method is to count the overall color distribution under the satellite view angle, and the formula is as follows:

[0053]

[0054] wherein E c (x), H c (x) and P c (x) are the probability distribution, histogram and cumulative distribution of the color channel C C (red, green, blue). Through the gray statistical distribution of the three channels, M(x) is the visual style feature of the satellite view.

[0055] Considering that a single image cannot represent all satellite images, we calculate the entire training set to obtain the average mapping function as formula (2), wherein M(x k ) is the transformed image of the k-th satellite image x k , and N is the number of satellite images in the training set. As Figure 2 described, the satellite image style is obtained based on statistics.

[0056]

[0057] Then the style transfer of the drone image is as formula (3), where I represents the input drone image, I ′ represents the mapping output. Thus, the drone image is transferred to the satellite visual style.

[0058] I ′ = M(I) = [P R (I R ), P G (I G ), P B (I B )] (3)

[0059] By the style alignment strategy, the difference between the drone view and the satellite view is reduced, while the different weather conditions under the drone view are also unified into a visual style.

[0060] After the preprocessing by the above method, we propose a dynamic observation module to extract features by imitating human observation patterns and solving the problem of constantly changing drone perspectives and surrounding environmental noise. It mainly includes ResNet GL, two square ring partition strategies and generalized pooling layer parts.

[0061] Firstly, the features are extracted based on ResNet-GL, which embeds a hierarchical attention module into ResNet-50 and partitions the image in a novel two-square ring manner. We use a lightweight structure to design the hierarchical attention module to simulate human observation patterns. In addition, the two-layer square ring partition strategy can reduce the possibility of partitioning the surrounding environment into different parts due to the height and angle changes in the drone view image, and at the same time, the generalized average pooling is introduced to capture the discriminative features of the two square ring partitions. As Figure 3 shown in the structure diagram of the hierarchical attention module.

[0062] As Figure 4 shown in the diagram of the two-square ring partition strategy, the two-square ring partition strategy divides the image into two parts: the central building and the surrounding environment, so that it can explicitly process and fuse the features of the two parts.

[0063] The most important part in ResNet GL is the lightweight hierarchical attention block. Its design is based on the idea that humans prefer to focus on the central building in cross-view geo-localization and may only glance at the surrounding environment. Therefore, the hierarchical attention module is designed as a dual-branch structure, one branch performs the maximum operation in the channel dimension of the central region, and the other branch performs the mean operation in the channel dimension of the entire region, thereby obtaining two attention maps, which are then spliced together through a 5x5 convolution layer, batch normalization and a sigmoid activation function. Therefore, the salient features of the central building can be extracted through the first branch, while if the central part does not have prominent features, the second branch retains the surrounding information. As shown in FIG. 2, the model attention visualization diagram for different heights and different building saliency degrees is shown. Figure 5

[0064] The ResNet GL structure is based on the design of ResNet-50, but embeds hierarchical attention after its convolution block 3 and convolution block 5. The feature map size of the ResNet GL output is a 2048*16*16 tensor, in which the central part occupies a feature size of 2048x8x8, and the remaining features represent the surrounding environment three times the size of the central feature. If the global average pool is applied at the top of the feature map, important information about the surrounding environment will be diluted by noise. Therefore, we introduce the generalized average pooling here, which is calculated as follows:

[0065]

[0066] where f k , k∈(1, 2,.., K) represents the output feature of the generalized pooling layer of channel k. F k is the channel K of the output feature of ResNet GL, and θ is the feature of each position of F k , f Gem is the output of the generalized pooling. p is a hyperparameter. When p tends to infinity, the generalized pooling will be approximately maximum pooling. Conversely, when p = 1, it will degenerate into average pooling. The default value is set to 3 in this embodiment.

[0067] The present embodiment provides a deconstruction loss function to reduce the correlation of geo-label features during the training process and make full use of the output vector of the classifier for supervision.

[0068] ​After feature extraction, the classification module is used to supervise the feature learning of each building. Specifically, the feature maps of the drone view and satellite view images are concatenated and normalized, and then used to calculate the center loss. The center loss learns the center of the deep features of each class and simultaneously penalizes the distance between the deep features and their corresponding class center. Subsequently, two fully connected layers are applied to compress the feature dimension from 4096 to 2048 and predict the label, respectively. Then, the cross-entropy loss and the disentangled loss are applied. The cross-entropy loss cares more about the accuracy of the true label. It only increases the confidence of the true prediction, but ignores the results of other predictions. To further reduce the relevance of landmarks and extract the most unique features of each landmark, we propose the disentangled loss as formula (5), where S is the covariance matrix of the prediction results, S ii is the diagonal element of S, Sij is the non-diagonal element of S. λ is used to control the contribution of non-diagonal elements, which is set to 0.2. It is obvious that the disentangled loss will encourage the diagonal elements to be 1 and the non-diagonal values to be 0, thereby reducing the relevance of different landmarks.

[0069]

[0070] In the retrieval stage, we respectively input the picture to be retrieved and the gallery into the model, while removing the classifier module, so as to obtain the feature vectors of the to-be-retrieved picture and the gallery respectively. The Euclidean distance of each picture in the to-be-retrieved picture and the gallery is calculated, and the Euclidean distance is sorted from small to large. The picture with the smallest Euclidean distance is the Recall@1, the first recalled picture. That is, by comparing the feature distances in this way, the cross-view relocalization is completed.

[0071] This embodiment particularly considers three aspects: (1) how to avoid the noise of the surrounding environment. Noise will seriously hinder the context consistency between the two views; (2) how to reduce the domain difference of different views caused by various differences such as weather, lighting conditions and camera color difference; (3) how to design a structured model with light-weight attention block, so that its performance is not limited by the device computing power.

[0072] To this end, the embodiment provides a cross-view geo-localization method based on dynamic observation. The method normalizes the visual style of images in two domains by adopting a style alignment strategy, reduces the noise of the surrounding environment, ensures the context consistency between two views, provides a dynamic observation model, realizes feature extraction in a human way, segments images in a novel way, simulates human observation by designing a light-weight hierarchical attention module, sets an image segmentation network to reduce the possibility of segmenting the surrounding environment into different parts due to the height and angle changes in the UAV view image, introduces generalized average pooling to capture the discriminative features of two square ring partitions, thereby effectively improving the recognition accuracy, and solves or partially solves the problem of unsatisfactory positioning accuracy of the existing geo-localization method.

[0073] To verify that the method has good robustness in transfer learning, first, evaluate our method and compare it with some of the most advanced methods on University1652 and Sues-200 datasets. Subsequently, the results of transfer learning also prove the robustness of our method. The evaluation indicators we use are Recall@1 and AP. Recall@1 represents the probability of correctly matching the first ranked image. Therefore, it is sensitive to the position of the first true matching image. AP considers the position of all true matching images in the evaluation, which is a measure of the overall performance of the model.

[0074] In comparison with other advanced methods, as described in Table 1, our method achieved the best performance in all comparisons, where Drone->Satellite is the retrieval of satellite view from UAV view, and Satellite->Drone is the retrieval of UAV view from satellite view. Our method achieved 84.08% Recall@1 accuracy from UAV to satellite view and 86.39% AP, as well as 91.44% Recall@1 accuracy from satellite to UAV and 82.02% AP. Its performance is better than LPN, with an improvement of more than 9% Recall@1 and 7% AP.

[0075] Table 1 Comparison of the method with the method of University1652

[0076]

[0077] Transfer learning can verify the robustness and effectiveness of our method in cross-domain cross-view geolocalization. We implement the experiment by transferring the model trained on University-1652 to Sues-200 to evaluate its cross-view geolocalization ability. In the Sues-200 dataset, the UAV height is divided into 150 meters, 200 meters, 250 meters, and 300 meters, and we will evaluate them separately. In preprocessing, we use the style alignment strategy on the Sues200 dataset. In particular, our style alignment strategy helps to quickly align the visual features between the UAV and satellite view images in different datasets, and the hierarchical attention block makes good use of environmental information. The performance of our model is shown in Table 2. The SE ResNet, LPN and our method (with or without style alignment strategy) are listed for comparison.

[0078] Table 2 Performance of the model provided by the present method

[0079]

[0080] As can be seen, compared with LPN, this means that the hierarchical attention module is robust in cross-domain feature alignment. Then, after adding SAS, our performance is improved by more than 7%, 5%, 3% and 2% in the experiments of 300 meters, 250 meters, 200 meters and 150 meters flight height, respectively. This shows that the style transfer strategy is an effective method to reduce the domain gap, and is also effective in transfer learning.

[0081] Therefore, the experimental results also prove that our method also has good robustness in transfer learning.

[0082] Embodiment 2

[0083] The embodiment provides an electronic device, comprising: one or more processors and a memory, the memory having one or more programs stored therein, the one or more programs comprising instructions for performing the cross-view geolocalization method based on dynamic observation as described in embodiment 1.

[0084] Embodiment 3

[0085] The embodiment provides a computer-readable storage medium comprising one or more programs for execution by one or more processors of an electronic device, the one or more programs comprising instructions for performing the cross-view geolocalization method based on dynamic observation as described in embodiment 1.

[0086] The above merely illustrates the specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any skilled person in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements shall be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A cross-view geographic view positioning method based on dynamic observation, characterized in that, Includes the following steps: Acquire satellite image data, and based on the satellite image data, obtain image style information of the satellite images; Acquire image data of the drone to be located, perform style alignment processing on the drone image data based on the image style information, and obtain aligned image information; The aligned image information and the satellite image data are input into a pre-trained dynamic observation model to obtain an output feature vector. Geographic view localization is then achieved based on the output feature vector. The dynamic observation model includes: Image segmentation network is used to obtain corresponding central building image information and surrounding environment image information based on input satellite image data and aligned image information; A feature extraction network based on hierarchical attention is used to dynamically balance the central building image information and the surrounding environment image information and obtain fused feature information. A generalized pooling network is used to obtain the output feature vector based on the fused feature information. The acquisition of the fused feature information includes the following steps: Based on the central building image information, a first attention map is obtained by performing a maximum value operation on the channel dimension; Based on the central building image information and the surrounding environment image information, a second attention map is obtained by performing a mean operation on the channel dimension; The fused feature information is obtained based on the first attention map and the second attention map.

2. The cross-view geographic view positioning method based on dynamic observation according to claim 1, characterized in that, The acquisition of image style information includes the following steps: The color distribution of each satellite image is calculated offline, and the image style information is obtained by averaging the color distribution of each satellite image.

3. The cross-view geographic view positioning method based on dynamic observation according to claim 2, characterized in that, The color distribution is obtained using the following formula: ， In the formula, , and These represent the probability distribution, histogram, and cumulative distribution of color channel C in the satellite image, where color channel C can be any one of red, green, or blue. The color distribution is described.

4. The cross-view geographic view positioning method based on dynamic observation according to claim 1, characterized in that, Based on the image style information, style alignment processing is performed on the UAV image data to obtain the aligned image information using the following formula: ， In the formula, This represents the input drone image. Indicates the mapped output. , , These represent images of the drone in the red, green, and blue channels, respectively.

5. The cross-view geographic view positioning method based on dynamic observation according to claim 1, characterized in that, The output feature vector is obtained by the following formula: ， In the formula, k∈(1,2,...,K) represents the output feature of the generalized pooling layer for channel k. These are the output features of channel K of the feature extraction network. yes Features at each position, It is the output feature vector. This is a hyperparameter.

6. The cross-view geographic view positioning method based on dynamic observation according to claim 1, characterized in that, The dynamic observation model also includes a classification network, which calculates the loss function value based on the output feature vector corresponding to the aligned image information and the output feature vector corresponding to the satellite image data during the training process. After the loss function value reaches the preset convergence condition, the pre-trained dynamic observation model is obtained.

7. The cross-view geographic view positioning method based on dynamic observation according to claim 6, characterized in that, Obtaining the loss function value includes the following steps: Based on the output feature vector corresponding to the aligned image information and the output feature vector corresponding to the satellite image data, the stitched vector is obtained by concatenation and normalization, and the center loss is calculated. Based on the concatenated vector, cross-entropy loss and deconstruction loss are obtained through label prediction. The loss function value is obtained based on the center loss, cross-entropy loss, and deconstruction loss.

8. The cross-view geographic view positioning method based on dynamic observation according to claim 7, characterized in that, The deconstruction loss is obtained using the following formula: ， In the formula, It is the covariance matrix of the prediction results. yes The Line 1 Column elements, Used to control the contribution of off-diagonal elements.

9. The cross-view geographic view positioning method based on dynamic observation according to claim 1, characterized in that, The central building image information is rectangular, and the surrounding environment image information is circular.

Images