Real-time matching geographic positioning method and system for cross-view images based on deep learning

Abstract

The present invention provides a real-time matching geographic positioning method and system for cross-viewing images based on deep learning, including using a domain alignment algorithm to preliminarily align an air-viewing image with a street view image; constructing a twin neural network, the structure of which includes two convolutional neural network branches for extracting feature maps, followed by a global description vector encoding module, and then a distance measurement layer; training the twin neural network based on an existing data set containing street view images and remote sensing images; constructing a data set for a specific application scenario, and continuing to perform fine-tuning training on the basis of the trained twin neural network; capturing street view pictures on site, and using the trained network to retrieve matching satellite images in the constructed data set, thereby completing the positioning of the street view pictures. The present invention can be applied to cross-viewing image geographic positioning in various scenarios, and has a certain degree of improvement in the accuracy and robustness of positioning in practical applications.

Description

Technical Field

[0001] The present invention relates to the field of computer vision image positioning technology, and in particular to a deep learning-based real-time matching geo-positioning method and system for ground-to-air cross-view images. Background Art

[0002] Image geolocation refers to the problem of determining the shooting location of an image based solely on visual information. This type of problem has a wide range of application scenarios in the field of computer vision, such as mobile terminal navigation, AR / VR, car autonomous driving, and video geographic trajectory generation. The traditional image positioning method is based on matching street view images with street view image databases. In recent years, with the continuous development of remote sensing satellites, a large number of satellite images with geographic location information have been collected. Therefore, the problem of estimating the geographic location of the captured image is converted into a matching problem between ground-view images and aerial-view images, that is, the geographic location of the captured image is determined by matching satellite images with geographic coordinates, thereby achieving image positioning on a global scale.

[0003] At present, the mainstream image geolocation methods are mainly divided into three steps: image feature extraction, image global description vector encoding, and image retrieval and matching. In the field of academic research. With the continuous development of deep learning technology, Workman and Jacobs first introduced deep features into image positioning and matching tasks. In terms of image feature extraction, Workman and other authors first proposed using convolutional neural networks to extract feature maps from query images, and then used ordinary fully connected neural networks to encode feature maps into global description vectors. Finally, they retrieved the street view image with the closest global description vector from the constructed street view database as the matching image. In the encoding of the image global description vector, in order to allow the network structure to learn the rotation-invariant features of the image, Hu et al. improved the VLAD pooling algorithm by introducing soft assignment to make it a differentiable algorithm, that is, the module can be trained end-to-end through the back-propagation algorithm, and NetVLAD is embedded in the convolutional neural network for cross-view image matching. In image retrieval and matching, Vo, Hays and other authors used the Euclidean distance of the global description vector as the basis for measuring the similarity between images. That is, the larger the Euclidean distance between the global description vectors, the lower the similarity between the two images, which was widely adopted by subsequent scholars.

[0004] However, the existing cross-view matching methods of ground images and aerial view images fail to take into account the huge domain differences and scale differences between the two view images. Specifically, firstly, ground images are generally street view images in the horizontal direction, which often obtain the side information of the object, while aerial view images are vertical downward perspectives, which often capture the upper surface information of the object; secondly, ground images are usually street view images of a small range, while aerial view images are generally remote sensing satellite images covering a large area. The present invention starts with the main problems existing in the matching of these two cross-view images, and first proposes a domain alignment algorithm to roughly align the spatial pattern of the ground view with the remote sensing satellite image, thereby accelerating the matching process and improving the matching accuracy. Then, the present invention proposes a multi-scale attention encoder that fully considers the multi-scale information of the two images and encodes it into the global description vector of the image to improve the overall robustness of the model.

[0005] In the field of practical application, Baidu has developed a location recognition technology that locates the location of a picture. Users upload the pictures to be identified to the server. When the server receives the pictures to be located, it encodes the pictures into a one-dimensional global description vector, and then compares this information with the information in its own landmark database, and uses a certain algorithm to identify the famous Chinese and foreign landmarks and scenic spots that appear in the photos. However, this positioning method requires that the images uploaded by users must contain famous architectural landmarks around the world, and the positioning accuracy is low, which cannot meet actual needs. Summary of the invention

[0006] In order to solve the problems of slow matching speed, low matching accuracy and small coverage of cross-perspective image positioning, the present invention discloses a deep learning-based ground-to-air cross-perspective image real-time matching geolocation method and system, which can perform geolocation using only image visual information and has the characteristics of fast matching speed, high positioning accuracy and wide coverage.

[0007] The technical solution adopted by the present invention to solve the technical problem is a real-time matching geolocation method for cross-viewing images based on deep learning, which includes the following steps:

[0008] Step S1, using a domain alignment algorithm to preliminarily align the air view image with the street view image;

[0009] Step S2, constructing a twin neural network, wherein the structure of the twin neural network includes two convolutional neural network branches for extracting feature maps, followed by a global description vector encoding module, and then a distance measurement layer;

[0010] Step S3, training the twin neural network described in step S2 based on an existing data set including street view images and remote sensing images;

[0011] Step S4, constructing a data set for a specific application scenario, and continuing to perform fine-tuning training based on the twin neural network trained in step S3;

[0012] Step S5, capture a street view picture on site, and use the trained network to retrieve a matching satellite image from the data set constructed in step S4, thereby completing the positioning of the street view picture.

[0013] Moreover, the domain alignment algorithm in step S1 is implemented as follows:

[0014] The street view image is approximately converted to the inverse polar coordinate transformation of the air view image, and the conversion is performed using the following formula,

[0015]

[0016]

[0017] in, is the pixel coordinate of the original street view image, is the pixel coordinate of the synthetic empty view image, W a and H a are the width and height of the original empty view image, W s and H s are the width and height of the original street view image respectively.

[0018] Alternatively, the domain alignment algorithm in step S1 is implemented as follows:

[0019] The sky view image is approximately converted into a street view image using the following formula:

[0020]

[0021]

[0022] in, is the original empty view image pixel coordinate, is the pixel coordinate of the synthetic street view image, W a and H a are the width and height of the original empty view image, W s and H s are the width and height of the original street view image respectively.

[0023] Moreover, the specific structure of the twin neural network in step S2 is implemented as follows:

[0024] Two convolutional neural network branches for extracting feature maps, using a network composed of convolutional layers and pooling layers;

[0025] The global description vector encoding module is a fully connected layer, where the number of nodes in each layer is set according to the actual application scenario;

[0026] The distance metric layer calculates the Euclidean distance of the global description vector.

[0027] Moreover, the network training of step S3 is implemented as follows:

[0028] During each training, the network is trained based on triplet sample pairs. A street view image and a satellite image taken from the same location are randomly selected from the training set as a positive sample pair. A satellite image from another location is selected to form a negative sample pair with the previously selected street view image. These three images form a triplet and are input into the network.

[0029] The sample is input into the network, and passes through the feature extraction layer, the global description vector encoding layer, and the distance measurement layer in sequence. Finally, the distance between the sample pairs is obtained and brought into the weighted soft distance sorting loss;

[0030] The network backpropagates according to the loss function to automatically adjust the parameters.

[0031] Furthermore, in step S5, the geographical location of the street view image is determined using the trained network, and the implementation method is as follows:

[0032] Input the street view images and satellite image datasets into the trained network;

[0033] The network automatically encodes the two-view images into a one-dimensional global description vector and calculates the distance between the two vectors. Finally, it retrieves the satellite image closest to the query street view image feature from the satellite image dataset and assigns its location coordinates to the query street view image.

[0034] On the other hand, the present invention provides a cross-view image real-time matching geolocation system based on deep learning, which is used to implement the cross-view image real-time matching geolocation method based on deep learning as described above.

[0035] Furthermore, the following modules are included,

[0036] The first module is used to preliminarily align the air view image with the street view image using a domain alignment algorithm;

[0037] The second module is used to construct a twin neural network, the structure of which includes two convolutional neural network branches for extracting feature maps, followed by a global description vector encoding module, and then a distance measurement layer;

[0038] The third module is used to train the twin neural network obtained in the second module based on the existing data set including street view images and remote sensing images;

[0039] The fourth module is used to build a data set for a specific application scenario and continue to perform fine-tuning training based on the twin neural network trained in the third module;

[0040] The fifth module is used to capture street view images on site and use the trained network to retrieve matching satellite images from the data set constructed in the fourth module, thereby completing the positioning of the street view images.

[0041] Alternatively, it includes a processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the above-mentioned cross-view image real-time matching geolocation method based on deep learning.

[0042] Alternatively, it includes a readable storage medium having a computer program stored thereon, and when the computer program is executed, the above-mentioned cross-view image real-time matching geolocation method based on deep learning is implemented.

[0043] The present invention uses a domain alignment algorithm to reduce the difficulty of cross-view image matching, and uses a multi-scale attention twin neural network to match and locate ground-air perspective images. The research on this problem can be used for unmanned driving navigation, AR / VR scene positioning, and mobile device navigation in areas with weak GPS signals. The multi-scale attention twin neural network can fully consider the scale differences between ground-air perspective images. Through a large amount of training, it is hoped that the problem of image positioning based on perspective information can be solved.

[0044] The present invention can be applied to cross-viewing image geolocation in various scenarios, and improves the accuracy and robustness of positioning in practical applications to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a flow chart of a method according to an embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram of a single training process of a twin neural network according to an embodiment of the present invention.

[0047] Figure 3 is a schematic diagram of the structure of a multi-scale attention encoding network according to an embodiment of the present invention;

[0048] Figure 4 It is a schematic diagram of a domain alignment algorithm according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme in the embodiment of the present application is clearly and completely described below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work should belong to the scope of protection of this application.

[0050] With the continuous development of remote sensing satellites, a large number of images with geographic data tags have been collected. Therefore, the problem of estimating the geographic location of the captured image is converted into a problem of matching ground-based images with air-based satellite images, that is, determining the geographic location of the captured image by matching satellite images with geographic coordinates, thereby achieving image positioning on a global scale. Due to the huge perspective difference between ground-to-air perspective images, predicting the geographic location of a photo is a very arduous task. Therefore, the present invention provides a unified cross-perspective image geolocation process based on deep learning technology, which is specifically used to solve the problems existing in cross-perspective image positioning, and finally obtains a cross-perspective image positioning standard process with high positioning accuracy, simple and easy deployment, and strong anti-noise ability. This process is mainly divided into four stages. The first is the domain alignment algorithm, which uses polar coordinate transformation to convert the aerial view image into a street view image or inverse polar coordinate transformation to convert the street view perspective into an aerial view perspective image, so as to reduce the domain difference between the two; the second is the image global description vector extraction stage, training the neural network to encode images of different perspectives to obtain a more robust image global vector descriptor; then there is the large-scale aerial view image database construction stage, that is, using the trained neural network to convert the aerial view image with geographic location tags into a global description vector and build a spatial index for subsequent retrieval and matching; the last is the image matching and positioning stage, that is, using the trained network to convert the captured street view image into a global description vector, selecting the candidate aerial view image with the highest matching similarity in the database as the matching image, and then assigning the matched aerial view image geographic coordinates to the street view image to complete the cross-view image geographic positioning.

[0051] like Figure 1 As shown, the embodiment provides a cross-view image positioning method based on deep learning, and the specific steps are as follows:

[0052] Step S1: using a domain alignment algorithm to preliminarily align the air view image with the street view image to reduce the difficulty of training;

[0053] See also Figure 4 There are two main domain alignment algorithms in step S1. You can select one of them when performing domain alignment:

[0054] S1.1: The street view image is approximately converted to the inverse polar coordinates of the sky view image using the following formula:

[0055]

[0056]

[0057] in, is the pixel coordinate of the original street view image, is the pixel coordinate of the synthetic empty view image, Wa and H a are the width and height of the original empty view image, W s and H s are the width and height of the original street view image respectively.

[0058] S1.2: The sky view image is approximately converted into a street view image using the following formula:

[0059]

[0060]

[0061] in, is the original empty view image pixel coordinate, is the pixel coordinate of the synthetic street view image, W a and H a are the width and height of the original empty view image, W s and H s are the width and height of the original street view image respectively.

[0062] Step S2: Construct a multi-scale attention twin neural network, whose structure is: two convolutional neural network branches with completely identical structures but no shared weights to extract feature maps, followed by a multi-scale attention encoding module, and then a distance measurement layer;

[0063] S2.1: The two-branch feature extraction network is a convolutional neural network, and all networks composed of convolutional layers and pooling layers including VGGNet, ResNet, DenseNet, etc. can be used.

[0064] The twin neural network preferably used in the embodiment is as follows Figure 3 As shown, each convolutional neural network branch is Figure 3 As shown in the specific structure in the middle and lower part, the VGG16 network with the original classification layer removed is used as the skeleton network to extract the discriminative features of the image. The process of extracting features by the VGG16 network is divided into five stages. Each stage consists of a set of convolution kernels of different sizes and a maximum pooling layer, and finally outputs different scales. i (i∈{1,2,...,5}) feature map, the feature map gradually decreases and the receptive field gradually increases, so each stage contains different levels of information, the shallow layer of the network contains the detailed information of the image, and the deep layer contains the overall semantic information. The present invention uses multiple groups of fixed-size convolution kernels g p To explicitly mine the context information of the feature space, a set of 3×3, 5×5 and 7×7 convolution kernels are used to capture the context information of the feature space at different scales in order to mine multi-scale information. The multi-scale information extraction formula is as follows:

[0065] d=∪(d i )

[0066] s=∪(g p (d)+c)

[0067] m=f(s)

[0068] s′=m×s

[0069] Among them, ∪() represents the channel splicing operation, d i Represents the feature map of the input i-th scale, g p (d) represents the output of the p∈{3,5,7}th group of convolution kernels, c represents the bias constant, s represents the original multi-scale feature map, f() represents a 1×1 convolution operation with an output channel of 1, m represents the attention mask, and s′ represents the final multi-scale feature map.

[0070] S2.2: The global description vector encoding module is a fully connected layer, where the number of nodes in each layer is set according to the actual application scenario. Its main function is to encode the feature map extracted by S2.1 into a one-dimensional description vector. Assume that f1 is the global description vector of the street view image and f2 is the global description vector of the satellite image.

[0071] S2.3: The distance metric layer is the Euclidean distance of the global description vector. The distance between two vectors is calculated using the following formula:

[0072]

[0073] Among them, j, d, and n are the dimension labels of the vector, the distance between the two vectors, and the total length of the vector, i.e., f1 (j) represents the j-th dimension of the vector, f2 (j) Represents the j-th dimension of a vector, j=1,2,…,n.

[0074] Step S3: Train the twin neural network described in step S2 on the CVUSA dataset, which is a large dataset containing tens of thousands of pairs of ground and aerial / satellite images from various places. The remote sensing airspace images with geographic coordinates and the corresponding ground street view images in this dataset are collected from Google Street View and Bing Maps. The training set of this dataset contains 35,532 pairs of street view images and remote sensing images, and the test set contains 8,884 pairs of street view images and remote sensing images for verification. In addition, the street view images in CVUSA also provide semantic segmentation labels. Since the method of this paper does not rely on any other additional information, the semantic segmentation labels are not used;

[0075] like Figure 2 As shown, the specific network training steps in step S3 are:

[0076] S3.1: During each training, the network is trained according to triplet sample pairs. A street view image and a satellite image taken from the same location are randomly selected from the training set as a positive sample pair. Then a satellite image from another location is selected to form a negative sample pair with the previously selected street view image. These three images form a triplet and are input into the network.

[0077] S3.2: Input the sample into the network, pass through the feature extraction layer, the global description vector encoding layer, and the distance measurement layer in turn, and finally obtain the distance between the sample pairs and bring it into the weighted soft distance sorting loss:

[0078]

[0079] Among them, α is the weighting constant 10, d pos is the distance between positive sample pairs, d neg is the distance between negative sample pairs, and e is the base of the natural logarithm function.

[0080] S3.3: The network backpropagates according to the loss function to automatically adjust the parameters.

[0081] Step S4: construct a data set for a specific application scenario, and continue to perform fine-tuning training based on the twin neural network trained in step S3;

[0082] The steps of constructing a data set for a specific application scenario and further training in step S4 are:

[0083] S4.1: Take several street view images of different locations at the actual site according to application requirements, and form a small dataset with remote sensing images of the corresponding locations.

[0084] S4.2: The training method is the same as that described in step S3, but the initial learning rate is reduced on this small dataset and the network is further trained.

[0085] Step S5: capture street view images on site, and use the trained network to retrieve matching satellite images from the dataset constructed in S4, thereby completing the positioning of the street view images.

[0086] The step of using the trained network to determine the geographical location of the street view image in step S5 is:

[0087] S5.1: Input the captured street view images and images in the satellite image database into the trained network.

[0088] S5.2: The network automatically encodes the two-view images into a one-dimensional global description vector and calculates the distance between the two vectors. Finally, it retrieves the satellite image closest to the query street view image feature from the satellite image dataset and assigns its location coordinates to the query street view image.

[0089] In specific implementation, the method proposed in the technical solution of the present invention can be implemented by technical personnel in this field using computer software technology to realize automatic operation process. System devices for implementing the method, such as computer-readable storage media storing the corresponding computer program of the technical solution of the present invention and computer equipment running the corresponding computer program, should also be within the scope of protection of the present invention.

[0090] In some possible embodiments, a cross-view image real-time matching geolocation system based on deep learning is provided, comprising the following modules:

[0091] The first module is used to preliminarily align the air view image with the street view image using a domain alignment algorithm;

[0092] The second module is used to construct a twin neural network, the structure of which includes two convolutional neural network branches for extracting feature maps, followed by a global description vector encoding module, and then a distance measurement layer;

[0093] The third module is used to train the twin neural network obtained in the second module based on the existing data set including street view images and remote sensing images;

[0094] The fourth module is used to build a data set for a specific application scenario and continue to perform fine-tuning training based on the twin neural network trained in the third module;

[0095] The fifth module is used to capture street view images on site and use the trained network to retrieve matching satellite images from the data set constructed in the fourth module, thereby completing the positioning of the street view images.

[0096] In some possible embodiments, a cross-view image real-time matching geolocation system based on deep learning is provided, including a processor and a memory, the memory being used to store program instructions, and the processor being used to call the stored instructions in the memory to execute a cross-view image real-time matching geolocation method based on deep learning as described above.

[0097] In some possible embodiments, a cross-view image real-time matching geolocation system based on deep learning is provided, including a readable storage medium, on which a computer program is stored. When the computer program is executed, a cross-view image real-time matching geolocation method based on deep learning as described above is implemented.

[0098] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A real-time matching geolocation method for cross-view images based on deep learning, characterized in that: The following steps are involved: Step S1, using a domain alignment algorithm to preliminarily align the air view image with the street view image; Step S2, constructing a twin neural network, wherein the structure of the twin neural network includes two convolutional neural network branches for extracting feature maps, followed by a global description vector encoding module, and then a distance measurement layer; Step S3, training the twin neural network described in step S2 based on an existing data set including street view images and remote sensing images; Step S4, constructing a data set for a specific application scenario, and continuing to perform fine-tuning training based on the twin neural network trained in step S3; Step S5, capturing a street view picture on site, and using the trained network to retrieve a matching satellite image from the data set constructed in step S4, thereby completing the positioning of the street view picture; The domain alignment algorithm in step S1 is implemented as follows: The street view image is approximately converted to the inverse polar coordinate transformation of the air view image, and the conversion is performed using the following formula, in, is the pixel coordinate of the original street view image, is the pixel coordinate of the synthetic empty view image, W q and H q are the width and height of the original empty view image, W s and H s are the width and height of the original street view image respectively; the domain alignment algorithm in step S1 is implemented as follows: The sky view image is approximately converted into a street view image using the following formula: in, is the original empty view image pixel coordinate, is the pixel coordinate of the synthetic street view image, W a and H a are the width and height of the original empty view image, W s and H s are the width and height of the original street view image respectively; the specific structure of the twin neural network in step S2 is implemented as follows: Two convolutional neural network branches for extracting feature maps, using a network composed of convolutional layers and pooling layers; The global description vector encoding module is a fully connected layer, where the number of nodes in each layer is set according to the actual application scenario; The distance metric layer calculates the Euclidean distance of the global description vector; The network training of step S3 is implemented as follows: During each training, the network is trained based on triplet sample pairs. A street view image and a satellite image taken from the same location are randomly selected from the training set as a positive sample pair. A satellite image from another location is selected to form a negative sample pair with the previously selected street view image. These three images form a triplet and are input into the network. The sample is input into the network, and passes through the feature extraction layer, the global description vector encoding layer, and the distance measurement layer in sequence. Finally, the distance between the sample pairs is obtained and brought into the weighted soft distance sorting loss; The network backpropagates according to the loss function to automatically adjust the parameters.

2. The method for real-time matching of cross-view images based on deep learning according to claim 1, characterized in that: In step S5, the geographical location of the street view image is determined using the trained network, and the implementation method is as follows: Input the street view images and satellite image datasets into the trained network; The network automatically encodes the two-view images into a one-dimensional global description vector and calculates the distance between the two vectors. Finally, it retrieves the satellite image closest to the query street view image feature from the satellite image dataset and assigns its location coordinates to the query street view image.

3. A real-time cross-view image matching geographic positioning system based on deep learning, characterized by: Used to implement a real-time matching geolocation method for cross-view images based on deep learning as described in any one of claims 1-2.

4. The cross-view image real-time matching geographic positioning system based on deep learning according to claim 3, characterized in that: Includes the following modules, The first module is used to preliminarily align the air view image with the street view image using a domain alignment algorithm; The second module is used to construct a twin neural network, the structure of which includes two convolutional neural network branches for extracting feature maps, followed by a global description vector encoding module, and then a distance measurement layer; The third module is used to train the twin neural network obtained in the second module based on the existing data set including street view images and remote sensing images; The fourth module is used to build a data set for a specific application scenario and continue to perform fine-tuning training based on the twin neural network trained in the third module; The fifth module is used to capture street view images on site and use the trained network to retrieve matching satellite images from the data set constructed in the fourth module, thereby completing the positioning of the street view images.

5. The cross-view image real-time matching geographic positioning system based on deep learning according to claim 3, characterized in that: It includes a processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a deep learning-based cross-view image real-time matching geolocation method as described in any one of claims 1-2.

6. The cross-view image real-time matching geographic positioning system based on deep learning according to claim 3, characterized in that: It includes a readable storage medium, on which a computer program is stored. When the computer program is executed, a deep learning-based cross-view image real-time matching geolocation method as described in any one of claims 1-2 is implemented.

Images