A visual positioning method and system based on dual attention and cross-viewing angles

Abstract

The present invention discloses a visual positioning method and system based on dual attention cross-viewing angle, calibrates a vehicle-mounted camera, collects ground view images through the calibrated vehicle-mounted camera, downloads multiple corresponding satellite images with coordinates from the network and converts them using polar coordinates to obtain multiple converted satellite images, builds a dual attention network model and trains it, selects one converted satellite image from multiple converted satellite images each time, inputs the ground view image and the selected converted satellite image into the trained dual attention network model for processing, outputs the one-dimensional feature vectors corresponding to the ground view image and the selected converted satellite image, calculates the cosine distance, and uses the longitude and latitude coordinates of the satellite image corresponding to the maximum cosine distance as the longitude and latitude coordinates of the ground view image to realize visual query positioning of the ground view image. The method can improve the cross-viewing matching capability of aerial satellite images and ground images, and improve the accuracy of geographic positioning.

Description

Technical Field

[0001] The present invention relates to the field of geographic positioning technology, and in particular to a visual positioning method and system based on dual attention and cross-viewing angles. Background Art

[0002] With the continuous development of the information society, devices such as smartphones have become increasingly popular, and people's daily travel can basically rely on positioning and navigation. Therefore, location information has received more and more attention from people, and the demand for location information in modern society has also increased. In recent years, image-based geolocation technology has received widespread attention in the field of computer vision due to its potential applications in autonomous driving and augmented reality. At the same time, it is difficult to rely directly on GPS for positioning and navigation in remote mountainous areas where there are sparse populations and signal base stations are difficult to cover. Without any prior knowledge, predicting the geographic location of a photo is a very difficult task because images taken under different conditions (for example, different weather, objects, or camera settings) will show huge changes. With the continuous development of remote sensing satellites, a large number of images with geographic data tags have been collected. Therefore, the problem of estimating the geographic location of the captured image is converted into a matching problem between ground-view images and air-view satellite images, that is, the geographic location of the captured image is determined by matching satellite images with geographic coordinates. Cross-view geographic visual positioning provides an auxiliary solution for positioning problems in the fields of autonomous driving, augmented reality, and in network-free environments, so it has theoretical value and practical significance. Summary of the invention

[0003] The purpose of the present invention is to improve the accuracy and stability of multimodal image registration. In order to achieve the above purpose, the present invention provides a visual positioning method based on dual attention across viewing angles, which comprises the following steps:

[0004] S1, calibrating the vehicle-mounted camera to obtain a calibrated vehicle-mounted camera;

[0005] S2, collecting data from the ground perspective through the calibrated vehicle-mounted camera to obtain a ground perspective image, downloading multiple corresponding satellite images with coordinates from the network, and converting the multiple satellite images with coordinates using polar coordinates to obtain multiple converted satellite images;

[0006] S3, carrying a dual attention network model, using a training set to train the dual attention network model and calculating the loss through a preset loss function to obtain a trained dual attention network model;

[0007] S4, randomly selecting a converted satellite image from the multiple converted satellite images, inputting the ground view image and the selected converted satellite image into the trained dual attention network model for processing, and outputting the one-dimensional feature vectors corresponding to the ground view image and the selected converted satellite image respectively;

[0008] S5, calculating the cosine distance between the one-dimensional feature vectors corresponding to the ground view image and the selected converted satellite image;

[0009] S6. Select another converted satellite image from the multiple converted satellite images until all the converted satellite images are selected, repeat steps S4 and S5, and use the longitude and latitude coordinates of the satellite image corresponding to the maximum cosine distance as the longitude and latitude coordinates of the ground perspective image to achieve visual query positioning of the ground perspective image.

[0010] Preferably, the vehicle-mounted camera is calibrated in S1, and the camera calibration can be specifically expressed by the formula:

[0011] ;

[0012] In the formula, , Respectively represent each pixel in the physical coordinate system of the image Axis and Physical dimensions on the axis, is the focal length of the onboard camera, represents the origin of the image's physical coordinate system, , They represent the scale factors in the image pixel coordinate system, represents a 3×3 rotation matrix, represents a 3D translation vector, Represents the three-dimensional point coordinates in the world coordinate system, Represents the coordinates of the corresponding points in the image pixel coordinate system projected from the three-dimensional space of the world coordinate system to the two-dimensional plane. Represents the camera coordinate system Axis coordinates.

[0013] Preferably, in S2, the plurality of satellite images with coordinates are respectively converted using polar coordinates to obtain a plurality of converted satellite images. The polar coordinate conversion can be specifically expressed by the formula:

[0014] ;

[0015] ;

[0016] In the formula, Indicates the size of the downloaded satellite image with coordinates. Indicates the size of the converted satellite image. Indicates the coordinates of the image points on the downloaded satellite image with coordinates. Represents the coordinates of the image point on the converted satellite image.

[0017] Preferably, the dual attention network model includes a twin network, a self-attention encoding module, a cross-attention encoding module and an average pooling module connected in sequence. In S4, the ground view image and the selected converted satellite image are input into the trained dual attention network model for processing, and the one-dimensional feature vectors corresponding to the ground view image and the selected converted satellite image are output, which specifically includes the following steps:

[0018] S41, the twin network receives and processes the ground view image and the selected converted satellite image, and accordingly outputs the encoded features corresponding to the ground view image and the selected converted satellite image;

[0019] S42, the self-attention encoding module receives the encoded features corresponding to the ground view image and the selected converted satellite image, respectively, and calculates the correlation of the encoded features themselves, to obtain first feature maps corresponding to the ground view image and the selected converted satellite image, respectively;

[0020] S43, the cross-attention encoding module receives the first feature maps corresponding to the ground view image and the selected converted satellite image, respectively, and calculates the correlation between the first feature maps, to obtain the second feature maps corresponding to the ground view image and the selected converted satellite image, respectively;

[0021] S44, the average pooling module receives the second feature maps corresponding to the ground view image and the selected converted satellite image, respectively, and performs average pooling on them, and outputs the one-dimensional feature vectors corresponding to the ground view image and the selected converted satellite image, respectively.

[0022] Preferably, the twin network is specifically a modified VGG16 network, and the encoded features corresponding to the ground view image and the selected converted satellite image in S41 can be specifically expressed as:

[0023] ;

[0024] ;

[0025] In the formula, Represents the encoded features corresponding to the ground view image, represents the encoded features corresponding to the selected converted satellite image, represents the modified VGG16 network, represents the ground view image, Represents the selected converted satellite image.

[0026] Preferably, the self-attention encoding module includes a first self-attention module, the first self-attention module is connected to the twin network, the first self-attention module includes a first convolutional layer, a second convolutional layer and a third convolutional layer connected in parallel, and the self-attention encoding module in S42 receives the encoded features corresponding to the ground perspective image and calculates the correlation of the encoded features themselves to obtain a first feature map of the ground perspective image. The specific process is as follows:

[0027] S421, inputting the encoded features corresponding to the ground view image into the first self-attention module;

[0028] S422, the first convolution layer, the second convolution layer, and the third convolution layer of the first self-attention module respectively process the encoded features corresponding to the ground view image, and correspondingly obtain a first feature map, a second feature map, and a third feature map of the ground view image;

[0029] S423, calculating the self-attention weight of the ground perspective image according to the first feature map and the second feature map of the ground perspective image;

[0030] S424. Calculate a first feature map of the ground perspective image according to the self-attention weight of the ground perspective image and the third feature map of the ground perspective image.

[0031] Preferably, the cross-attention encoding module includes a first cross-attention module and a second cross-attention module, the first cross-attention module is connected to the first self-attention module, and the second cross-attention module is connected to the second self-attention module. The specific process of S43 is as follows:

[0032] S431, inputting the first feature map of the ground view image into the first cross attention module, and inputting the first feature map of the selected converted satellite image into the second cross attention module;

[0033] S432, the first cross-attention module and the second cross-attention module respectively perform convolution processing on the first feature maps received by each, and obtain a fourth feature map, a fifth feature map and a sixth feature map corresponding to the ground view image and the selected converted satellite image respectively;

[0034] S433, calculating the cross attention weight of the ground view image according to the fourth feature map of the selected converted satellite image and the fifth feature map of the ground view image, and calculating the cross attention weight of the selected converted satellite image according to the fourth feature map of the ground view image and the fifth feature map of the selected converted satellite image;

[0035] S434. Calculate the second feature maps corresponding to the ground perspective image and the selected converted satellite image respectively according to the cross attention weights and the sixth feature maps corresponding to the ground perspective image and the selected converted satellite image respectively.

[0036] Preferably, the one-dimensional feature vectors corresponding to the ground view image and the selected converted satellite image are output in S44, which can be specifically expressed by the formula:

[0037] ;

[0038] ;

[0039] In the formula, represents average pooling, is the one-dimensional feature vector of the ground view image, is the one-dimensional feature vector of the selected converted satellite image, A second feature map representing the ground view image, A second feature map representing the selected transformed satellite image.

[0040] Preferably, in S5, the cosine distance between the one-dimensional feature vectors corresponding to the ground view image and the selected converted satellite image is calculated, and the specific formula is as follows:

[0041] ;

[0042] In the formula, represents the cosine distance, One-dimensional feature vector representing the ground view image The model, Represents the one-dimensional feature vector of the selected transformed satellite image Model.

[0043] Another aspect of the present invention provides a visual positioning system, which uses the above-mentioned dual-attention cross-view visual positioning method for positioning, and the system includes a vehicle, a vehicle-mounted camera fixed on the vehicle, and a computer system, the computer system is connected to the vehicle-mounted camera, and the computer system is provided with a processing unit, a storage unit and a dual attention model connected in sequence, wherein:

[0044] The image acquisition unit is used to acquire a ground view image and send the ground view image to a computer system;

[0045] Downloading a plurality of corresponding satellite images with coordinates from the network and sending them to a computer system, a processing unit in the computer system performs polar coordinate conversion on the plurality of satellite images with coordinates to obtain a plurality of converted satellite images;

[0046] The storage unit is used to store ground view images, multiple converted satellite images, and parameters of the dual attention network model;

[0047] The dual attention network model receives the ground view image and multiple converted satellite images and uses a dual attention cross-view visual positioning method to complete the similarity matching between the ground view image and the corresponding satellite image, and outputs the latitude and longitude coordinates corresponding to the ground view image.

[0048] The vehicle's geographic positioning is achieved based on the latitude and longitude coordinates corresponding to the ground perspective image.

[0049] The above-mentioned visual positioning method and system based on dual attention across perspectives first calibrates the vehicle-mounted camera, then collects images of the ground perspective through the calibrated vehicle-mounted camera, downloads multiple corresponding satellite images with coordinates from the network and converts them using polar coordinates to obtain multiple converted satellite images, then builds a dual attention network model and trains it, selects one converted satellite image from the multiple converted satellite images each time, inputs the ground perspective image and the selected converted satellite image into the trained dual attention network model for processing, outputs the one-dimensional feature vectors corresponding to the ground perspective image and the selected converted satellite image, and calculates the cosine distance, and uses the longitude and latitude coordinates of the satellite image corresponding to the maximum cosine distance as the longitude and latitude coordinates of the ground perspective image to realize the visual query positioning of the ground perspective image. The present invention proposes a visual positioning method under cross-perspective, which can improve the ability of aerial satellite images and ground images to match across perspectives, and improve the accuracy of geographic positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a flow chart of a visual localization method based on dual attention across viewing angles in one embodiment of the present invention;

[0051] Figure 2 Schematic diagram of the network structure of the dual attention network model in one embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings.

[0053] See also Figure 1 and Figure 2 , Figure 1 is a flow chart of a visual localization method based on dual attention across viewing angles in one embodiment of the present invention. Figure 2 Schematic diagram of the network structure of the dual attention network model in one embodiment of the present invention.

[0054] A visual localization method based on dual attention and cross-viewing angles, the method comprising the following steps:

[0055] S1. Calibrate the vehicle-mounted camera to obtain a calibrated vehicle-mounted camera.

[0056] Specifically, by calibrating the vehicle-mounted camera, the geometric relationship and distortion characteristics of the vehicle-mounted camera imaging can be determined, thereby improving the accuracy and stability of the vehicle-mounted camera imaging.

[0057] Furthermore, the specific method of camera calibration is as follows:

[0058] (1)

[0059] (2)

[0060] In the formula, , Respectively represent each pixel in the physical coordinate system of the image Axis and The physical size of the axis in millimeters / pixels, is the focal length of the onboard camera, represents the origin of the image's physical coordinate system, , They represent the scale factors in the image pixel coordinate system, in millimeters, represents a 3×3 rotation matrix, represents a 3D translation vector, represents the coordinates of the three-dimensional point in the world coordinate system w, Represents the coordinates of the corresponding points in the image pixel coordinate system projected from the three-dimensional space of the world coordinate system to the two-dimensional plane. Represents the camera coordinate system Axis coordinates.

[0061] Through the above formula, a point in three-dimensional space can be converted from the world coordinate system to the camera coordinate system, then from the camera coordinate system to the image physical coordinate system, and finally from the image physical coordinate system to the image pixel coordinate system.

[0062] S2. Collect data from the ground perspective through the calibrated vehicle-mounted camera to obtain a ground perspective image, download a plurality of corresponding satellite images with coordinates from the network, and convert the plurality of satellite images with coordinates using polar coordinates to obtain a plurality of converted satellite images.

[0063] Specifically, a plurality of corresponding satellite images with coordinates downloaded from the Internet (such as Google) are converted using polar coordinates to obtain a plurality of converted satellite images of ground perspective, so as to reduce the difference between the ground perspective images and the corresponding satellite images with coordinates.

[0064] Furthermore, in S2, multiple satellite images with coordinates are converted using polar coordinates to obtain multiple converted satellite images. The polar coordinate conversion can be specifically expressed by the formula:

[0065] (3)

[0066] (4)

[0067] In the formula, Indicates the size of the downloaded satellite image with coordinates. Indicates the size of the converted satellite image. Indicates the coordinates of the image points on the downloaded satellite image with coordinates. Represents the coordinates of the image point on the converted satellite image.

[0068] S3. Equipped with a dual attention network model, the dual attention network model is trained using a training set and the loss is calculated using a preset loss function to obtain a trained dual attention network model.

[0069] S4. Randomly select a converted satellite image from the multiple converted satellite images, input the ground view image and the selected converted satellite image into the trained dual attention network model for processing, and output the one-dimensional feature vectors corresponding to the ground view image and the selected converted satellite image respectively.

[0070] Furthermore, the dual attention network model includes a twin network, a self-attention encoding module, a cross-attention encoding module and an average pooling module connected in sequence. In S4, the ground view image and the selected converted satellite image are input into the trained dual attention network model for processing, and the one-dimensional feature vectors corresponding to the ground view image and the selected converted satellite image are output, which specifically includes the following steps:

[0071] S41. The twin network receives and processes the ground perspective image and the selected converted satellite image, and accordingly outputs the encoded features corresponding to the ground perspective image and the selected converted satellite image.

[0072] Furthermore, the twin network is specifically a modified VGG16 network, which includes three modules connected in sequence, wherein the first module consists of two convolutional layers and one maximum pooling layer, whose input is a 288×288 3-channel RGB image, and whose output is a 144×144 64-channel feature map; the second module consists of two convolutional layers and one maximum pooling layer, whose input is a 144×144 64-channel feature map, and whose output is a 72×72 128-channel feature map; the third module consists of three convolutional layers and one maximum pooling layer, whose input is a 72×72 128-channel feature map, and whose output is a 36×36 256-channel feature map.

[0073] The ground view image and the selected converted satellite image are input into the modified VGG16 network for feature encoding, and the encoded features corresponding to the ground view image and the selected converted satellite image are output respectively. and , which can be specifically expressed as:

[0074] (5)

[0075] (6)

[0076] In the formula, Represents the encoded features corresponding to the ground view image, represents the encoded features corresponding to the selected converted satellite image, represents the modified VGG16 network, represents the ground view image, Represents the selected converted satellite image.

[0077] S42. The self-attention encoding module receives the encoded features corresponding to the ground view image and the selected converted satellite image, respectively, and calculates the correlation of the encoded features themselves, to obtain the first feature mapping images corresponding to the ground view image and the selected converted satellite image, respectively.

[0078] Specifically, the self-attention encoding module is used to calculate the correlation of the encoded features themselves. The self-attention encoding module includes a first self-attention module and a second self-attention module. The first self-attention module and the second self-attention module are both connected to the twin network. The first self-attention module includes three parallel connections, the first convolution layer, the second convolution layer and the third convolution layer of size 1 × 1, and the second self-attention module includes three parallel connections, the seventh convolution layer, the eighth convolution layer and the ninth convolution layer of size 1 × 1. Now take the encoded features corresponding to the ground view image In S42, the self-attention encoding module receives the encoded features corresponding to the ground view image and calculates the correlation of the encoded features themselves to obtain the first feature map of the ground view image. The specific process is as follows:

[0079] S421, the encoded features corresponding to the ground view image Enter the first self-attention module;

[0080] S422, the first convolutional layer, the second convolutional layer and the third convolutional layer of the first self-attention module respectively perform the encoded features After processing, three first feature maps of size C × H × W are obtained accordingly , the second feature map and the third feature map , where C represents the number of channels of the feature map, H represents the height of the feature map, and W represents the width of the feature map;

[0081] S423: Mapping the first feature of the ground view image and the second feature map Calculate the self-attention weights for the ground view image , the specific formula is:

[0082] (7)

[0083] In the formula, represents the weight of self-attention of the ground view image, represents the softmax activation function, Indicates the number of channels of the feature map;

[0084] S424, self-attention weight according to ground view image and the third feature map of the ground view image Calculate the first feature map of the ground view image , the specific formula is:

[0085] (8)

[0086] In the formula, A first feature map representing a ground view image.

[0087] Similarly, the encoded features corresponding to the selected converted satellite images are Input the second self-attention encoding module and calculate in the same way as above to obtain the first feature map of the selected converted satellite image. , I will not go into details here.

[0088] S43. The cross-attention encoding module receives the first feature maps corresponding to the ground perspective image and the selected converted satellite image, respectively, and calculates the correlation between the first feature maps to obtain the second feature maps corresponding to the ground perspective image and the selected converted satellite image, respectively.

[0089] Specifically, the cross-attention encoding module is used to calculate the correlation between the first feature maps corresponding to the ground view image and the selected converted satellite image, respectively, to achieve the interaction of feature information. The cross-attention encoding module includes a first cross-attention module and a second cross-attention module, wherein the first cross-attention module is connected to the first self-attention module, the second cross-attention module is connected to the second self-attention module, the first cross-attention module includes three parallel connected fourth convolutional layers, fifth convolutional layers and sixth convolutional layers, the second cross-attention module includes three parallel connected tenth convolutional layers, eleventh convolutional layers and twelfth convolutional layers, and the sizes of the fourth to sixth convolutional layers and the tenth to twelfth convolutional layers are the same as the size of the first convolutional layer.

[0090] The specific process of S43 is as follows:

[0091] S431, inputting the first feature map of the ground view image into the first cross attention module, and inputting the first feature map of the selected converted satellite image into the second cross attention module;

[0092] S432, the first cross-attention module and the second cross-attention module respectively perform convolution processing on the first feature maps they receive, and accordingly obtain the fourth feature map, the fifth feature map and the sixth feature map corresponding to the ground perspective image and the selected converted satellite image.

[0093] Specifically, the fourth convolutional layer, the fifth convolutional layer, and the sixth convolutional layer in the first cross attention module respectively perform the first feature map of the ground view image. Convolution processing is performed to obtain the fourth feature map of three ground view images of size C × H × W. , the fifth feature map and the sixth feature map ;

[0094] The tenth convolutional layer, the eleventh convolutional layer, and the twelfth convolutional layer in the second cross-attention module respectively select the first feature map of the converted satellite image Convolution processing is performed to obtain the fourth feature maps of three selected converted satellite images of size C × H × W. , the fifth feature map and the sixth feature map .

[0095] S433, according to the fourth feature map of the selected converted satellite image Fifth feature map of ground-level view images Calculate the cross attention weights for the ground view image , according to the fourth feature map of the ground view image And the fifth feature map of the selected transformed satellite image Calculate the cross attention weights of the selected transformed satellite images , the specific formula is:

[0096] (9) -1

[0097] In the formula, represents the cross attention weight of the ground view image, is the fourth feature map of the selected converted satellite image, is the fifth feature map of the ground view image;

[0098] (9) -2

[0099] In the formula, represents the cross attention weight of the selected transformed satellite image, is the fourth feature map of the ground view image, is the fifth feature map of the selected converted satellite image.

[0100] S434. Calculate the second feature maps corresponding to the ground perspective image and the selected converted satellite image respectively according to the cross attention weights and the sixth feature maps corresponding to the ground perspective image and the selected converted satellite image respectively.

[0101] Specifically, according to the cross attention weight of the ground view image And the sixth feature map of the ground view image Calculate the second feature map of the ground view image , according to the cross attention weight of the selected converted satellite image And the sixth feature map of the selected transformed satellite image Calculate the second feature map of the selected transformed satellite image , the specific formula is as follows:

[0102] (10) -1

[0103] In the formula, A second feature map representing the ground view image, a sixth feature map representing a ground view image;

[0104] (10) -2

[0105] In the formula, represents the second feature map of the selected transformed satellite image, Represents the sixth feature map of the selected transformed satellite image.

[0106] S44, the average pooling module receives the second feature maps corresponding to the ground view image and the selected converted satellite image, respectively, and performs average pooling on them, and outputs the one-dimensional feature vectors corresponding to the ground view image and the selected converted satellite image, respectively.

[0107] Furthermore, the one-dimensional feature vectors corresponding to the ground view image and the selected converted satellite image can be specifically expressed as:

[0108] (11)

[0109] (12)

[0110] In the formula, represents average pooling, is the one-dimensional feature vector of the ground view image, is the one-dimensional feature vector of the selected converted satellite image, A second feature map representing the ground view image, A second feature map representing the selected transformed satellite image.

[0111] S5. Calculate the cosine distance between the one-dimensional feature vectors corresponding to the ground view image and the selected converted satellite image. The specific formula for calculating the cosine distance is as follows:

[0112] (13)

[0113] In the formula, represents the cosine distance, One-dimensional feature vector representing the ground view image The model, Represents the one-dimensional feature vector of the selected transformed satellite image Model.

[0114] S6. Select another selected converted satellite image from the multiple selected converted satellite images, until all the multiple selected converted satellite images are selected, repeat steps S4 and S5, and use the longitude and latitude coordinates of the satellite image corresponding to the maximum cosine distance as the longitude and latitude coordinates of the ground perspective image to achieve visual query positioning of the ground perspective image.

[0115] Furthermore, the most similar feature vector pair is found through the cosine distance. The smaller the cosine distance, the smaller the similarity of the feature vector pair. The larger the cosine distance, the more similar the pair of feature maps are. The longitude and latitude coordinates of the satellite map corresponding to the maximum cosine similarity are used as the longitude and latitude coordinates of the ground view image taken by the current camera.

[0116] Furthermore, when training the dual attention network model, the training set is first obtained. The specific process is as follows:

[0117] The ground view image is obtained according to the calibrated vehicle-mounted camera, denoted as , download ground-level images from the Internet The matched satellite image with coordinates is converted using polar coordinates to obtain the image with ground perspective Matching satellite imagery , the ground view image and matching satellite imagery Form a pair of matching ground-to-air image samples (i.e., ground-to-air image and satellite image); download the ground-to-air image from the Internet Unmatched satellite images with coordinates are converted using polar coordinates to obtain multiple images with ground perspective Mismatched satellite imagery , the ground view image and multiple satellite images that do not match Multiple pairs of unmatched ground-to-air image samples (i.e., ground-to-air images and satellite images) are formed. A pair of matched ground-to-air image sample pairs and multiple pairs of unmatched ground-to-air image sample pairs are used as training sets.

[0118] The dual attention network model is trained using the training set and the loss is calculated using the preset loss function to obtain the trained dual attention network model. The specific process is as follows:

[0119] A pair of matching ground-to-air perspective image sample pairs and multiple pairs of unmatched ground-to-air perspective image sample pairs are respectively input into the dual attention network model for training, and the ground perspective images in the training set are obtained accordingly. The one-dimensional feature vector of , satellite images in the training set that match the ground view images The one-dimensional feature vector of , the one-dimensional feature vectors corresponding to multiple satellite images in the training set that do not match the ground view image , according to the one-dimensional feature vector and Calculate the Euclidean distance of matching ground-to-air image sample pairs ; According to the one-dimensional feature vector and Calculate the Euclidean distance between unmatched ground-to-air image sample pairs , where the Euclidean distance Number and multiple satellite images The number is consistent.

[0120] During the training process, the following loss function is used to calculate the loss of the dual attention network model until the calculated loss value of the dual attention network model is minimized, and the trained dual attention network model is obtained:

[0121] (14)

[0122] In the formula, represents the loss of the dual attention network model, Adjust the loss gradient parameters to adjust the loss gradient. It is usually set to 10, which can speed up the convergence of the algorithm. is the Euclidean distance of the matching ground-to-air image sample pairs, is the Euclidean distance of unmatched ground-to-space image sample pairs.

[0123] The loss is calculated based on the above loss function, so that the one-dimensional feature vectors of the ground-to-air perspective images that are originally a pair (the ground perspective image and the satellite image that matches the ground perspective image) are made closer and closer, and the one-dimensional feature vectors of the ground-to-air perspective images that are not a pair (the ground perspective image and the satellite image that does not match the ground perspective image) are made farther and farther away, thereby converging the loss value of the dual attention network model.

[0124] In another embodiment, a visual positioning system adopts the above-mentioned visual positioning method based on dual attention across viewing angles for positioning, the system includes a vehicle, a vehicle-mounted camera fixed on the vehicle, and a computer system, the computer system is connected to the vehicle-mounted camera, and the computer system is provided with a processing unit, a storage unit and a dual attention model connected in sequence, wherein:

[0125] The image acquisition unit is used to acquire a ground view image and send the ground view image to a computer system;

[0126] Downloading a plurality of corresponding satellite images with coordinates from a network (e.g., Google) and sending them to a computer system, wherein a processing unit in the computer system performs polar coordinate conversion on the plurality of satellite images with coordinates to obtain a plurality of converted satellite images;

[0127] The storage unit is used to store ground view images, multiple converted satellite images, and parameters of the dual attention network model;

[0128] The dual attention network model receives the ground view image and multiple converted satellite images and uses the above-mentioned dual attention cross-view visual positioning method to complete the similarity matching between the ground view image and the corresponding satellite image, and outputs the latitude and longitude coordinates corresponding to the ground view image;

[0129] The vehicle's geographic positioning is achieved based on the latitude and longitude coordinates corresponding to the ground perspective image.

[0130] For the specific definition of a visual positioning system, please refer to the definition of the visual positioning method based on dual attention across perspectives in the above text, which will not be repeated here.

[0131] The above-mentioned visual positioning method and system based on dual attention across perspectives first calibrates the vehicle-mounted camera, then collects images of the ground perspective through the calibrated vehicle-mounted camera, downloads multiple corresponding satellite images with coordinates from the network and converts them using polar coordinates to obtain multiple converted satellite images, then builds a dual attention network model and trains it, selects one converted satellite image from the multiple converted satellite images each time, inputs the ground perspective image and the selected converted satellite image into the trained dual attention network model for processing, outputs the one-dimensional feature vectors corresponding to the ground perspective image and the selected converted satellite image, and calculates the cosine distance, and uses the longitude and latitude coordinates of the satellite image corresponding to the maximum cosine distance as the longitude and latitude coordinates of the ground perspective image to realize the visual query positioning of the ground perspective image. The present invention proposes a visual positioning method under cross-perspective, which can improve the ability of aerial satellite images and ground images to match across perspectives, and improve the accuracy of geographic positioning.

[0132] The above is a detailed introduction to a visual positioning method and system based on dual attention across viewing angles provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the core idea of ​​the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A visual localization method based on dual attention and cross-viewing angles, characterized in that: The method comprises the following steps: S1, calibrating the vehicle-mounted camera to obtain a calibrated vehicle-mounted camera; S2, collecting data from the ground perspective through the calibrated vehicle-mounted camera to obtain a ground perspective image, downloading multiple corresponding satellite images with coordinates from the network, and converting the multiple satellite images with coordinates using polar coordinates to obtain multiple converted satellite images; S3, carrying a dual attention network model, using a training set to train the dual attention network model and calculating the loss through a preset loss function to obtain a trained dual attention network model; S4, randomly selecting a converted satellite image from the multiple converted satellite images, inputting the ground view image and the selected converted satellite image into the trained dual attention network model for processing, and outputting the one-dimensional feature vectors corresponding to the ground view image and the selected converted satellite image respectively; S5, calculating the cosine distance between the one-dimensional feature vectors corresponding to the ground view image and the selected converted satellite image; S6, selecting another converted satellite image from the multiple converted satellite images, until the multiple converted satellite images are all selected, repeating steps S4 and S5, and using the longitude and latitude coordinates of the satellite image corresponding to the maximum cosine distance as the longitude and latitude coordinates of the ground view image, so as to achieve visual query positioning of the ground view image; The dual attention network model includes a twin network, a self-attention encoding module, a cross-attention encoding module and an average pooling module connected in sequence. In S4, the ground view image and the selected converted satellite image are input into the trained dual attention network model for processing, and the one-dimensional feature vectors corresponding to the ground view image and the selected converted satellite image are output, which specifically includes the following steps: S41, the twin network receives and processes the ground view image and the selected converted satellite image, and accordingly outputs the encoded features corresponding to the ground view image and the selected converted satellite image; S42, the self-attention encoding module receives the encoded features corresponding to the ground view image and the selected converted satellite image, respectively, and calculates the correlation of the encoded features themselves, to obtain first feature maps corresponding to the ground view image and the selected converted satellite image, respectively; S43, the cross-attention encoding module receives the first feature maps corresponding to the ground view image and the selected converted satellite image, respectively, and calculates the correlation between the first feature maps, to obtain the second feature maps corresponding to the ground view image and the selected converted satellite image, respectively; S44, the average pooling module receives the second feature maps corresponding to the ground view image and the selected converted satellite image, respectively, and performs average pooling on them, and outputs the one-dimensional feature vectors corresponding to the ground view image and the selected converted satellite image, respectively; The twin network is specifically a modified VGG16 network; the self-attention encoding module includes a first self-attention module and a second self-attention module, and the first self-attention module and the second self-attention module are both connected to the twin network; the cross-attention encoding module includes a first cross-attention module and a second cross-attention module, and the first cross-attention module is connected to the first self-attention module, and the second cross-attention module is connected to the second self-attention module.

2. The visual localization method based on dual attention and cross-viewing angles as claimed in claim 1, characterized in that: In S1, the vehicle-mounted camera is calibrated. The camera calibration can be specifically expressed by the formula: ； In the formula, , Respectively represent each pixel in the physical coordinate system of the image Axis and Physical dimensions on the axis, is the focal length of the onboard camera, represents the origin of the image's physical coordinate system, , They represent the scale factors in the image pixel coordinate system, represents a 3×3 rotation matrix, represents a 3D translation vector, Represents the three-dimensional point coordinates in the world coordinate system, Represents the coordinates of the corresponding points in the image pixel coordinate system projected from the three-dimensional space of the world coordinate system to the two-dimensional plane. Represents the camera coordinate system Axis coordinates.

3. The visual localization method based on dual attention and cross-viewing angles as claimed in claim 2, characterized in that: In S2, multiple satellite images with coordinates are converted using polar coordinates to obtain multiple converted satellite images. The polar coordinate conversion can be specifically expressed by the formula: ； ； In the formula, Indicates the size of the downloaded satellite image with coordinates. Indicates the size of the converted satellite image. Indicates the coordinates of the image points on the downloaded satellite image with coordinates. Represents the coordinates of the image point on the converted satellite image.

4. The visual localization method based on dual attention and cross-viewing angles as claimed in claim 3, characterized in that: The encoded features corresponding to the ground view image and the selected converted satellite image in S41 can be specifically expressed as: ； ； In the formula, Represents the encoded features corresponding to the ground view image, represents the encoded features corresponding to the selected converted satellite image, represents the modified VGG16 network, represents the ground view image, Represents the selected converted satellite image.

5. The visual localization method based on dual attention and cross-viewing angles as claimed in claim 4, characterized in that: The first self-attention module includes a first convolutional layer, a second convolutional layer, and a third convolutional layer connected in parallel. In S42, the self-attention encoding module receives the encoded features corresponding to the ground view image and calculates the correlation of the encoded features themselves to obtain a first feature map of the ground view image. The specific process is as follows: S421, inputting the encoded features corresponding to the ground view image into the first self-attention module; S422, the first convolution layer, the second convolution layer, and the third convolution layer of the first self-attention module respectively process the encoded features corresponding to the ground view image, and correspondingly obtain a first feature map, a second feature map, and a third feature map of the ground view image; S423, calculating the self-attention weight of the ground perspective image according to the first feature map and the second feature map of the ground perspective image; S424. Calculate a first feature map of the ground perspective image according to the self-attention weight of the ground perspective image and the third feature map of the ground perspective image.

6. The visual localization method based on dual attention and cross-viewing angles as claimed in claim 5, characterized in that: The specific process of S43 is as follows: S431, inputting the first feature map of the ground view image into the first cross attention module, and inputting the first feature map of the selected converted satellite image into the second cross attention module; S432, the first cross-attention module and the second cross-attention module respectively perform convolution processing on the first feature maps received by each, and obtain a fourth feature map, a fifth feature map and a sixth feature map corresponding to the ground view image and the selected converted satellite image respectively; S433, calculating the cross attention weight of the ground view image according to the fourth feature map of the selected converted satellite image and the fifth feature map of the ground view image, and calculating the cross attention weight of the selected converted satellite image according to the fourth feature map of the ground view image and the fifth feature map of the selected converted satellite image; S434. Calculate the second feature maps corresponding to the ground perspective image and the selected converted satellite image respectively according to the cross attention weights and the sixth feature maps corresponding to the ground perspective image and the selected converted satellite image respectively.

7. The visual localization method based on dual attention and cross-viewing angles as claimed in claim 6, characterized in that: The one-dimensional feature vectors corresponding to the ground view image and the selected converted satellite image are output in S44, which can be specifically expressed by the formula: ； ； In the formula, represents average pooling, is the one-dimensional feature vector of the ground view image, is the one-dimensional feature vector of the selected converted satellite image, A second feature map representing the ground view image, A second feature map representing the selected transformed satellite image.

8. The visual localization method based on dual attention and cross-viewing angles as claimed in claim 7, characterized in that: In S5, the cosine distance between the one-dimensional feature vectors corresponding to the ground view image and the selected converted satellite image is calculated. The specific formula is as follows: ； In the formula, represents the cosine distance, One-dimensional feature vector representing the ground view image The model, Represents the one-dimensional feature vector of the selected transformed satellite image Model.

9. A visual positioning system, using the visual positioning method based on dual attention and cross-viewing angles as described in any one of claims 1 to 8 for positioning, characterized in that: The system includes a vehicle, a vehicle-mounted camera fixed on the vehicle, and a computer system, wherein the computer system is connected to the vehicle-mounted camera, and the computer system is provided with a processing unit, a storage unit, and a dual attention model connected in sequence, wherein: The image acquisition unit is used to acquire a ground view image and send the ground view image to a computer system; Downloading a plurality of corresponding satellite images with coordinates from the network and sending them to a computer system, a processing unit in the computer system performs polar coordinate conversion on the plurality of satellite images with coordinates to obtain a plurality of converted satellite images; The storage unit is used to store ground view images, multiple converted satellite images, and parameters of the dual attention network model; The dual attention network model receives the ground view image and multiple converted satellite images and uses a dual attention cross-view visual positioning method to complete the similarity matching between the ground view image and the corresponding satellite image, and outputs the latitude and longitude coordinates corresponding to the ground view image. The vehicle's geographic positioning is achieved based on the latitude and longitude coordinates corresponding to the ground perspective image.

Images