Cross-view-angle image template matching and positioning method for large scene

Abstract

The invention provides a cross-view-angle image template matching and positioning method for a large scene. The method comprises the following steps: step 1, converting a ground panoramic image into a bird's-eye view angle image through geometric mapping; 2, inputting a bird's-eye view angle image and a large-scene satellite image into an image template matching and positioning model to obtain a semantic probability prediction map; wherein the feature extraction backbone network of the image template matching and positioning model adopts a semantic segmentation network. According to the method, semantic features instead of local features are selected and matched between two images, so that an image template matching and positioning problem is converted into a semantic probability prediction task from a traditional feature matching task, and an output result of a whole image template matching and positioning model is a probability prediction map; the positioning accuracy and efficiency of cross-view image template matching positioning in a large scene can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of image geographic positioning technology, and in particular to a large-scene cross-viewing angle image template matching positioning method. Background Art

[0002] At present, most of the research on cross-view image geolocation tasks chooses to use image retrieval. Its positioning accuracy depends on the offset between the center coordinates of the satellite image and the coordinates of the ground panoramic image. Therefore, in order to ensure the positioning accuracy, it is necessary to crop the large satellite image densely enough, which greatly increases the complexity of this work. Even though some relevant scholars have studied the use of center coordinate offset regression prediction to achieve sub-meter positioning, this method places quite high requirements on experimental data and network training, so cross-view image matching positioning based on retrieval is more suitable for relatively small-scale scene-level coarse positioning. In addition, in order to cope with drastic perspective differences, algorithms based on image retrieval usually use global descriptors for matching, which determines that the range covered by the reference satellite image cannot be too large. According to the currently public cross-view image geolocation data set, the range of satellite images covering the real world is about 150m×150m. Therefore, when fast and accurate positioning is required in a large-scale scene, it is obviously necessary to consider new positioning standards. Template matching-based methods can meet this demand, but there is a lack of relevant research because there are several key problems:

[0003] (1) Cross-domain image template matching is too difficult. Satellite images and ground panoramic images belong to different image domains. Without domain alignment, the drastic difference in viewing angles means that the matching network can only extract a few similarity features, which cannot meet the requirements for successful matching.

[0004] (2) The image scale difference is too large. Although the template matching algorithm generally has a certain ability to resist scale changes, if the matching is based solely on ground panoramic images, there is a lack of relevant technology to bridge this huge scale difference.

[0005] (3) Lack of suitable datasets. Deep learning is a data-driven technology, but there is currently a lack of large datasets for cross-view image geolocation template matching.

[0006] Rao et al. proposed using the idea of ​​template matching to solve the problem of cross-view matching and positioning of large-scene satellite images. This method requires the use of handheld photography equipment to collect multi-view images of the ground, and the use of three-dimensional point cloud technology to realize the conversion from the ground side view to the aerial view. Finally, template matching is completed on the large-scale satellite image to achieve precise positioning. However, its complex processing solution and multi-view shooting are obviously not suitable for all scenarios. Summary of the invention

[0007] In order to enable rapid and accurate positioning based on cross-view images in large-scale scenes, the present invention provides a cross-view image template matching positioning method for large scenes.

[0008] The cross-viewing image template matching and positioning method for large scenes provided by the present invention includes:

[0009] Step 1: Convert the ground panoramic image that conforms to the geometric mapping principle into a bird's-eye view image through geometric mapping;

[0010] Step 2: Input the bird's-eye view image and the large-scene satellite image into the image template matching and positioning model to obtain a semantic probability prediction map; wherein the feature extraction backbone network of the image template matching and positioning model adopts a semantic segmentation network.

[0011] Furthermore, the training process of the image template matching positioning model includes:

[0012] Constructing a training data set; each training sample in the training data set is an image pair consisting of a bird's-eye view image and a large-scene satellite image carrying a matching label; wherein the matching label is used to indicate a matching position of a foreground feature of the bird's-eye view image in the large-scene satellite image;

[0013] The semantic segmentation network is trained using the training data set to obtain the image template matching positioning model.

[0014] Furthermore, the position label adopts a Gaussian label; the generation process of the Gaussian label includes:

[0015] Collecting ground panoramic images with known shooting coordinates, and converting the ground panoramic images into bird's-eye view images through geometric mapping;

[0016] Determine the pixel coordinate center of the bird's-eye view image on the large-scene satellite image by coordinate conversion according to the shooting coordinates;

[0017] On the large-scene satellite image, a label image is produced with the pixel coordinate center as the center of a circle and a set number of pixels as a radius.

[0018] Furthermore, in the process of training the image template matching and positioning model, a combined loss function consisting of a binary cross entropy loss, a focal loss, and a center of gravity invariant loss is adopted.

[0019] Furthermore, the geometric mapping conversion relationship between the ground panoramic image and the bird's-eye view image is:

[0020]

[0021] Where, f = 0.5W b / tan(fov) represents the focal length of the bird's-eye view image, fov represents the given mapping perspective parameter, H p and W p Respectively represent the height and width of the ground panoramic image, H b and W b Respectively represent the height and width of the bird's-eye view image, (u b ,v b ) represents the pixel coordinates on the imaging plane of the bird's-eye view image, (u p ,v p ) represents the ground panoramic image coordinates.

[0022] Furthermore, the feature extraction backbone network adopts the DeepLabv3+ network.

[0023] Beneficial effects of the present invention:

[0024] (1) Based on the geometric mapping conversion of the ground panoramic image to realize the conversion of the ground panoramic image into the bird's-eye view image, the present invention adopts the idea of ​​semantic information matching and introduces a semantic probability prediction template matching framework, which maps the feature matching task into the probability prediction of the semantic space position, thereby improving the accuracy and efficiency of image template matching positioning.

[0025] (2) The model was trained and tested by improving the loss function and building a self-built data set. Experimental results show that the method of the present invention can match and locate satellite images covering a real ground range of 1200m×1200m. Compared with the traditional retrieval method of geographic positioning, it is a precise positioning on large scene images. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 One of the flow diagrams of the cross-view image template matching and positioning method for large scenes provided by an embodiment of the present invention;

[0027] Figure 2 A second flow chart of a large-scene cross-view image template matching and positioning method provided by an embodiment of the present invention;

[0028] Figure 3 A diagram of the DeepLabv3+ encoding network structure provided by an embodiment of the present invention;

[0029] Figure 4 A schematic diagram of an ASPP module provided in an embodiment of the present invention;

[0030] Figure 5 A geometric principle diagram of ground panoramic image imaging provided by an embodiment of the present invention;

[0031] Figure 6A schematic diagram of the spherical mapping principle of a ground panoramic image provided by an embodiment of the present invention;

[0032] Figure 7 Geometric mapping effect diagram of ground panoramic images provided by the embodiments of the present invention: (a) ground panoramic image, (b) bird's-eye view image, (c) satellite image;

[0033] Figure 8 Sample display of the training data set provided for the embodiment of the present invention: (a) preprocessed ground panoramic image, (b) large scene satellite image, and (c) Gaussian label. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] Traditional template matching is generally a feature-based method. First, the point features, line features (edges and contours) and surface features of the image are extracted, and then the similarity measurement between the features is used for matching. The representative method is the SIFT operator, but this operator cannot cope with too strong perspective differences, and due to the influence of grayscale differences between images, it cannot extract common features that can reflect the matching between cross-domain images. Therefore, the inventor believes that the method of selecting template matching in this task should abandon the similarity measurement of local features, and should choose semantic features that can effectively handle nonlinear radiation differences and shape distortion.

[0036] Based on this idea, Figure 1 As shown, an embodiment of the present invention provides a cross-viewing image template matching and positioning method for a large scene, comprising the following steps:

[0037] S101: converting the ground panoramic image that complies with the geometric mapping principle into a bird's-eye view image through geometric mapping;

[0038] Specifically, using geometric mapping to process ground panoramic images can effectively align the feature domains of ground panoramic images and large-scene satellite images, making the corresponding feature maps closer in the two feature spaces. The effect of geometric mapping is as follows: Figure 7As shown in the figure, from the conversion effect, the facilities such as roads located on the ground plane are well restored, and have a large appearance similarity with the real large-scale satellite images. This makes it easier for the feature extraction network to extract more discriminative features when performing cross-view image matching and positioning tasks, and makes the image matching process more interpretable.

[0039] S102: Inputting the bird's-eye view image and the large-scene satellite image into the image template matching and positioning model to obtain a semantic probability prediction map; wherein the feature extraction backbone network of the image template matching and positioning model adopts a semantic segmentation network.

[0040] Specifically, in order to better match the bird's-eye view image and the large-scene satellite image, the bird's-eye view image and the large-scene satellite image can be preprocessed in different channels as needed before being input into the model.

[0041] It can be understood that the image template matching positioning model includes a feature fusion module in addition to the feature extraction backbone network. This is because the input of the model includes two image branches (i.e., bird's-eye view image and large-scene satellite image). After the two image branches pass through the feature extraction backbone network, the feature fusion module is needed to fuse the features of the two image branches output by the feature extraction backbone network, and finally obtain the semantic probability prediction map. Feature fusion can adopt the fusion framework in the existing method, which will not be described in detail in the present invention.

[0042] Since semantic features can effectively deal with nonlinear radiation differences and appearance distortions between images, the cross-view image template matching and positioning method for large scenes provided in an embodiment of the present invention chooses to match the semantic features between two images rather than local features, thereby converting the image template matching and positioning problem from a traditional feature matching task to a semantic probability prediction task. The output result of the entire image template matching and positioning model is a probability prediction map, which can effectively improve the positioning accuracy and efficiency of cross-view image template matching and positioning in large scenes.

[0043] In one embodiment, Figure 2 As shown in the figure, the feature extraction backbone network adopts the DeepLabv3+ network.

[0044] Specifically, in large-scene satellite images, key foreground features that match the converted ground images (i.e., bird's-eye view images) may be contained in complex background information, so the selected semantic segmentation network should be able to better separate the foreground information from the background information; in addition, there is a certain scale difference between the converted ground images and the large-scene satellite images, so the selected semantic segmentation network contains an expanded space pyramid pooling mechanism that should be able to reduce this scale difference. The DeepLabv3+ network is a professional semantic segmentation network that can well meet the above two requirements. It can be understood that the embodiment of the present invention only takes the DeepLabv3+ network as an example, and does not specifically limit the network structure of the semantic segmentation network.

[0045] The overall network architecture of DeepLabv3+ network is as follows Figure 2 As shown in the figure. The network innovatively introduces an encoding-decoding structure. The encoding network is used to gradually reduce the spatial dimension of the feature map, thereby allowing the capture of long-distance information; the decoding network is used to gradually restore the target details and spatial dimensions. The backbone of the encoding network is a deep convolutional network with dilated convolutions, which uses the ResNet50 operator and adds dilated convolutions to increase the receptive field while retaining detail information. The encoding network structure is shown in the figure. Figure 3 shown.

[0046] The geometric mapping relationship between ground panoramic images and satellite images realizes the perspective conversion of images, so that ground panoramic street views have visual similarity features that match satellite images. However, there is still a large scale difference between the converted ground images and the large-scene satellite images that require template matching. Therefore, it is particularly important to extract multi-scale features between images.

[0047] The DeepLabv3+ network contains the Expanded Spatial Pyramid Pooling (ASPP) structure. The ASPP module has multiple parallel dilated convolutional layers with different sampling rates. Convolution kernels with different receptive fields are constructed through different dilation rates to obtain multi-scale object information. The features extracted by multiple different sampling rates are further processed in separate branches to enhance the continuity of information and fuse to generate the final result. Therefore, the ASPP module can reduce this scale difference and increase the scale adaptability of the network. The structure of the ASPP module is as follows: Figure 4 shown.

[0048] Combination Figure 5 and Figure 6 As shown, in one embodiment, a derivation process of the geometric mapping conversion relationship between the ground panoramic image and the perspective image is given.

[0049] It can be understood that there is a fixed geometric relationship between the satellite image and the ground panoramic image taken at its coordinate center. This relationship is called the geometric constraint between the ground panoramic image and the satellite image. This geometric constraint can be used to achieve the geometric mapping transformation between the two images. However, this conversion requires the participation of photographic parameters such as the shooting height of the ground camera, which is difficult to achieve in real application scenarios. Therefore, the embodiment of the present invention explores a geometric mapping method based on the imaging principle of ground panoramic images to achieve domain transfer from ground panoramic images to bird's-eye view images.

[0050] Specifically, the imaging method of the ground panoramic camera is as follows: Figure 5 As shown, it captures a full 360° view on a spherical projection plane and displays it through equirectangular projection, that is, a cylinder is used to represent the image surface, but the image pixels are parameterized in a spherical coordinate system. Let P = (x1, y1, z1) be a point in the ground camera coordinate system, P' = (x2, y2) represent the projection point in the normalized equirectangular coordinate system of the panorama, (ψ, θ) be the spherical coordinates, which can be obtained from (x1, y1, z1) using inverse trigonometric functions, as shown in formulas (1) and (2). The conversion formulas between spherical coordinates (ψ, θ) and normalized equirectangular coordinates (x2, y2) of the panoramic image are shown in (3) and (4).

[0051]

[0052] On the basis of the above-mentioned ground panoramic image imaging principle, in order to realize the mapping of ground panoramic image to bird's-eye view image, a tangent plane is placed at the south pole of the spherical imaging plane as a new imaging plane, such as Figure 6 Assume that the size of the ground panoramic image is H p ×W p , the size of the bird's-eye view image after geometric mapping conversion is H b ×W b Connect the optical center of the camera with each pixel point on the bird's-eye view imaging plane (hereinafter referred to as the bird's-eye view imaging plane), and determine the corresponding pixel position in the ground panoramic image through the intersection of the connecting line and the spherical imaging plane. Suppose the pixel coordinates on the bird's-eye view imaging plane are (u b ,v b ), we can directly get the corresponding camera coordinates (x1, y1, z1). Set the mapping perspective parameter fov = 85° to determine the field of view of the bird's-eye view image. The focal length of the bird's-eye view image during imaging is:

[0053] f=0.5W b / tan(fov) (5)

[0054] Therefore, each corresponding ground panoramic image coordinate (up ,v p ) can be expressed as:

[0055]

[0056] The above formula (6) can be used to realize the mapping conversion between ground panoramic image and bird's-eye view image. The geometric mapping effect is as follows: Figure 7 Using geometric mapping to process ground panoramic images can effectively align the feature domains of the two images, making the corresponding feature maps closer in the two feature spaces.

[0057] Combination Figure 8 As shown, in one embodiment, the training process of the image template matching and positioning model includes: constructing a training data set; each training sample in the training data set is an image pair consisting of a bird's-eye view image and a large-scene satellite image carrying a matching label; wherein the matching label is used to indicate the matching position of the foreground feature of the bird's-eye view image in the large-scene satellite image; and using the training data set to train the semantic segmentation network to obtain the image template matching and positioning model.

[0058] Furthermore, the traditional template matching label sets the correct matching position as a positive sample, and the probability is uniformly set to 1, while the other negative sample positions are all 0, so it is called a binary label. However, this label setting is usually not in line with the actual situation of the embodiment of the present invention, because the network probability prediction results often have different weights at different positions, and the results show that the center of the prediction result has a high weight while the prediction edge has a low weight. If you blindly use a uniform high-weight label for experiments, it will make the network difficult to train.

[0059] Currently, in the task of building extraction, the signed distance function is used to solve this problem. The signed distance function can encode the fine-grained information of the scene, and assigns values ​​to the label by calculating the distance from the pixel to the boundary, thereby converting the problem into a multi-classification problem. Using this idea, this embodiment selects Gaussian labels as matching labels for training samples. Gaussian labels gradually move the center of gravity to the edge position according to the equal-decreasing paradigm, which is filled with the maximum value 1.0 and the minimum value 0. The position weighted distribution can be completed by any probability distribution function. Based on the fact that the present invention studies the correspondence between the centroids of semantic templates, its position must be given a higher weight, which is related to the matching accuracy of the entire template. The result of the discrete Gaussian distribution function is selected as the label value of the semantic space distribution. The graph of the Gaussian kernel function is a bell-shaped positive ground distribution, that is, the closer the label value is to the centroid, the greater the probability. The farther away from the centroid, the smaller the probability. The formula of the Gaussian distribution function is as follows:

[0060]

[0061] Where x is the input, μ is the mean, σ is the standard deviation, and the Gaussian label effect is as follows Figure 8 (c) as shown.

[0062] Furthermore, the generation process of the Gaussian label includes: collecting ground panoramic images with known shooting coordinates, converting the ground panoramic images into bird's-eye view images through geometric mapping; determining the pixel coordinate center of the bird's-eye view image on the large-scene satellite image through coordinate transformation according to the shooting coordinates; on the large-scene satellite image, with the pixel coordinate center as the center of the circle and a set number of pixels as the radius, to produce a label image.

[0063] Specifically, after calculation, the visual feature size of the converted ground panoramic image in the newly cropped 720 pixels × 720 pixels large-scene satellite image is close to 150 pixels × 150 pixels. When the shooting coordinates of the ground panoramic image are known, the pixel coordinate center on the satellite image is determined through coordinate transformation, and a label image is created with the pixel coordinate as the center and 150 pixels as the radius.

[0064] In one embodiment, the process of training the image template matching positioning model adopts a combined loss function consisting of a binary cross entropy loss, a focal loss, and a center of gravity invariant loss.

[0065] Specifically, the present invention chooses to convert the feature matching task into a semantic probability prediction problem. The overall model output result is a probability prediction map, and the label of the model training is a Gaussian probability map. Because the training of the model is similar to the change detection task, the loss function of the change detection task is introduced, and the cross entropy loss is selected to predict the probability distribution of the overall semantic space position. However, because the key matching information obtained by converting the ground image only accounts for a small part of the large-scene satellite image, most of it is useless background information, and the background information dominates the training loss, that is, the extreme imbalance of positive and negative samples. The existence of this defect will cause the loss function to tilt, and it will tend to the category with more samples during the model training process, resulting in the poor performance of the model for the category with fewer samples.

[0066] In order to better deal with the classification prediction and imbalance problems of positive and negative samples in labels, the present invention chooses to adopt a composite loss function that combines binary cross entropy loss and focal loss, and introduces the invariant loss of centroid coordinates proposed in the remote sensing image matching task. By taking the centroid position of the predicted result and the centroid coordinates of the label as the matching point and minimizing the difference between the two, the model is constrained to correctly predict the true matching position.

[0067] The binary cross entropy loss is used to deal with the binary classification situation. The final prediction result of the model has only two categories, that is, the matching area is classified as 1 and the non-matching area is classified as 0. The predicted probability for each category is p and 1-p. The expression at this time is shown in (8):

[0068]

[0069] where y i represents the label of sample i, the positive class is 1 and the negative class is 0; p i It represents the probability that sample i is predicted to be a positive class.

[0070] Focus loss is a dynamically scaled cross entropy loss that specifically addresses the imbalance problem between positive and negative samples. It uses a dynamic scaling factor to dynamically reduce the weight of easily distinguishable samples during training, thereby quickly focusing on samples that are difficult to distinguish. Its formula is shown in (9):

[0071]

[0072] make Then the expression of focal loss can be unified as:

[0073] L2=-α(1-p t ) γ log(p t ) (10)

[0074] Where p represents the label value, represents the predicted value of the prediction, and α and γ both represent modulatable parameters.

[0075] The center-of-gravity invariance loss converts semantic space position alignment into point-to-point matching, and achieves the purpose of accurately locating the template position by penalizing the predicted center position offset. The specific formula is as follows:

[0076]

[0077] Where V(i,j) is the semantic space position probability value of point (i,j), M ij Represents the sum of the semantic position probabilities of all points (i, j).

[0078]

[0079] Among them, M i and M j denote the weights of all output positions in the i and j coordinates respectively. Then, use M ij 、M i and M j Calculate the weighted average of the prediction results in the i and j coordinates to obtain the center of gravity position coordinates. Therefore, the formula can be expressed as:

[0080]

[0081] So the barycentric coordinate invariance loss can be expressed as:

[0082] L3=(x b -x true ) 2 +(y b -y true ) 2 (15)

[0083] where x true and true Represents the true coordinate position of the template center of gravity in the reference image. Finally, combining all losses, the overall loss of this model is:

[0084] Loss=L1+L2+L3 (16)

[0085] Based on the combined loss function provided in the embodiment of the present invention, model training can be better constrained, thereby improving model performance.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cross-view image template matching positioning method for large scenes, characterized in that: include: Step 1: Convert the ground panoramic image that conforms to the geometric mapping principle into a bird's-eye view image through geometric mapping; Step 2: Input the bird's-eye view image and the large-scene satellite image into the image template matching and positioning model to obtain a semantic probability prediction map; wherein the feature extraction backbone network of the image template matching and positioning model adopts a semantic segmentation network.

2. The cross-view image template matching and positioning method for large scenes according to claim 1 is characterized in that: The training process of the image template matching positioning model includes: Constructing a training data set; each training sample in the training data set is an image pair consisting of a bird's-eye view image and a large-scene satellite image carrying a matching label; wherein the matching label is used to indicate a matching position of a foreground feature of the bird's-eye view image in the large-scene satellite image; The semantic segmentation network is trained using the training data set to obtain the image template matching positioning model.

3. The cross-view image template matching and positioning method for large scenes according to claim 2 is characterized in that: The position label adopts a Gaussian label; the generation process of the Gaussian label includes: Collecting ground panoramic images with known shooting coordinates, and converting the ground panoramic images into bird's-eye view images through geometric mapping; Determine the pixel coordinate center of the bird's-eye view image on the large-scene satellite image by coordinate conversion according to the shooting coordinates; On the large-scene satellite image, a label image is produced with the pixel coordinate center as the center of a circle and a set number of pixels as a radius.

4. The cross-view image template matching positioning method for large scenes according to claim 2, characterized in that: In the process of training the image template matching positioning model, a combined loss function consisting of a binary cross entropy loss, a focal loss, and a center of gravity invariant loss is adopted.

5. The cross-view image template matching and positioning method for large scenes according to any one of claims 1 to 4, characterized in that: The geometric mapping conversion relationship between the ground panoramic image and the bird's-eye view image is: Where, f = 0.5W b / tan(fov) represents the focal length of the bird's-eye view image, fov represents the given mapping perspective parameter, H p and W p Respectively represent the height and width of the ground panoramic image, H b and W b Respectively represent the height and width of the bird's-eye view image, (u b ,v b ) represents the pixel coordinates on the imaging plane of the bird's-eye view image, (u p ,v p ) represents the ground panoramic image coordinates.

6. The cross-view image template matching and positioning method for large scenes according to any one of claims 1 to 4, characterized in that: The feature extraction backbone network adopts the DeepLabv3+ network.

Images