A scene matching method for remote sensing images combined with view angle conversion

Abstract

The present application belongs to the technical field of image processing, and particularly relates to a remote sensing image scene matching method combined with view angle conversion. The method comprises the following steps: 1) obtaining multi-view images of a target, and performing view angle conversion on the multi-view images to obtain aerial view images of the target; 2) extracting features of the aerial view images by using a constructed feature extraction model to obtain aerial view image fusion features fused with high spatial resolution and rich semantic information; 3) extracting features of each region of a satellite image by using the constructed feature extraction model to obtain satellite image fusion features of each region; and 4) matching the aerial view image fusion features with the satellite image fusion features of each region respectively to find a region with the best matching degree with the aerial view images, so as to realize scene matching. The present application is suitable for various regions with distinguishable features, and can improve the accuracy and success rate of the matching result through view angle conversion and feature fusion, so that more reliable results can be obtained.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a remote sensing image scene matching method combined with view angle conversion. BACKGROUND

[0002] Cross-view image matching technology refers to a technology of matching a ground image (multi-angle side view image) with a satellite image (space view image) with known position information, so as to obtain position information of the ground image. The technology has important value in target tracking and positioning based on open source information, target positioning without Global Positioning System (GPS) and the like auxiliary information, and the like, and has important significance for forming a multi-level full-space monitoring system based on remote sensing images.

[0003] Early multi-scale cross-view image matching algorithms have problems such as long time consumption, being easily disturbed by other factors, high requirement for image quality, and poor applicability to matching tasks in a large range of scenes. Considering that traditional scene matching methods gradually have difficulty in meeting current task requirements in terms of efficiency and accuracy, deep learning technology is introduced into the matching work, and different degrees of progress have been made in various technical indicators. The current mainstream methods mainly include model-based methods and image representation-based methods.

[0004] The model-based method is to extract representative features according to an existing model or a model reconstructed based on original images, to explore the correspondence relationship of features under different view angles, and to obtain a positioning point through a certain matching strategy. The representation-based method is to extract point lines and the like features from images, and then to directly match the images through feature coding and description. The current superior algorithms almost all use global feature description to establish a graph and graph matching relationship between satellite images and ground images, but such a matching relationship between graphs can only be applied to small satellite images, and it is very difficult to directly match large satellite images and ground images, and often needs to be preprocessed in terms of cutting the satellite images. Therefore, after the matching relationship between the space view image and the ground image is established, only a rough regional positioning can be performed. In addition, these methods mostly depend on large data sets and do not have good generalization ability, for example, a model trained on a city data set is often no longer applicable in rural areas, and differences in building styles in different countries will greatly reduce the model accuracy. SUMMARY

[0005] The present application aims to provide a remote sensing image scene matching method combined with view angle conversion, to solve the problem that the accuracy of traditional scene matching methods is difficult to meet current requirements.

[0006] To solve the above technical problems, the application provides a remote sensing image scene matching method combined with view conversion, which comprises the following steps:

[0007] 1) Obtain multi-view images of a target, and perform view conversion on the multi-view images to obtain an aerial view image of the target; wherein the aerial view image is an image taken from a top view angle, and the multi-view images are images taken from different angles;

[0008] 2) Extract features of the aerial view image by using a constructed feature extraction model to obtain aerial view image fusion features that fuse high spatial resolution and rich semantic information;

[0009] The feature extraction model is an improved VGG19 network model, and the improved VGG19 network model comprises a feature fusion processing module, which is used for performing feature fusion processing on low-scale features C3 output before the third stage of pooling in the VGG19 network, medium-scale features C4 output before the fourth stage of pooling and high-scale features C5 output before the fifth stage of pooling, and the feature fusion processing mode is as follows:

[0010] The medium-scale features C4 are subjected to pooling processing to obtain first features, the low-scale features C3 are subjected to a convolution layer to obtain second features, the medium-scale features C4 are subjected to upsampling processing to obtain third features, the second features and the third features are subjected to fusion processing, and then the first features are fused with the fusion processing result to obtain first fusion features;

[0011] The high-scale features C5 are subjected to pooling processing to obtain fourth features, the medium-scale features C4 are subjected to a convolution layer to obtain fifth features, the high-scale features C5 are subjected to upsampling processing to obtain sixth features, the fifth features and the sixth features are subjected to fusion processing, and then the fourth features are fused with the fusion processing result to obtain second fusion features;

[0012] The first fusion features and the second fusion features are subjected to fusion processing to obtain a final fusion processing result;

[0013] 3) Extract features of each region of the satellite image by using the constructed feature extraction model to obtain satellite image fusion features of each region;

[0014] 4) Match the aerial view image fusion features with the satellite image fusion features of each region respectively to find a region with the best matching degree with the aerial view image, and realize scene matching.

[0015] The beneficial effects are that: because the low-layer feature resolution is higher and contains more position information, and the high-layer feature has strong semantic information, the application performs fusion processing on features of different scales to obtain rich fusion features containing high spatial resolution and rich semantic information, the specific processing mode is to improve the VGG19 network model, the improvement mode is to set a feature fusion processing module, the feature fusion processing module is used for performing feature fusion processing on the low-scale feature C3 output before the third stage of the VGG19 network, the medium-scale feature C4 output before the fourth stage, and the high-scale feature C5 output before the fifth stage, so as to improve the matching ability of the network to small targets in the image and the comprehensive ability of resisting large-scale differences, and then add a pooling strategy to obtain richer feature information, so as to obtain the final fusion processing result; the extracted space image fusion features are matched with the satellite image fusion features of each region respectively, and finally high-quality matching results are obtained, different scale features are fully utilized to realize scene matching and improve the scene matching precision.

[0016] Further, in step 2), the following formula is used for fusion processing:

[0017]

[0018] Wherein, When the fifth feature and the sixth feature are fused, the corresponding X i The second feature and the third feature are fused. When the first fusion feature is obtained, the corresponding X i The second feature and the third feature are fused. When the fifth feature and the sixth feature are fused, the corresponding X i The fifth feature and the sixth feature are fused. When the second fusion feature is obtained, the corresponding X i The fifth feature and the sixth feature are fused.

[0019] Further, in step 2), the following formula is used for fusion processing to obtain the final fusion processing result:

[0020]

[0021] Wherein, The final fusion processing result is obtained, N=2, and the corresponding X1, X2 are the first fusion feature and the second fusion feature.

[0022] The beneficial effect is that: according to the extracted features, the multi-scale features are fused to obtain the fusion features containing high spatial resolution and rich semantic information, the global features and local features of the image are fully utilized, and therefore the accurate matching of the space-air image is realized.

[0023] Further, in step 1), the following method is used to obtain the empty view image:

[0024] ①Obtain the multi-view image of the target, perform point cloud reconstruction to obtain the point cloud data of the target;

[0025] ②Define a projection plane, and project the point cloud data of the target to the defined projection plane according to the following formula:

[0026]

[0027] Wherein, T is a transformation matrix composed of eigenvalues of the covariance matrix of the sample X, X is the value of the reconstructed point before coordinate system transformation, and X is the mean value of the discrete three-dimensional points;

[0028] ③Interpolation operation is performed on the projected point cloud data to obtain a digital surface model (DSM), and the obtained digital surface model (DSM) is used to calculate as follows to obtain the empty view image point x rec :

[0029] x rec =λ(X dsm -X dsm_min )

[0030] Wherein, λ is the proportional relationship between the DSM and the empty view image pixel, X dsm is the value on the DSM, and X dsm_min is the minimum value of the DSM.

[0031] The beneficial effect is that: the obtained multi-view image is reconstructed to obtain the target point cloud data, the value X of the reconstructed point before coordinate system transformation is subtracted from the mean value X of the discrete three-dimensional points, and multiplied by the transformation matrix T composed of the eigenvalues of the covariance matrix of the sample X, to obtain the projection plane X PCA . Combined with the proportional relationship λ between the DSM and the empty view image pixel, the value X dsm on the DSM, and the minimum value X dsm_min of the DSM, the empty view image point x rec is obtained, and the accurate matching of the space-air image is realized.

[0032] Further, in step 4), the quality-aware template matching algorithm (QATM) is used to realize the fusion feature matching.

[0033] Further, when the feature fusion matching is performed by using the quality-aware template matching (QATM) algorithm, the matching score between the template and the image is evaluated according to the defined QATM(s,t), and the QATM is defined as the likelihood product between the feature s and the feature t:

[0034] QATM(s,t)=L(t|s)·L(s|t)

[0035]

[0036]

[0037] wherein L(t|s) is the soft ranking of the current patch t compared with all other patches in the template image, L(s|t) is the possibility score of the matching t in s, alpha is a coefficient parameter, rho(·) is a predefined similarity measure between s and t, f s and f t are the feature representations of s and t;

[0038] According to the matching quality score, the region with the best matching degree is obtained as follows:

[0039]

[0040] q(r s )=max{QATM(s,t)|t∈T}

[0041] wherein q(r s ) represents the matching quality function.

[0042] The beneficial effects are as follows: the QATM of the present application introduces the feature pyramid module to solve the scale difference problem between the template image and the sample image after the perspective conversion. The QATM is defined as the likelihood product between the feature s and the feature t, the coefficient parameter alpha, the predefined similarity measure rho(·) between s and t are combined, and the matching score between the template and the image is evaluated according to the obtained result QATM(s,t); then the region with the best matching degree R * is obtained according to the matching quality function q(r s ), so as to realize the accurate matching of the space-air image.

[0043] Further, the optimal value of the coefficient parameter is determined by using the following method:

[0044]

[0045] wherein L(t|s) + indicates that the matching similarity increases with the increase of alpha, L(t|s) - is on the contrary, and alpha * indicates the optimal value.

[0046] The beneficial effect is that the optimal value of the coefficient parameter alpha is obtained according to the matching similarity with the increase and decrease of alpha * , and the ground-air image accurate matching is realized.

[0047] Further, the pooling processing is global average pooling processing.

[0048] The beneficial effect is that the more accurate matching result is obtained by using the more abundant feature information of the global average pooling.

[0049] Further, the digital surface model (DSM) is obtained by using the B-spline fitting method for interpolation operation.

[0050] The beneficial effect is that the digital surface model (DSM) is obtained by combining the B-spline fitting method for interpolation operation, and the ground-air image accurate matching is realized. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 is the method flowchart of the present application;

[0052] Figure 2 is the view conversion flowchart of the present application;

[0053] Figure 3 is the feature extraction and fusion schematic diagram of the present application;

[0054] Fig. 4(a) is the CVUSA experimental data example diagram of the present application;

[0055] Fig. 4(b) is the Vo and Hays experimental data example diagram of the present application;

[0056] Fig. 4(c) is the Google experimental data example diagram of the present application;

[0057] Fig. 4(d) is the uav experimental data example diagram of the present application;

[0058] Fig. 5(a) is the first center point offset visualization diagram example diagram of the present application;

[0059] Fig. 5(b) is the second center point offset visualization diagram example diagram of the present application;

[0060] Fig. 6(1a)-(1e) is the multi-view of the first scene of the present application;

[0061] Fig. 6(1f) is the multi-view generated air view image result diagram of the first scene of the present application;

[0062] Fig. 6(2a)-(2e) is the multi-view of the second scene of the present application;

[0063] Figure 6(2f) is a diagram showing the result of generating a multi-view aerial image for the second scenario of the present invention;

[0064] Figures 6(3a)-(3e) are multiple views of the third scenario of the present invention;

[0065] Figure 6(3f) is a diagram showing the result of generating a spatial view image for the third scenario of the present invention;

[0066] Figure 7(a) is an example of the specific location of the city image to be matched in the satellite image according to the present invention;

[0067] Figure 7(b) is an example of the specific location of the rural image to be matched in the satellite image according to the present invention;

[0068] Figure 8(1a) is a label image of the first scene of the present invention;

[0069] Figure 8(1b) is a matching result diagram of the first scenario of the present invention;

[0070] Figure 8(1c) is a matching result diagram of Method 2 in the first scenario of the present invention;

[0071] Figure 8(1d) is a matching result diagram of the method of the present invention in the first scenario of the present invention;

[0072] Figure 8(2a) is a label image of the second scene of the present invention;

[0073] Figure 8(2b) is a matching result diagram of Method 1 in the second scenario of the present invention;

[0074] Figure 8(2c) is a matching result diagram of Method 2 in the second scenario of the present invention;

[0075] Figure 8(2d) is a matching result diagram of the method of the present invention in the second scenario of the present invention;

[0076] Figure 8(3a) is a label image of the third scene of the present invention;

[0077] Figure 8(3b) is a matching result diagram of method one in the third scenario of the present invention;

[0078] Figure 8(3c) is a matching result diagram of Method 2 in the third scenario of the present invention;

[0079] Figure 8(3d) is a matching result diagram of the method of the present invention in the third scenario of the present invention;

[0080] Figure 8(4a) is a label image of the fourth scenario of the present invention;

[0081] Figure 8(4b) is a matching result diagram of Method 1 in the fourth scenario of the present invention;

[0082] Figure 8(4c) is a matching result map of the second method of the fourth scenario of the present application;

[0083] Figure 8(4d) is a matching result map of the present application method of the fourth scenario of the present application;

[0084] Figure 8(5a) is a label image of the fifth scenario of the present application;

[0085] Figure 8(5b) is a matching result map of the first method of the fifth scenario of the present application;

[0086] Figure 8(5c) is a matching result map of the second method of the fifth scenario of the present application;

[0087] Figure 8(5d) is a matching result map of the present application method of the fifth scenario of the present application;

[0088] Figure 9 is a threshold-success rate curve of the present application;

[0089] Figure 10(a) is a good texture image data example map of the present application;

[0090] Figure 10(b) is a good image data example map of the present application;

[0091] Figure 10(c) is a flaw texture image data example map of the present application;

[0092] Figure 10(d) is a flaw image data example map of the present application. DETAILED DESCRIPTION

[0093] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application is further described in detail below in combination with the drawings and specific embodiments.

[0094] Embodiment of the present application combining the scene matching method of remote sensing image with perspective conversion:

[0095] The embodiment of the present application combining the scene matching method of remote sensing image with perspective conversion is briefly as follows: Figure 1As shown, the whole is divided into three parts. The first part is the multi-view image perspective conversion, using the portability and flexibility of handheld camera to obtain the multi-view image of the ground object, defining the projection plane after establishing the target point cloud by dense matching, and using principal component analysis (PCA) to complete the projection to generate the empty view image; the second part is multi-scale feature extraction and fusion, using the semantic information of VGG network deep features and the spatial information of shallow features, so that the fused features can have high resolution and strong semantic information to alleviate the difficulty of matching large-scale difference images; the third part is the depth template matching module, which uses QATM to obtain the soft ranking of matching quality to complete the screening and update of the matching results, and finally obtains the high-quality matching results. The method comprises the following steps:

[0096] 1) Multi-view image perspective conversion. Obtain a multi-view image of a target, and perform perspective conversion on the multi-view image to obtain an empty view image of the target; wherein the empty view image is an image taken from a top-down angle, and the multi-view image is an image taken from different angles.

[0097] The positioning task based on ground images usually needs to use the GPS information carried by the sensor, but due to the influence of factors such as obstruction and interference of GPS signal, the use of GPS to assist the matching and positioning of ground and air images will be limited in many actual situations. Remote sensing images have the characteristics of large field of view, object rotation, small object, and contain rich texture, color and shape features, and the relative positioning between remote sensing images can be completed by using these characteristics. Therefore, the present application selects the image visual matching mode to replace the GPS mode for the matching and positioning task of ground and air images.

[0098] Due to the dramatic change in perspective between ground images and air images, there is usually a lack of reliable environmental features during matching. If the perspective of ground and air images can be unified through multi-view images of the ground, or the negative effects caused by the perspective difference can be eliminated as much as possible, the feasibility and accuracy of the matching task of ground and air images can be greatly improved. Considering that the rapid acquisition technology of images is quite mature today, researchers can obtain good quality digital images anytime and anywhere. On the other hand, the advantage of dense matching to generate point clouds over other methods is that it has lower cost and can preserve the texture information of the target object to a large extent, and these texture information will play a crucial role in the matching process of ground and air images. Therefore, the multi-view images of the ground object are obtained by using a camera, and the perspective conversion is completed by projecting the point cloud on the plane after generating the point cloud by dense matching.

[0099] The purpose of this invention is to obtain a viewpoint that is the same as or similar to that of aerial view images. Therefore, after reconstructing point clouds from multi-view images, it is necessary to convert them into aerial view images as much as possible. After acquiring point cloud data, it is necessary to preprocess the point cloud data, filter the data and remove some isolated points to generate the point cloud to be projected, select the aerial view plane as the principal plane and establish the model coordinate system. At this time, the point cloud is distributed in three-dimensional space. The experimental data of this invention are all Earth observation data, and it can be assumed that the main components of the point cloud are located on the ground. Due to factors such as terrain undulation, the assumed plane and the actual ground direction may differ, but the angle is small, so the error in this regard has a very slight impact on the results. On the other hand, since the point cloud data is discrete, plane fitting is required when selecting the principal plane. There are many methods for fitting the plane. Least squares, random sample consistency, and PCA are commonly used plane fitting algorithms. Existing technology shows that PCA is most suitable for processing this type of experimental data. If the ground is approximated as a plane, when the point cloud is basically located on a plane, the first principal component of PCA corresponds to the ground. The preprocessed 3D point cloud data is used as sample points to construct a covariance matrix. The eigenvector corresponding to the smallest eigenvalue is then used as the Z-axis. The discrete points are then transformed into a coordinate system, and the Z-axis coordinates of the points are discarded. The point cloud is then projected onto the principal plane to complete the viewpoint transformation. The relationship between the reconstructed points before and after the coordinate system transformation is shown in Equation (1):

[0100]

[0101] Where T is the transformation matrix composed of the eigenvalues ​​of the covariance matrix of sample X. It represents the mean of discrete three-dimensional points.

[0102] On the other hand, after obtaining the dense point cloud, a digital surface model (DSM) is obtained by fitting B-splines and then interpolating. After obtaining the DSM, the generated aerial image points x... rec With the corresponding point X on the DSM dsm The projection between them is calculated as shown in equation (2):

[0103] x rec =λ(X) dsm -X dsm_min (2)

[0104] Where λ represents the ratio between the DSM and the pixels of the aerial image, X dsm_min This is the minimum value of the DSM. The final perspective transition process is as follows: Figure 2 As shown.

[0105] 2) Multi-scale feature extraction and fusion. The feature extraction model constructed is used to extract the features of the empty visual image to obtain the empty visual image fusion features with high spatial resolution and rich semantic information. The feature extraction model constructed is used to extract the features of each region of the satellite image to obtain the satellite image fusion features of each region.

[0106] The matching of QATM needs image features as support. In recent years, deep learning technology has been widely used in image feature extraction. For example, the convolutional neural network (CNN) can learn global features and local features of an image at the same time. This property can simultaneously consider the corresponding relationship between images and images, feature points and feature points in the scene matching of space-air images. However, most current network models directly feed the global features extracted by the network to the last fully connected layer, and the local features learned by the convolutional layer are often ignored. Therefore, the design goal of the present application is to create a network that can extract and fuse multi-scale features from images, fully utilize the global features and local features of the images, and thus realize accurate matching of space-air images.

[0107] VGG increases the depth of the network to affect the performance of the network, uses a small convolution kernel in succession to replace a larger convolution kernel, and increases the depth of the network by using multiple nonlinear layers to ensure learning more complex patterns. In short, a plurality of 3x3 convolution kernels are used to replace a larger convolution kernel, and the purpose is to increase the depth of the network to improve the performance of the network under the condition of having the same receptive field, and the rich features extracted by the multiple layers of convolution contained in the deep network are exactly what the research needs. As described above, the present application selects VGG19 as the baseline network to extract features from images, but directly using the VGG19 model to extract features for the matching task has the problems of low scale tolerance and a large number of false matches. Therefore, in view of the scale problem of the template and the sample image after perspective conversion, the present application modifies the VGG19. The output features used in the original algorithm have strong robustness when the appearance and background change, but they perform poorly in terms of scale. In order to fully utilize features of different scales, the present application extracts the outputs before pooling in the conv3-4, conv4-4, and conv5-4 stages of the VGG19 network as low, medium, and high scale features, respectively, and they are denoted as C3, C4, and C5. The low layer features have high resolution and can contain more position information. The high layer features have strong semantic information but low resolution. Therefore, the present application uses a feature pyramid network to fuse the features learned by different convolution layers to improve the tolerance to scale differences during matching. The fusion method uses summation operation as shown in formula (1) to process, and adds the corresponding channel matrix elements to form a new feature:

[0108]

[0109] The feature pyramid network structure adopts a feature transmission mode of top-down and horizontal connection, fuses shallow features with high spatial resolution and deep features with rich semantic information, thereby improving the matching ability of the network to small targets in the image and the comprehensive ability to resist large-scale differences. Finally, a global average pooling strategy is added to obtain more rich feature information, and the specific structure is as shown in Figure 3 .

[0110] The specific processing mode is: the mesoscale feature C4 is processed by pooling to obtain a first feature, the low-scale feature C3 is processed by a convolution layer to obtain a second feature, the mesoscale feature C4 is processed by upsampling to obtain a third feature, the second feature and the third feature are fused, and then the first feature is fused to obtain a first fused feature;

[0111] 3) Deep template matching. The empty view image fusion features are matched with the satellite image fusion features of each region respectively to find the region with the best matching degree with the empty view image, thereby realizing scene matching.

[0112] Directly using feature points to find the corresponding relationship on large-scale image scene matching often causes algorithm failure or a large number of false matches, so it is necessary to aggregate multiple features on the image to make up for this defect. The present application draws lessons from the idea of template matching algorithm, defines the window to expand the available feature points while defining the range, thereby aggregating local features. QATM is a template matching algorithm that can act on deep neural networks, and the main idea is to use the soft ranking between all matching pairs to evaluate the quality of the matching pairs, that is, the uniqueness of the matching result. The matching score accurately reflects the degree of distinction of the matching, and more importantly, it has learnable parameters and can be differentiated. Therefore, this method can be combined with deep learning or embedded in existing models to achieve high-precision matching. This algorithm for determining the best matching region according to the window matching score is extremely suitable for large-scale scene matching tasks, and if the fusion features are input into QATM, the purpose of the present application can be well achieved.

[0113] Suppose that the image to be tested S (satellite image) and the template T (empty view image) are obtained, the image patch s and the template patch t are obtained, and f s and f twhere ρ(·) is a predefined similarity measure between s and t. L(t|s) is the soft ranking of the current patch t compared to all other patches in the template image, which can be viewed as an activation layer, and L(s|t) is the likelihood score of t matching in s. To evaluate the matching quality, the algorithm defines Quality(s, t) as the score that evaluates the matching quality between s and t, and outputs the best matching region by maximizing the overall matching quality. Then QATM can be defined as the likelihood product between s and t:

[0114] QATM(s, t) = L(t|s) · L(s|t) (4)

[0115]

[0116]

[0117] When f s and f t are similar, ρ(·) will give a higher value, and when t and s are truly matched, ρ(f s , f t ) should be greater than ρ(·) in those unmatched cases, i.e., the highest score ρ(·) is the best match. After choosing a proper α parameter and using the softmax activation, the score should be 1 in the ideal one-to-one case. The α parameter acts to make the matching patch ranking score as close to 1 as possible, and the non-matching patch ranking score should be as close to 0 as possible. Therefore, training a proper α will greatly improve the efficiency and accuracy of the matching results, and its calculation method is shown in equation (7). In equation (7), L(t|s) + indicates that the matching similarity increases with the increase of α, and L(t|s) - is the opposite, and α * indicates the optimal value. When obtaining the paired results between S and T, the matching quality of the region of interest (ROI) can be represented as equation (8):

[0118]

[0119] q(r s ) = max{QATM(s, t) | t ∈ T} (8)

[0120] where q(s r ) represents the matching quality function. Finally, the region that maximizes the overall matching quality, i.e., the best matching region R, can be found.

[0121]

[0122] The effectiveness of the method of the present application will be illustrated below by applying the method of the present application to specific examples.

[0123] 1) Experimental data and environment.

[0124] The present application uses Google Earth L19-L20 level satellite images of Zhengzhou area, CVUSA urban area satellite images, Vo and Hays Boston urban area satellite images and unmanned aerial vehicle images of Zhengzhou area and Dengfeng area as the space view image, and the target image collected by handheld photography equipment as the ground multi-view image. The spatial resolution of the space view image data is in the order of meters or sub-meters, the handheld photography equipment is mainly mobile phone, the collection height is about two meters, and the data details are shown in Table 1. Figure 4(a) is a CVUSA experimental data example of the present application, Figure 4(b) is a Vo and Hays experimental data example of the present application, Figure 4(c) is a Google experimental data example of the present application, and Figure 4(d) is a uav experimental data example of the present application. The present application adopts a deep learning method in the feature fusion and template matching module, and the training set and the verification set adopt Vo and Hays, CVUSA part of urban area images and collected unmanned aerial vehicle images of Dengfeng area, and the test set is all the collected Google Earth images and ground images of Zhengzhou area. The total amount of experimental samples is 200 large satellite images, 200 unmanned aerial vehicle images and about 640 multi-view images collected according to the ground target. Among them, the multi-view images collected according to the ground target do not participate in the training, and are only used for the view conversion module. The training set, the test set and the verification set are divided in the ratio of 7:2:1, wherein the test set is all the collected Google Earth images and ground images of Zhengzhou area. The network adopts Adam optimizer, the learning rate is 0.00001, the batch size is 4, and a total of 20 times of training is performed.

[0125] Table 1 Details of data set

[0126]

[0127] All programs in the experiment run on the Windows system, Python3.6 and Matlab2018 are used for programming, the CPU is i9-9900, 64Gb. The used GPU is NVIDIA GeForce RTX 3090 24Gb, and the deep learning network is realized using the Pytorch framework. Except that Deformable Diversity Similarity (DDIS) is performed in Matlab, the rest of the network is compiled using PyCharm.

[0128] 2) Evaluation method.

[0129] The application uses three indexes of matching success rate, number of mis-matches and center point offset to evaluate network performance. The matching success rate and the number of mis-matches are used as evaluation indexes for comparing network performance, and the center point offset is added as a further evaluation index for the successful matching results of each network. When the overlap rate of the matching result and the real ground target label exceeds 60%, it is considered that the matching is successful, and the number of successful matches is denoted as P a , the total number of matches is denoted as P, the matching success rate O is calculated, and the calculation formula is as follows:

[0130]

[0131] In the experimental data used in the application, only one-to-one correct matching meets the experimental requirements, so the number of mis-matches P b is also an important index for measuring network performance, and the calculation formula is as follows:

[0132] P b = P-P a (11)

[0133] The center point offset is calculated in pixel units to calculate the error between the matching result and the true value label, which can more accurately reflect the actual situation of the matching error, and is specifically represented as the distance between the intersection points of the diagonal lines of the rectangular frames of the matching result and the true value label. When each network can successfully perform matching and the matching success rate is not much different, this index can more objectively evaluate network performance. Fig. 5(a) is a first center point offset visualization example diagram of the application, and Fig. 5(b) is a second center point offset visualization example diagram of the application.

[0134] 3) Experimental results and analysis.

[0135] a, view conversion and matching result.

[0136] In order to verify the effectiveness of the view conversion module proposed in the application, ground close-range images are acquired by a handheld photography device within the corresponding range of the aerial view image for experimental verification. The experimental range is the urban area of Zhengzhou and the rural area of Dengfeng. The real ground target is about 6.5m x 6.5m, and in the experiment, 18 ground view images are collected in the to-be-determined area using a handheld photography device. The collected images try to cover the panorama of the to-be-determined area, and the number of collected images depends on the size of the ground target, as shown in Figs. 6(1a)-(1e), Figs. 6(2a)-(2e) and Figs. 6(3a)-(3e), and the corresponding aerial view images are generated as Figure 6(1f) 、 6(2f)and 6(3f) are shown. The test results show that the multi-view images taken by the handheld photography device can generate high-quality aerial images after the view conversion. After generating the corresponding aerial images, they can be input into the subsequent network to complete the large-scale difference between the ground and the air matching and positioning task, and the data examples are shown in Figs. 7(a) and 7(b).

[0137] To verify the effectiveness of the multi-scale quality perception template matching method designed in the application, several advanced template matching methods are compared using the same test data, wherein the test data are aerial images in Zhengzhou area which are not involved in the training, and the architectural style is greatly different from that in Boston, Seattle and other places in the United States in the training data. The first comparison method is Deformable Diversity Similarity (DDIS), and the second comparison method is the unimproved QATM. The label images and the matching results of different methods are shown in Figs. 8(1a)-(1d), 8(2a)-(2d), 8(3a)-(3d), 8(4a)-(4d) and 8(5a)-(5d), wherein the first three rows are examples of matching results in urban areas, and the last two rows are examples of matching results in rural areas. On the other hand, since the aerial images generated from the ground side view images are small in target and lack of information, a large range of satellite images must contain a large amount of similar information. In most ground-to-air matching tasks, only one-to-one correct matching can be considered as accurate and efficient, so eliminating the wrong matching is also a major factor affecting the effectiveness of the method. Based on the above analysis, the application selects non-maximum suppression to eliminate the wrong matching. The application uses three methods to conduct experiments on the collected data set, and according to the test results, the threshold and success rate curves are drawn to determine the best matching relationship, and the results are shown in Figs. 9(a) and 9(b). Figure 9

[0138] Figs. 8(1a)-(1d), 8(2a)-(2d), 8(3a)-(3d), 8(4a)-(4d), 8(5a)-(5d) and Figure 9 ​It can be seen from the matching results that, overall, DDIS is at a disadvantage in the multi-scale cross-view image matching task compared to other algorithms, and QATM can match the correct target in most cases, but is accompanied by more false matches. The reason is mainly that the data set collected by the present application has too large a difference in scale, method one does not have high scale robustness, and method two only uses VGG19 deep features, which are characterized by rich semantic information but lack spatial information, so there are a large number of false matches. In addition, the VGG19 network is deep, so compared to method one, the other two algorithms can extract high-dimensional features to resist the influence of large scale differences, and the method of the present application adds a multi-scale feature fusion mechanism compared to method two, which improves the accuracy of matching. On the other hand, rural areas are mostly reservoirs, farmland, roads, hills, etc., and their field of view is relatively open. Therefore, the satellite images in rural areas have lower background complexity than in urban areas, but lack obvious targets and have a large number of similar ground objects. From the results of Figures 8(1a)-(1d), 8(2a)-(2d), 8(3a)-(3d), 8(4a)-(4d), and 8(5a)-(5d), when the ground target with distinguishable features can be obtained in the experiment, the method of the present application is still applicable in rural areas, but is accompanied by obvious redundant false matches. Comprehensive analysis of the results shows that the method of the present application is applicable to target areas with obvious features, regardless of whether it is in a city or a rural area, and the accuracy of the method of the present application is reduced in areas with a large number of repeated targets.

[0139] b. Precision evaluation and ablation experiment.

[0140] In order to further explore the feasibility of the method of the present application, the present application selects a texture target area with good quality in view conversion and a multi-view dense matching generated defect image (i.e. the generated empty view image has a large distortion or is missing) for verification, as shown in Figures 10(a), 10(b), 10(c), and 10(d). In order to further evaluate the generalization ability of the method of the present application, the average matching success rate and the average number of false matches are counted in urban and rural areas respectively, and the results are shown in Table 2.

[0141] It can be seen from the analysis of the experimental results that the method has effectiveness in both urban and rural areas. However, the accuracy in rural areas is lower than that in urban areas because there are fewer targets with obvious distinguishable features in rural areas. On the other hand, when the ground multi-view images cannot cover the target area or the image overlap is insufficient, the flawed images shown in FIG. 10(c) and FIG. 10(d) will appear. Using the flawed images as input to the network model will have a negative impact on the results, reducing the matching accuracy, which proves that the quality of the view conversion result directly affects the accuracy of the matching result. Moreover, the impact of flawed images on urban areas is greater than that on rural areas, which may be due to the missing pixels and the distortion of the features in the flawed images, reducing the reliability of the features and leading to false matching. In rural areas, there are already a large number of similar targets, and the reduction in the reliability of the features in the flawed images has a similar or related impact on the network as the confusion of similar targets, so the impact of flawed images on urban areas is greater, while the impact on rural areas is smaller.

[0142] Table 2 Comparison of urban-rural experimental results

[0143]

[0144] In order to explore the influence and effect of the feature pyramid network structure on the matching result, the present application carries out an ablation experiment: the features extracted by using conv3-4, conv4-4, conv5-4, fusion conv3-4 and conv5-4, fusion conv4-4 and conv5-4 and the method of the present application are respectively input into the model, and the average success rate of the input view angle converted data, the input unconverted data and the defect image is obtained to evaluate the effect and effect. On the other hand, according to the final matching result of the network using various fusion features, the center point offset is counted, and the visual result is shown in Fig. 5(a) and Fig. 5(b), wherein the small box is the ground truth label, and the large box is the matching result. From the visual result of the matching, it can be seen that the overlap rate of the matching result and the true value label is good, but the center point offset can better reflect the accuracy of the matching result. In addition, 15 groups of ground-air matching results are counted, and the success rate and the center point offset are calculated, as shown in Table 3, wherein the average success rate less than 20% is regarded as the method failure, and the average mismatch number more than 5 is regarded as the method failure. Through the analysis of the effect of the feature fusion module on the final matching result, the matching success rate obtained by fusing the features of conv3-4, conv4-4 and conv5-4 is the highest, which shows that fusing the spatial information of the low-level features and the semantic information of the high-level features can positively affect the overall result, and all the methods input the image data without view angle conversion and fail to perform under the set standard. Although the center point offset is more than 5 pixels, the purpose of the present application is to obtain the regional positioning result of the ground target through matching, so the center point offset error is within the allowable range, and the influence of the present application method is small. In summary, in the ground-air image matching task, the use of fusion features after view angle conversion reduces the scale effect and improves the matching accuracy of small targets.

[0145] Table 3 Ablation experiment result success rate

[0146]

[0147] Note: The bold font is the optimal value of each column, and indicates that the method fails.

[0148] In summary, the present application has the following characteristics:

[0149] (1) Because the low layer feature resolution is high, contains more position information, the high layer feature has strong semantic information, therefore, the application carries out fusion processing to features of different scales to obtain rich fusion features containing high spatial resolution and rich semantic information, the specific processing mode is to improve the VGG19 network model, the improvement mode is to set a feature fusion processing module, the feature fusion processing module is used for carrying out feature fusion processing to low scale features C3 output before the third stage of the VGG19 network, medium scale features C4 output before the fourth stage and high scale features C5 output before the fifth stage, so that the matching ability of the network to small targets in the image and the comprehensive ability of resisting large scale differences are improved at the same time, and richer feature information is obtained by adding the pooling strategy to obtain the final fusion processing result, the extracted space view image fusion features are matched with satellite image fusion features of each region respectively, and finally high-quality matching results are obtained, different scale features are fully utilized to realize scene matching and improve the precision of scene matching.

[0150] (2) According to the extracted features, the multi-scale features are fused to obtain fusion features containing high spatial resolution and rich semantic information, and the global features and local features of the image are fully utilized to realize accurate ground-air image matching.

[0151] (3) The obtained multi-view image is reconstructed to obtain target point cloud data, the value X of the reconstructed point before coordinate system transformation is subtracted from the mean value of discrete three-dimensional points , and multiplied by the transformation matrix T composed of the eigenvalues of the covariance matrix of the sample X to obtain the projection plane X PCA . Combined with the proportional relationship lambda between the DSM and the space view image pixels, the value X dsm on the DSM, the minimum value X dsm_min of the DSM, the space view image point x rec is obtained, and then accurate ground-air image matching is realized.

[0152] (4) The QATM of the application introduces a feature pyramid module to solve the scale difference problem between the template image and the sample image after the perspective conversion. The QATM is defined as the likelihood product between the features s and t, the similarity measure rho(·) between the coefficient parameters alpha, s and t is combined, and the matching score between the template and the image is evaluated according to the obtained result QATM(s,t); according to the matching quality function q(r s ), the area R * with the best matching degree is obtained, and then accurate ground-air image matching is realized.

Claims

1. A scene matching method for remote sensing images combined with view angle conversion, characterized in that, The method comprises the following steps: 1) obtaining multi-view images of a target, and performing perspective conversion on the multi-view images to obtain an aerial view image of the target; wherein the aerial view image is an image taken from a top-down perspective, and the multi-view images are images taken from different angles; 2) extracting features of the aerial view image using a constructed feature extraction model to obtain aerial view image fusion features that fuse high spatial resolution and rich semantic information; The feature extraction model is an improved VGG19 network model, which comprises a feature fusion processing module for performing feature fusion processing on low-scale features C3 output before the third stage of pooling, medium-scale features C4 output before the fourth stage of pooling, and high-scale features C5 output before the fifth stage of pooling in the VGG19 network, and the feature fusion processing mode is as follows: The medium-scale features C4 are processed by pooling to obtain first features, the low-scale features C3 are processed by a convolution layer to obtain second features, the medium-scale features C4 are processed by upsampling to obtain third features, the second features and the third features are fused, and the first features are fused after the fusion to obtain first fusion features; The high-scale features C5 are processed by pooling to obtain fourth features, the medium-scale features C4 are processed by a convolution layer to obtain fifth features, the high-scale features C5 are processed by upsampling to obtain sixth features, the fifth features and the sixth features are fused, and the fourth features are fused after the fusion to obtain second fusion features; The first fusion features and the second fusion features are fused to obtain the final fusion result; 3) extracting features of each region of the satellite image using the constructed feature extraction model to obtain satellite image fusion features of each region; 4) matching the aerial view image fusion features with the satellite image fusion features of each region respectively to find a region with the best matching degree with the aerial view image, and realizing scene matching.

2. The method according to claim 1, wherein, In step 2), the following formula is used for fusion processing: ； wherein, when the first feature is the first fused feature, the first feature is comprises the second feature and the third feature; when the first feature is the first fused feature, the first feature is comprises the second feature and the third feature after fusion processing, and the first feature; when the first feature is the first fused feature, the first feature is comprises the fifth feature and the sixth feature; when the first feature is the first fused feature, the first feature is comprises the fifth feature and the sixth feature after fusion processing, and the fourth feature. 3.The method of claim 1, wherein, In step 2), the following formula is used for fusion processing to obtain the final fusion result: ； Wherein, is the final fusion processing result, N = 2, the corresponding X 1, X 2 are respectively a first fusion feature and a second fusion feature.

4. The method of claim 1, wherein, In step 1), the following method is used to obtain the aerial view image: ①obtain multi-view images of the target, perform point cloud reconstruction to obtain point cloud data of the target; ②define a projection plane, and project the point cloud data of the target to the defined projection plane according to the following formula: ； wherein T is the covariance matrix of the sample X is the transformation matrix composed of the eigenvalues of the covariance matrix of the sample X is the value of the reconstructed point before the coordinate system transformation, and is the mean value of the discrete three-dimensional points (3) performing interpolation operation on the projected point cloud data to obtain a digital surface model (DSM), and using the obtained digital surface model (DSM) to perform the following calculation to obtain the empty-view image point : ； wherein, is the ratio between the DSM and the empty visual image pixel, is the value on the DSM, is the minimum value of the DSM.

5. The method for scene matching of remote sensing images with view conversion according to claim 1, wherein, In step 4), a quality-aware template matching algorithm QATM is used to realize fusion feature matching.

6. The method according to claim 5, wherein, When using the quality-aware template matching (QATM) algorithm for feature fusion matching, the matching score between the template and the image is evaluated according to the defined QATM ( s , t ), and the QATM is defined as the product of the likelihoods between the feature s and the feature t . ； ； wherein, is a soft ranking of the current patch compared to all other patches in the template image, t is s a likelihood score of a match in t is a coefficient parameter, is s and t is a predefined similarity measure between and is a feature representation of s and t ​​​ According to the matching quality score, the region with the best matching degree is obtained as follows: ； wherein denotes the matching quality function, R denotes the best matching region.

7. The method according to claim 6, wherein, The optimal value of the coefficient parameter is determined by the following method: ； wherein, indicates that the matching similarity increases with the increase of on the contrary, indicates the optimal value.​ 8. The method of claim 1, wherein, The pooling processing is global average pooling processing.

9. The method for scene matching of remote sensing images with view conversion according to claim 4, wherein, The B-spline fitting method is used for interpolation operation to obtain a digital surface model DSM.

Images