Image matching optimization method and system based on geometric constraints and convolutional neural network

Abstract

The application discloses an image matching optimization method and system based on geometric constraints and a convolutional neural network, which comprises the following steps: firstly, projecting three-dimensional points of initial matching images to a reference image and a search image respectively to obtain initial matching image pair point coordinates; determining a reference image window and a matching search range with the initial matching image pair as the center; then taking the initial matching image window as a reference, finding the best matching point position in the matching search range by using a convolutional neural network; finally, bringing the optimal matching pair position coordinates into a forward intersection error equation to calculate corresponding photographed object point accurate three-dimensional coordinates. The application adopts spatial geometric projection relationship to constrain the matching search range, which weakens the mismatch caused by repeated textures; the convolutional neural network is used to replace the traditional image gray-based matching optimization method, which reduces the probability of matching failure and mismatch, and improves the matching optimization precision and the completeness of the reconstructed point cloud.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of remote sensing image processing and image matching, and relates to an image matching optimization method and system in remote sensing image processing, in particular to an image matching optimization method and system based on geometric constraints and convolutional neural networks. BACKGROUND

[0002] It has become an effective means to obtain urban three-dimensional information in recent years to quickly recover the three-dimensional model of an object or a scene by using the pixel-by-pixel dense matching of overlapping oblique images. However, the traditional matching optimization method based on grayscale still has defects in matching the areas with depth discontinuity (building corner lines, road boundary lines), weak texture, etc. in artificial buildings. The traditional matching optimization method is based on the cross-correlation of the grayscale of the image window, wherein the larger the matching window, the more the pixels involved in the matching process, the lower the probability of mismatching, and the higher the completeness of the reconstructed point cloud. This is because rich image information is involved in the matching process, which reduces the weight of image noise and thus improves the robustness of image matching. However, when the matching image window is enlarged, the reconstructed point cloud model is deformed more relative to the true ground model. This is because too many pixels are involved in the matching optimization, and the weight of the pixels where the point and line features of the image are located becomes smaller, so the features will be blurred more seriously after matching optimization. The reconstructed model is deformed more. SUMMARY

[0003] To solve the above technical problems, the present application provides an optimization method and system for mining image pixel depth information and improving the matching accuracy and the completeness of the reconstructed point cloud in low-altitude remote sensing images.

[0004] The technical scheme adopted by the method of the present application is as follows: an image matching optimization method based on geometric constraints and convolutional neural networks, comprising the following steps:

[0005] Step 1: project the initial matching image three-dimensional points into the reference image and the search image respectively to obtain the initial matching image pair of point coordinates p r (x r ,y r ), p s (x s ,y s ), and determine the reference image window and the matching search range with the initial matching image pair as the center;

[0006] Step 2: taking the initial matching image window as the reference, find the best matching point position in the matching search range by using the convolutional neural network;

[0007] Step 3: bring the optimal matching pair position coordinates into the forward intersection error equation to calculate the accurate three-dimensional coordinates of the corresponding photographed object points.

[0008] Preferably, in step 1, the coordinates of the object points of the initial matching image are obtained using the following formula;

[0009]

[0010] Among them, (x (r / s) ,y (r / s) ) represents the projection of a 3D point P(X,Y,Z) onto the reference image I. r Or search for images I s The obtained image point p r or p s in I r Or I s The position coordinates in the image are (x0, y0), where (x0, y0) represents the position coordinates of the principal point of the camera in the image, and f represents the principal distance of the camera. s(r / s) ,Y s(r / s) Z s(r / s) ) indicates the shooting reference image I r Or search for images I s The coordinates of the camera's optical center in a three-dimensional coordinate system are captured at any time. Indicates shooting reference image I r Or search for images I s The rotation matrix of the camera during shooting.

[0011] Preferably, in step 1, an n×n pixel image window centered on the projected image point is selected in the reference image as the image window to be matched; and an m×m pixel matching search range centered on the projected image point is selected in the search image; where m and n are preset values, and m>n.

[0012] Preferably, in step 2, (mn)×(mn) image windows and reference image windows are extracted from left to right and from top to bottom within the search range, and the optimal match is found using a convolutional neural network.

[0013] Preferably, in step 2, the matching optimization convolutional neural network includes a convolutional image information extraction module and a fully connected optimal matching search module;

[0014] The convolutional image information extraction module includes 16 convolutional layers arranged in series. The sum of the first to the 15th convolutional layers contains residual blocks. After the 16th layer, a normalization layer, an activation layer, and a residual block are added in sequence.

[0015] The full connection optimal matching search module comprises a window feature vector descriptor submodule and an optimal matching search submodule, the feature vector descriptor submodule is composed of a full connection layer of 48*1000, and the output result is a 1000-dimensional feature vector of the matching image window; the optimal matching search submodule is composed of a full connection layer of 1000*64, and the output result is a similarity measure of the reference image and the search image window.

[0016] As preferred, in step 3, the following formula is used to obtain the three-dimensional coordinates of the corresponding object point of the matching image pair:

[0017]

[0018]

[0019] Wherein, P(X, Y, Z) is the three-dimensional coordinates of the object point to be solved, l1-l6 are coefficients of the equation; l x and l y are constant terms of the equation; f represents the principal distance of the camera, (x0, y0) represents the position coordinates of the image principal point of the camera in the image, p(x, y) represents the optimal matching image point coordinates searched in step 2, (X s , Y s , Z s ) represents the coordinates of the optical center of the camera in the three-dimensional space, represents the rotation matrix of the camera.

[0020] The technical scheme of the system of the application is as follows: an image matching optimization system based on geometric constraints and convolutional neural networks, comprising:

[0021] One or more processors;

[0022] A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors realize the image matching optimization method based on geometric constraints and convolutional neural networks.

[0023] The technical scheme of the application improves the matching efficiency by constraining the matching search range through geometric projection relationship, and weakens the false matching caused by repeated textures, then enriches the information amount of image pixels by mining the deep information of images through convolutional neural networks to realize the matching optimization purpose, and finally completes the acquisition of high-precision and high-integrity three-dimensional point cloud data by calculating the accurate object point three-dimensional coordinates through the space forward intersection of the matching image pair.

[0024] Compared with the prior art, the application has the following beneficial effects:

[0025] (1) Using the spatial geometric projection relationship, the matching search range is constrained from the whole image to a small range of the same name projection beam, which greatly reduces the matching pixel search time and weakens the false matching caused by repeated textures.

[0026] (2) A small image window with few pixels is used as input data, and a matching optimization convolutional neural network without a pooling layer is constructed to achieve the purpose of matching efficiency and high-precision matching.

[0027] (3) The matching optimization convolutional neural network is used to mine the depth information in the image window, replacing the traditional image gray-based matching optimization method, enriching the information amount of the image pixels, reducing the probability of matching failure and false matching, and improving the matching optimization precision and the completeness of the reconstructed point cloud. BRIEF DESCRIPTION OF DRAWINGS

[0028] The technical solutions described herein are further illustrated below using examples and specific embodiments. In addition, some drawings are also used in the process of explaining the technical solutions. For those skilled in the art, other drawings and intentions of the present application can also be obtained without creative labor on the basis of these drawings.

[0029] Figure 1 The flowchart of the embodiment of the present application.

[0030] Figure 2 The schematic diagram of the matching optimization convolutional neural network established in the embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to facilitate those skilled in the art to understand and implement the present application, the present application is further described in detail below in combination with the drawings and examples. It should be understood that the implementation examples described herein are only used to illustrate and explain the present application, and are not used to limit the present application.

[0032] See Figure 1 The image matching optimization method based on geometric constraints and convolutional neural network provided by the present application comprises the following steps:

[0033] Step 1: Project the initial matching image three-dimensional points to the reference image and the search image respectively using the collinear condition equation, to obtain the initial matching image pair of image point coordinates p r (x r ,y r )、p s (x s ,y s );

[0034] In this embodiment, image-related parameters and initial matching 3D point cloud data are first input, and the coordinates of the projection points are calculated using spatial geometric projection relationships. The matching image window and matching search range are determined with the coordinates of the projection image points as the center.

[0035] In one implementation, for each three-dimensional point P(X,Y,Z), the coordinates p of the projected image point are calculated according to the following collinearity condition equation. r (x r ,y r ) and p s (x s ,y s ):

[0036]

[0037] Among them, (x (r / s) ,y (r / s) ) represents the projection of a 3D point P(X,Y,Z) onto the reference image I. r Or search for images I s The obtained image point p r or p s in I r or I s The position coordinates in the image are (x0, y0), where (x0, y0) represents the position coordinates of the principal point of the camera in the image, and f represents the principal distance of the camera. s(r / s) ,Y s(r / s) Z s(r / s) ) indicates the shooting reference image I r Or search for images I s The coordinates of the camera's optical center in a three-dimensional coordinate system are captured at any time. Indicates the shooting reference image I r Or search for images I s The rotation matrix of the camera during shooting.

[0038] In one implementation, the reference image is represented by the projected image point p. r (x r ,y r A 11×11 pixel image window is selected as the image window to be matched, centered on the image. The image is then searched using the projected image point p. s (x s ,y s The matching search range is 19×19 pixels centered on the image. The purpose of selecting an image matching window of 11×11 pixels is to reduce the matching window size and increase the weight of feature pixels. The purpose of constraining the matching search range is to improve matching efficiency and reduce false matches caused by repeated textures.

[0039] Step 2: Using the initial matching image window as a reference, a convolutional neural network is used to find the best matching point within the matching search range;

[0040] See Figure 2 In an embodiment, 64 image windows and a reference image window are constructed by intercepting from left to right and from top to bottom in a search range Figure 2 The matching optimization convolutional neural network shown in the figure is as follows:

[0041] In an embodiment, the input data is an 11x11 pixel RGB color image, and the matching optimization convolutional neural network comprises a convolutional image information extraction module and a fully connected optimal matching search module.

[0042] The convolutional image information extraction module comprises 16 convolutional layers arranged in series, and the convolution kernel size is 3 and the step size is 1; the sum of the first layer to the 15th layer convolutional layers is added with a residual block to weaken image noise, and a normalization layer, an activation layer and a residual block are sequentially added after the 16th layer to suppress gradient vanishing and gradient explosion and accelerate network operation. The purpose of the convolutional layer is to extract deep information in the image and enrich the information amount of the image pixel.

[0043] The fully connected optimal matching search module comprises a window feature vector descriptor submodule and an optimal matching search submodule. The feature vector descriptor submodule is composed of a 48x1000 fully connected layer, and the output result is a 1000-dimensional feature vector describing the global information of the image matching image window; the optimal matching search submodule is composed of a 1000x64 fully connected layer, and the output result is the similarity measure of the reference image and the search image window.

[0044] For the output result of the matching optimization convolutional neural network, the center pixel of the search image window with the highest similarity measure score is selected as the optimal matching point. The purpose of canceling the pooling layer in the matching optimization convolutional neural network described in the application is to retain the deep information of each pixel in the image window to achieve the purpose of high-precision matching.

[0045] Step 3: Enter the optimal matching image pair position coordinates into the forward intersection error equation to calculate the accurate three-dimensional coordinates of the corresponding photographed ground object point.

[0046] In an embodiment, the following formula is used to obtain the three-dimensional coordinates of the corresponding ground point of the matching image pair:

[0047]

[0048]

[0049] Where P(X, Y, Z) is the three-dimensional coordinates of the ground object point to be solved, and l1-l6 are the coefficients of the equation; l x and l yis a constant term of the equation; f represents a principal distance of the shooting camera, (x0, y0) represents position coordinates of an image principal point in the image, p(x, y) represents the optimal matching image point coordinates searched in step 2, (X s ,Y s ,Z s ) represents coordinates of the optical center of the shooting camera in the three-dimensional space, represents a rotation matrix of the shooting camera.

[0050] In the implementation process, it is known that one image point coordinate p(x, y) can establish two linear error equations about (X, Y, Z), then the matching after optimization is obtained, and the matching relative to p ri (x ri ,y r ) and p si (x si ,y si ) is brought into the error equation to establish a 4*3 linear equation group, that is:

[0051]

[0052] that is:

[0053]

[0054] wherein, is a coefficient matrix, is a constant term vector, since the number of linear equations is greater than the number of unknowns, the least square method can be used for solving:

[0055]

[0056] wherein, L T is a transposed matrix of the coefficient matrix L, (L T L) -1 is an inverse matrix of the product of the matrix L T * L.

[0057] The results obtained by optimizing and calculating all initial three-dimensional points one by one are recorded as a txt document, and the document content can provide comprehensive results, including: initial three-dimensional point coordinates, corrected three-dimensional point coordinates, reference image point coordinates, search image point coordinates, optimized matching window correlation coefficient, whether optimization is successful and the like.

[0058] The application further provides an image matching optimization system based on geometric constraints and a convolutional neural network, comprising:

[0059] one or more processors;

[0060] A storage device is configured to store one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the image matching optimization method based on geometric constraints and convolutional neural networks.

[0061] The effectiveness of the present application is verified by simulation experiments as follows:

[0062] The simulation experiment uses two groups of urban area UAV images with overlapping headings, and obtains accurate interior and exterior orientation elements. Each group of images extracts two groups of point cloud data of regularly distributed conventional points and building corner edge feature points. The effectiveness of the present application is verified by comparing the matching optimization results of the least square matching (LSM) based on image gray scale and the present application. The detailed information of the experimental data is shown in Table 1:

[0063] Table 1: Basic parameters of experimental images and initial data

[0064]

[0065] Evaluation index: The matching optimization results are evaluated from three aspects of matching optimization success rate, effective matching optimization probability and correlation coefficient between image windows.

[0066] (1) Matching optimization success rate: that is, the percentage of three-dimensional points in the total number of points after the matching image points in the optimization process are located within the image range of the reference image and the search image.

[0067] (2) Effective matching optimization probability: that is, the percentage of three-dimensional points in the total number of points within the reasonable range of the digital surface model (DSM) of the final ground point three-dimensional coordinates obtained by the forward intersection.

[0068] (3) Average normalized correlation coefficient (NCC) between image windows: the correlation coefficient between image windows is an important evaluation index for evaluating whether two image windows are related. The higher the correlation coefficient between image windows, the higher the probability that the image center point is a homonym.

[0069] According to the evaluation index, the experimental data of the simulation experiment is shown in Tables 2-5:

[0070] Table 2: Image matching optimization results of Northwest University (58014 points)

[0071]

[0072] Table 3: Image matching optimization results of Yangjiang City (165353 points)

[0073]

[0074] Table 4: Northwestern University edge point cloud matching optimization results (105619 points)

[0075]

[0076] Table 5: Yangjiang City image edge point cloud optimization results (147555 points)

[0077]

[0078] From the experimental results of Tables 2-5, it can be seen that, since the depth information of the image is mined by using the convolutional neural network to enrich the information amount of the image pixels, the matching optimization results of the present application are superior to the LSM method in terms of the probability of successful matching and effective matching optimization and the correlation coefficient between the matching windows, especially in the improvement of edge point cloud matching optimization, because the information amount contained in the edge point cloud is more abundant than that of the ordinary point cloud, and the convolutional neural network can more easily eliminate matching interference terms to obtain better matching results. In theory, each three-dimensional point in the reconstructed three-dimensional point cloud should be located on the DSM surface of the object, so the discrete points in the point cloud data represent very low accuracy and are very likely to be mismatched points. As can be seen from the enlarged point cloud details in the experimental comparison graph, the matching optimization point cloud of the present application has fewer discrete points and more abundant details. These results verify that the present application is superior to the LSM algorithm in terms of matching accuracy and point cloud integrity.

[0079] Compared with the traditional least squares matching method, the method of the present application has obvious advantages, both in higher matching optimization success rate and in higher effective matching rate, and is a feasible image matching optimization method with fewer abnormal matches and more abundant reconstructed point cloud texture details.

[0080] It should be understood that the above description of the preferred embodiments is detailed and should not be considered as limiting the scope of patent protection of the present application. Those skilled in the art can make substitutions or modifications without departing from the scope of the present application, and all such substitutions or modifications fall within the scope of protection of the present application. The scope of protection of the present application should be subject to the appended claims.

Claims

1. A method for image matching optimization based on geometric constraints and convolutional neural network, characterized in that, The method comprises the following steps: Step 1: project the initial matching image three-dimensional points to the reference image and the search image respectively to obtain initial matching image pair image point coordinates p r (x r ,y r ), p s (x s ,y s ) , determine the reference image window and the matching search range with the initial matching image pair as the center; Step 2: Taking the initial matching image window as a reference, an optimal matching point is searched in a matching search range by using a matching optimization convolutional neural network; The matching optimization convolutional neural network comprises a convolutional image information extraction module and a full-connection optimal matching search module; The convolutional image information extraction module comprises 16 convolutional layers arranged in series, and a residual block is added to each of the first to 15th convolutional layers; a normalization layer, an activation layer and a residual block are sequentially added after the 16th convolutional layer; The full-connection optimal matching search module comprises a window feature vector descriptor submodule and an optimal matching search submodule; the feature vector descriptor submodule is composed of a full-connection layer with a size of 48x1000, and the output result is a 1000-dimensional feature vector of the matching image window; the optimal matching search submodule is composed of a full-connection layer with a size of 1000x64, and the output result is a similarity measure of the reference image and the search image window. Step 3: The optimal matching image pair position coordinates are brought into a forward intersection error equation, and the accurate three-dimensional coordinates of the corresponding photographed object points are calculated. 2.The method of claim 1, wherein: In step 1, the initial matching image pair point coordinates are obtained by using the following formula: Among them, (x (r / s) ,y (r / s) ) represents the projection of a 3D point P(X,Y,Z) onto the reference image I. r Or search for images I s The obtained image point p r or p s in I r Or I s The position coordinates in the image are (x0, y0), where (x0, y0) represents the position coordinates of the principal point of the camera in the image, and f represents the principal distance of the camera. s(r / s) ,Y s(r / s) Z s(r / s) ) indicates the shooting reference image I r Or search for images I s The coordinates of the camera's optical center in a three-dimensional coordinate system are captured at any time. Indicates shooting reference image I r Or search for images I s The rotation matrix of the camera during shooting. 3.The method of claim 1, wherein: In step 1, an image window with n x n pixels is selected as a to-be-matched image window in the reference image with the projection image point as the center; and an image window with m x m pixels is selected as a matching search range in the search image with the projection image point as the center; wherein m and n are preset values, and m > n. 4.The method of claim 1, wherein: In step 2, (m-n) x (m-n) image windows are cut from left to right and from top to bottom in the search range and the reference image window, and the optimal matching is searched by using the convolutional neural network.

5. The method of claim 1, wherein: In step 3, the three-dimensional coordinates of the object points corresponding to the matching image pair are obtained by using the following formula: Where P(X, Y, Z) is the three-dimensional coordinate of the sought ground object point, and li-i6are the coefficients of the equation; li x and l y are the constant terms of the equation; f represents the focal length of the camera, (x0, y0) represents the position coordinates of the principal point of the camera in the image, p(x, y) represents the optimal matching image point coordinates searched in step 2, and (X s , Y s , Z s ) represents the coordinates of the optical center of the camera in the three-dimensional space, and R represents the rotation matrix of the camera.

6. A geometric constraint and convolutional neural network based image matching optimization system, comprising: One or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the image matching optimization method based on geometric constraints and convolutional neural networks according to any one of claims 1 to 5. ​

Images