Image target matching method and device based on local homography

Abstract

The application provides an image target matching method and device based on local homography, and relates to the field of image processing; the method adopts a registration algorithm based on image local features to pre-acquire natural feature point pairs between background image pairs; secondly, the source background image is evenly divided into multiple rectangular grids; a moving direct linear transformation algorithm is used to calculate the local homography matrix of each rectangular grid; and the mapping point of the image target to be matched in the source image in the target image is calculated by using the local homography matrix; and the image target uniquely existing in the mapping area or the image target closest to the mapping point in the Euclidean distance is taken as the matching point. The application uses the local homography matrix to weaken the mapping error introduced by the parallax change, improve the image target matching precision, reduce the mis-matching probability of the cluster target with similar features and dense distribution in a complex environment, and significantly improve the image target matching speed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image target matching method and device based on local homography. BACKGROUND

[0002] In the field of computer stereo vision geometry measurement, such as target positioning, three-dimensional reconstruction, deformation measurement, etc., image target matching is an indispensable step. Image target matching is a process of using existing target image information to extract target features, matching with feature information of a to-be-matched image, and estimating the target position.

[0003] At present, image target matching methods include a template matching-based method, an image registration-based matching method, and a deep learning-based matching method.

[0004] The template matching-based method includes a gray template-based matching method, a feature template-based matching method, and an active wavelet basis-based matching method. The gray template-based matching algorithm does not have invariance to rotation changes, scale changes, affine changes, etc. between images, and it is difficult to have high matching accuracy in the case of severe changes in illumination. The feature template-based matching method generally uses a histogram of oriented gradients feature HOG and a SIFT feature, and uses a support vector machine SVM and an AdaBoost classification algorithm to distinguish targets and determine positions. The above classification algorithms have high detection accuracy but slow execution speed. The active wavelet basis-based matching method encodes the target contour of a target image using an active wavelet basis, and determines the target position of a to-be-matched image using a sum-max mapping method in a sliding window matching process. This method is suitable for application scenarios where the target contour is obvious and the rotation and scale change is small.

[0005] The image registration-based matching method calculates the geometric transformation between images using local feature points of a source image and a target image, so as to estimate the position of the target in the to-be-matched image. The geometric transformation between images includes affine transformation and homography transformation, and the homography transformation includes global homography transformation and local homography transformation. A moving direct linear transformation algorithm is proposed in a proximity projection transformation algorithm in the prior art. The image is divided into a rectangular grid image, and a smoothing extrapolation method is used to obtain a local homography transformation matrix to describe the mapping relationship of the overlapping area between different scene images. The image registration-based matching method makes up for the defects of the template-based matching method that does not have rotation change and scale change invariance. Using the known geometric transformation between images, online matching of a single or multiple targets can be realized.

[0006] The target matching method based on deep learning firstly trains a large number of target image samples, and uses the trained network model to identify and locate the target of the image to be matched. The matching method based on deep learning has good effect on the identification and location of the target in the scene image, but the complex training model needs a rich image training set, and cannot solve the matching of multiple targets of the same type.

[0007] The existing target matching algorithm sacrifices time to improve the matching accuracy, and has poor effect on the matching of the cluster image target with similar features. Therefore, it is necessary to propose a new image target matching method based on image registration to estimate the mapping position of the target according to the geometric constraint between the scene images, and improve the matching accuracy and speed of the cluster target. SUMMARY

[0008] The present application aims to provide an image target matching method and device based on local homography, to solve the defects of the existing image target matching algorithm, such as poor robustness when there are significant rotation changes, scale changes and illumination changes between images, and to avoid the problem of easy mismatching of cluster targets with similar features and dense distribution in complex scenes.

[0009] To achieve the above purpose, the present application proposes the following technical solutions:

[0010] In a first aspect, an image target matching method based on local homography is proposed, comprising:

[0011] determining a background image pair, and obtaining natural feature point pairs between the background image pair; wherein the background image pair is composed of a source background image and a target background image;

[0012] dividing the source background image into C1xC2 rectangular grids uniformly, and calculating the local homography matrix corresponding to each rectangular grid in combination with the natural feature point pairs; wherein C1 and C2 are the number of divisions of the source background image in the height direction and the width direction respectively, and C1≥10, C2≥10;

[0013] using an image target detection method to identify the image targets in the source image and the target image, obtaining the sub-pixel centroid coordinates of each image target, determining the corresponding rectangular grid of the image target to be matched in the source background image and the local homography matrix corresponding to the rectangular grid; wherein the source image and the target image are a pair of images to be matched with image targets;

[0014] calculating the mapping point of the image target to be matched in the target image according to the sub-pixel centroid coordinates of the image target in the source image and the local homography matrix of the corresponding rectangular grid;

[0015] determining a mapping region of the mapping point in the target image, judging a number of image targets in the mapping region; when there is only a single image target in the mapping region, determining that the image target is a matching point of the to-be-matched image target in the target image; when there are multiple image targets in the mapping region, determining that an image target with the minimum Euclidean distance from the mapping point is a matching point of the to-be-matched image target in the target image.

[0016] Further, the process of calculating the local homography matrix corresponding to each rectangular grid by combining the natural feature point pairs is to calculate the local homography matrix of each rectangular grid in the source background image by combining the natural feature point pairs using a moving direct linear transformation algorithm, including the following steps:

[0017] 2.1) The mapping relationship between the background image pairs is described by using global homography, and a matrix equation for solving a global homography matrix is established according to the natural feature point pairs, specifically as follows:

[0018] s[x′y′ 1] T =H g [x y 1] T Formula (1);

[0019]

[0020] Wherein, s is a scale factor; H g is a global homography matrix; {b(x,y),b′(x′,y′)} is a pair of natural feature point pairs between the background image pairs; h=[h 11 ,h 12 ,h 13 ,h 21 ,h 22 ,h 23 ,h 31 ,h 32 ,1] T ; h ij (i,j=1,2,3) is the i-th row and j-th column element of the global homography matrix H g ;

[0021] 2.2) The global homography matrix H g between the background image pairs is calculated by using a direct linear transformation method, and the calculation formula is as follows:

[0022]

[0023] Wherein, n is the number of natural feature point pairs; a i ∈R 2×9 , is the i-th group of natural feature point pairs {x i ,x iThe coefficients in the two rows of the matrix equation of the global homography matrix. This is the optimal solution for vector h;

[0024] 2.3) Based on natural feature point pairs and the global homography matrix H g The local homography matrix H of the center points of each rectangular grid in the source background image is calculated using a weighted method. * The calculation formula is as follows:

[0025]

[0026] Among them, h * The local homography matrix H * A column vector composed of each element; weights The formula for calculating it varies depending on the x-coordinate of the image:

[0027] w * i =exp(-||x * -x i || 2 / σ 2 ) formula (5);

[0028] In the formula, σ is the proportionality parameter; x * x i These are the pixel coordinates of the center point of any rectangular grid and the i-th natural feature point on the source image, respectively.

[0029] Furthermore, in step 2.3), a weighted method is used to calculate the local homography matrix H of the center points of each rectangular grid in the source background image. * The calculation formula is described in matrix form as follows:

[0030]

[0031] W * =diag([w * 1 w * 1 … w * n w * n ]) Formula (7);

[0032] Where A is Vertical stacking; weight W * ∈R 2n×2n It is composed of a diagonal matrix.

[0033] Furthermore, when the center point x of the rectangular grid * With natural feature point xi When the distance exceeds a preset distance threshold, a threshold γ, 0≤γ≤1 is used to offset the weight, and the weight calculation is adjusted as follows:

[0034] w * i = max(exp(-||x * -x i || 2 / σ 2 ), γ) Formula (8).

[0035] Further, the process of obtaining the sub-pixel centroid coordinates of each image target includes:

[0036] The frame difference method or the background subtraction method based on sequence images is used to identify the image targets in the source image and the target image, and the region of interest of each image target is obtained;

[0037] For the region of interest of any image target, the connected domain of the image target is obtained by using the morphological processing method, and the sub-pixel centroid coordinates of the image target are calculated by using the average value method, so as to determine the position of the image target in the source image and the target image;

[0038] The formula for calculating the sub-pixel centroid coordinates of the image target is as follows:

[0039]

[0040] Where k is the total number of contour pixel points in the connected domain of the image target, (u i ,v i ) is the image coordinates of the i-th contour pixel point, and (x c ,y c ) is the sub-pixel centroid coordinates of the image target.

[0041] Further, the process of calculating the mapping point of the image target to be matched in the target image includes:

[0042] According to the sub-pixel centroid coordinates of the image target to be matched, the corresponding rectangular grid of the image target to be matched in the source background image is found, and the row and column number of the rectangular grid in the rectangular network is determined;

[0043] According to the row and column number of the rectangular grid in the rectangular network, the local homography matrix corresponding to the rectangular grid is obtained;

[0044] According to the local homography, the mapping point of the image target to be matched in the target image is calculated;

[0045] The formula for calculating the row and column number of the image target to be matched in the rectangular network is as follows:

[0046]

[0047] The calculation formula of the mapping point (x', y') is:

[0048]

[0049] Wherein, row, col are the row number and column number of the image target to be matched in the rectangular network, (height, width) is the pixel size of the source image, floor represents the truncation and rounding, is the corresponding local homography matrix H of the image target to be matched in the source image * the i-th row vector.

[0050] In a second aspect, an image target matching device based on local homography is provided, comprising:

[0051] A determination and acquisition module is configured to determine a pair of background images, and acquire a pair of natural feature points between the pair of background images; wherein the pair of background images is composed of a source background image and a target background image;

[0052] A division and calculation module is configured to divide the source background image into C1xC2 rectangular grids uniformly, and calculate a local homography matrix corresponding to each rectangular grid in combination with the pair of natural feature points; wherein C1 and C2 are the number of parts in the height direction and the width direction of the source background image divided uniformly, and C1≥10 and C2≥10;

[0053] An identification and acquisition module is configured to identify image targets in a source image and a target image by using an image target detection method, acquire sub-pixel centroid coordinates of each image target, and determine a rectangular grid corresponding to an image target to be matched in the source background image and a local homography matrix corresponding to the rectangular grid; wherein the source image and the target image are a pair of images to be matched with image targets;

[0054] A calculation module is configured to calculate a mapping point of the image target to be matched in the target image according to the sub-pixel centroid coordinates of the image target in the source image and the local homography matrix of the rectangular grid corresponding to the image target;

[0055] A determination and judgment module is configured to determine a mapping area of the mapping point in the target image, judge the number of image targets in the mapping area, determine that an image target is a matching point of the image target to be matched in the target image when there is only a single image target in the mapping area, and determine that an image target with the smallest Euclidean distance from the mapping point is the matching point of the image target to be matched in the target image when there are multiple image targets in the mapping area.

[0056] Further, the process of calculating the local homography matrix corresponding to each rectangular grid by the division calculation module in combination with the natural feature point pairs comprises the following steps:

[0057] The mapping relationship between the background image pair is described by using global homography, and a matrix equation for solving the global homography matrix is established according to the natural feature point pairs, specifically as follows:

[0058] s[x'y' 1] T = H g [x y 1] T Equation (1);

[0059]

[0060] wherein s is a scale factor; H g is a global homography matrix; {b(x,y),b'(x',y')} is a pair of natural feature points between the background image pair; h = [h 11 ,h 12 ,h 13 ,h 21 ,h 22 ,h 23 ,h 31 ,h 32 ,1] T ; h ij (i,j = 1,2,3) is the element of the i-th row and j-th column of the global homography matrix H g ;

[0061] The global homography matrix H g between the background image pair is calculated by using direct linear transformation method, and the calculation formula is as follows:

[0062]

[0063] wherein n is the number of natural feature points; a i ∈ R 2×9 is the two-row coefficient of the i-th group of natural feature point pairs {x i ,x i '} in the matrix equation of the global homography matrix, is the optimal solution of the vector h;

[0064] According to the natural feature point pairs and the global homography matrix H g , the local homography matrix H * of the center point of each rectangular grid in the source background image is calculated by using a weighted method, and the calculation formula is as follows:

[0065]

[0066] wherein h* H is a local homography matrix * a column vector composed of each element in H; weight According to the different image coordinates x, the calculation formula is as follows:

[0067] w * i = exp (-||x * -x i || 2 / σ 2 ) formula (5);

[0068] In the formula, σ is a proportional parameter; x * , x i are pixel coordinates of any rectangular grid center point and the i-th natural feature point on the source image respectively.

[0069] Further, the computing module calculates the mapping point of the image target to be matched in the target image according to the sub-pixel centroid coordinates of the image target in the source image and the local homography matrix of the corresponding rectangular grid.

[0070] According to the sub-pixel centroid coordinates of the image target to be matched, the corresponding rectangular grid of the image target to be matched in the source background image is found, and the row and column number of the rectangular grid in the rectangular network is determined.

[0071] According to the row and column number of the rectangular grid in the rectangular network, the local homography matrix corresponding to the rectangular grid is obtained.

[0072] According to the local homography, the mapping point of the image target to be matched in the target image is calculated.

[0073] The calculation formula of the row and column number of the image target to be matched in the rectangular network is:

[0074]

[0075] The calculation formula of the mapping point (x', y') is:

[0076]

[0077] Wherein, row and col are the row and column number of the image target to be matched in the rectangular network, (height, width) is the pixel size of the source image, floor represents the truncation and rounding, is the i-th row vector of the local homography matrix H* corresponding to the image target to be matched in the source image.

[0078] In a third aspect, an electronic device is provided, including a computer program stored in a computer readable storage medium; when a processor of the electronic device reads the computer program from the computer readable storage medium, the processor executes the computer program, so that the electronic device executes the steps of the image target matching method based on local homography described above.

[0079] From the above technical solutions, the technical solutions of the present application have the following beneficial effects:

[0080] The image target matching method and device based on local homography disclosed by the present application have the following beneficial effects:

[0081] The present application divides the source background image into multiple rectangular grids uniformly, and calculates the local homography matrix by using the natural feature point pairs between the background image pairs, and calculates the mapping position of the image target in the target image; compared with the global homography matrix, the local homography matrix weakens the mapping error introduced by the parallax change, improves the target matching accuracy, and reduces the mis-matching probability of the cluster target with similar features and dense distribution in the complex environment; compared with the prior art, the speed of target matching is also greatly improved.

[0082] It should be understood that all combinations of the aforementioned concepts and additional concepts described in greater detail below can be seen as part of the subject matter of the present disclosure as long as such concepts are not mutually contradictory in terms of their meaning.

[0083] The foregoing and other aspects, embodiments and features of the present teachings can be more fully understood from the following description taken in conjunction with the accompanying drawings. Other aspects, embodiments and features of the present teachings will be apparent from the description that follows, and from the claims. BRIEF DESCRIPTION OF DRAWINGS

[0084] The accompanying drawings are not necessarily drawn to scale. In the drawings, each identical, or nearly identical, component that is illustrated in various figures is represented with a like numeral. For purposes of clarity, not every component is called out in every drawing. There is shown in the drawings, by way of illustration, various embodiments in which the principles of the present application can be implemented. The embodiments are described with reference to the accompanying drawings in which:

[0085] Figure 1 Flow chart of the image target matching method based on local homography of the present application;

[0086] Figure 2 Schematic diagram of the image target matching using local homography matrix of the present application;

[0087] Fig. 3(a) is a distribution diagram of 8 image target positions in the source image of the embodiment;

[0088] Fig. 3(b) is a distribution diagram of 8 image target positions in the target image of the embodiment;

[0089] Figure 4 Schematic diagram of the natural feature point pairs between the background image pairs of the embodiment;

[0090] Figure 5 Schematic diagram of the source background image being divided into uniform rectangular grid of the embodiment;

[0091] Figure 6 Schematic diagram of the 8 image target mapping points and the real matching points in the target image of the embodiment;

[0092] Figure 7 Block diagram of the image target matching device based on local homography of the embodiment;

[0093] Figure 8 Block diagram of the electronic device of the embodiment. DETAILED DESCRIPTION

[0094] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the embodiments of the present application will be described clearly and completely below with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the described embodiments of the present application, all other embodiments obtained by a person of ordinary skill in the art without any creative effort belong to the protection scope of the present application. Unless otherwise defined, the technical terms or scientific terms used herein should be understood as the common meanings by those skilled in the art.

[0095] The terms "first", "second", and similar terms used in the specification and claims of the present patent application do not denote any order, quantity, or importance, but are used to distinguish different components. Also, the singular forms "a", "an", and "the" do not denote the quantity limitation, but denote the existence of at least one, unless the context clearly indicates otherwise. The terms "comprise", "include", and similar terms mean that the elements or objects before "comprise" or "include" encompass the features, integers, steps, operations, elements, and / or components listed after "comprise" or "include", and do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0096] Based on the characteristics of the distribution density of the target existing in the cluster image, the feature similarity, and the like, when the target matching algorithm pushed down, such as the template matching-based method, the image registration-based matching method, and the deep learning-based matching method, is applied to the cluster image target matching, the matching accuracy is improved at the cost of time, and the matching effect of the cluster image target with similar features is poor. Therefore, the present application aims to provide an image target matching method and device based on local homography, which can effectively improve the target matching accuracy and the matching speed for the cluster image target with similar features.

[0097] Specifically, as shown in Figure 1 The image target matching method based on local homography provided by the present application specifically comprises the following steps:

[0098] 1) determining a background image pair, and obtaining natural feature point pairs between the background image pair; wherein the background image pair is composed of a source background image and a target background image;

[0099] In specific implementation, the registration algorithm based on image local features (VL-SIFT) is used to extract the natural feature point pairs between the background image pair, and the random sample consensus algorithm (RANSAC) is further used for purification.

[0100] 2) dividing the source background image into C1xC2 rectangular grids uniformly, and calculating the local homography matrix corresponding to each rectangular grid in combination with the natural feature point pairs; wherein C1 and C2 are the number of parts in the height direction and the width direction of the source background image divided uniformly, and C1≥10 and C2≥10;

[0101] In the embodiment, the moving direct linear transformation algorithm (Moving DLT) is used in combination with the natural feature point pairs to calculate the local homography matrix of each rectangular grid in the source background image; specifically comprising the following steps:

[0102] 2.1) The mapping relationship between the background image pairs is described by global homography, as shown in formula (1); the matrix equation for solving the global homography matrix is established according to the natural feature point pairs, as shown in formula (2), specifically:

[0103] s[x′y′ 1] T =H g [x y 1] T Formula (1);

[0104]

[0105] Wherein, s is a scale factor; H g is a global homography matrix; {b(x,y),b′(x′,y′)} is a pair of natural feature points between the background image pairs; h=[h 11 ,h 12 ,h 13 ,h 21 ,h 22 ,h 23 ,h 31 ,h 32 ,1] T ; h ij (i,j=1,2,3) is the i-th row and j-th column element of the global homography matrix H g ;

[0106] 2.2) The global homography matrix H g between the background image pairs is calculated by using the direct linear transformation method (DLT), and the calculation formula is as follows:

[0107]

[0108] Wherein, n is the number of natural feature points; a i ∈R 2×9 , is the two-row coefficient of the i-th set of natural feature point pairs {x i ,x i '} in the matrix equation of the global homography matrix, is the optimal solution of the vector h;

[0109] 2.3) According to the natural feature point pairs and the global homography matrix H g , the local homography matrix H* of each rectangular grid center point in the source background image is calculated by using the weighted method, and the calculation formula is as follows:

[0110]

[0111] Wherein, h* is a column vector composed of each element in the local homography matrix H*; the weight changes according to the different image coordinates x, and the calculation formula is as follows:

[0112] w * i = exp(-||x * -x i || 2 / σ 2 ) Formula (5);

[0113] where σ is a scale parameter; x* and x i are pixel coordinates of any rectangular grid center and the i-th natural feature point on the source image, respectively.

[0114] Optionally, the formula (4) for calculating the local homography H* of each rectangular grid center between the background image pairs can be described in matrix form, i.e.,

[0115]

[0116] W * = diag([w * 1 w * 1 … w * n w * n ]) Formula (7);

[0117] where A is the vertical stack of ; the weight W * ∈ R 2n×2n , and consists of a diagonal matrix; and formula (6) is a weighted singular value decomposition problem, and the solution is the right singular vector of W * A. In addition, in order to improve the calculation efficiency, the local homography matrix of any rectangular grid center obtained by formula (6) is used as the local homography matrix of all target points in the rectangular grid.

[0118] For formula (6), it is not applicable in some scenarios, such as when the distance between the rectangular grid center x * and the natural feature point x i exceeds the preset distance threshold, i.e., the distance is far, at this time the value of the weight has little significance, and the solution of formula (6) is unstable; at this time, in order to prevent numerical problems in the calculation of the local homography matrix, a threshold γ, 0≤γ≤1 is used to offset the weight, and the weight calculation formula is adjusted to:

[0119] w * i = max(exp(-||x * -x i || 2 / σ 2), γ) Formula (8).

[0120] 3) identifying image targets in the source image and the target image by using an image target detection method, obtaining sub-pixel centroid coordinates of each image target, and determining a rectangular grid corresponding to a to-be-matched image target in the source background image and a local homography matrix corresponding to the rectangular grid; wherein the source image and the target image are a pair of to-be-matched images in which image targets are photographed, the background of the source image is the source background image, and the background of the target image is the target background image;

[0121] The process of obtaining the sub-pixel centroid coordinates of each image target includes: identifying the image targets in the source image and the target image by using a frame difference method or a background subtraction method based on sequence images, and obtaining a region of interest of each image target; for the region of interest of any image target, a connected domain of the image target is obtained by using a morphological processing method, and a sub-pixel centroid coordinate of the image target is calculated by using an average value method, so as to determine the position of the image target in the source image and the target image; wherein the formula for calculating the sub-pixel centroid coordinate of the image target is as follows:

[0122]

[0123] wherein k is the total number of contour pixel points in the connected domain of the image target, (u i ,v i ) is the image coordinate of the i th contour pixel point, (x c ,y c ) is the sub-pixel centroid coordinate of the image target.

[0124] 4) calculating a mapping point of the to-be-matched image target in the target image according to the sub-pixel centroid coordinate of the image target in the source image and the local homography matrix corresponding to the rectangular grid;

[0125] In a specific implementation, the process of calculating the mapping point of the to-be-matched image target in the target image includes:

[0126] According to the sub-pixel centroid coordinate of the to-be-matched image target, the corresponding rectangular grid of the to-be-matched image target in the source background image is searched, and the number of rows and columns of the rectangular grid in the rectangular network is determined;

[0127] According to the number of rows and columns of the rectangular grid in the rectangular network, the local homography matrix corresponding to the rectangular grid is obtained;

[0128] According to the local homography, the mapping point of the to-be-matched image target in the target image is calculated;

[0129] The formula for calculating the number of rows and columns of the to-be-matched image target in the rectangular network is:

[0130]

[0131] The calculation formula of the mapping point (x', y') is:

[0132]

[0133] Wherein, row and col are the row number and column number of the image target to be matched in the rectangular network, (height, width) is the pixel size of the source image, floor represents the truncation and rounding, H is the corresponding local homography matrix of the image target to be matched in the source image * The i-th row vector of the matrix.

[0134] 5) Determine the mapping area of the mapping point in the target image, judge the number of image targets in the mapping area; when there is only a single image target in the mapping area, determine that the image target is the matching point of the image target to be matched in the target image; when there are multiple image targets in the mapping area, determine that the image target with the smallest Euclidean distance from the mapping point is the matching point of the image target to be matched in the target image.

[0135] In combination with the image target matching process shown in Figure 2 , c1 and c2 in the figure are the first and second cameras respectively, the first camera obtains a source image by shooting, the second camera obtains a target image by shooting, l and l' are epipolar lines, e and e' are epipoles, b1 is the projection point of target 1 on the source image, b1' and b2' are the projection points of target 1 and target 2 on the target image respectively; after determining the position of the image target to be matched b1 in the source image, the rectangular grid where the image target to be matched is located is searched, and the mapping point m of the image target in the target image is estimated according to the position of the image target in the source image and the local homography matrix of the matrix grid where the image target is located, the mapping area with the mapping point m as the center and the radius r is searched as the target, wherein the radius r is 0.75 times the diagonal length of the rectangular grid; there are two image targets b1' and b2' in the mapping area, and the image target b1' with the closest Euclidean distance from the mapping point is selected as the matching point of target 1 in the target image.

[0136] In the specific implementation process of the image target matching method of the application, steps 1)-2) are the pre-off-line operation process before image target matching, and steps 3)-5) are the on-line operation process during image target matching; the time-consuming local homography matrix calculation is an off-line operation before image target matching, and the time consumption of image target matching mainly includes target recognition, target mapping point estimation and nearest Euclidean distance search, which effectively improves the image target matching speed.

[0137] The image target matching method based on local homography is further specifically introduced below in combination with the embodiments shown in the drawings.

[0138] As shown in the drawings, Figures 3(a) to 3(b) The 8 image target points in the source image and the target image are matched by the above steps, and the positions of the 8 image target points in the source image and the target image in the embodiment are determined by placing markers, and it is assumed that the sub-pixel center coordinates of each image target are obtained by using the average value method after image target identification; Fig. 3(a) and Fig. 3(b) are obtained by a pair of binocular cameras with a baseline distance of 30 m and a measurement distance of about 100 m, and the sizes of the source image and the target image are both 1920 pixels x 1200 pixels. Figure 4 As shown in the drawings, Figure 5 The moving direct linear transformation algorithm (DLT) is used to estimate the mapping points of the 8 image target points in the source image in the target image, and the Euclidean distance of the mapping points from the real matching points is taken as the measurement basis of the image target matching precision.

[0139] As shown in the drawings, Figure 6 The data shows that the mapping points of the image targets in the target image calculated by the method of the present application are close to the real matching points, and the root mean square error (RMSE) of the Euclidean distance of the mapping points from the matching points is not more than 12.9 pixels when the image size is 1920 pixels x 1200 pixels.

[0140] It is particularly pointed out that the source background image is uniformly divided into multiple rectangular grids in the present application, and the method of calculating the local homography matrix of the rectangular grid by using the moving linear transformation algorithm is suitable for application scenarios where the natural feature points of the background image are very rich. For scenarios where the natural feature points of the background image are sparse, the target mapping position estimated by using the local homography matrix is close to the mapping position estimated by using the global homography matrix, and the image target matching precision is reduced. For application scenarios where the natural feature points of the background image are rich, appropriately increasing the number of uniformly divided rectangular grids helps to improve the target matching precision.

[0141] In the embodiments of the present application, an electronic device is also provided, which includes a computer program stored in a computer readable storage medium; when a processor of the electronic device reads the computer program from the computer readable storage medium, the processor executes the computer program, so that the electronic device executes the steps of the image target matching method based on local homography described above. Taking an electronic device running on a computer as an example, Figure 8As shown, the electronic device can include one or more (only one is shown in the figure) processors (the processor can include but not limited to a microprocessor MCU or a programmable logic device FPGA processing device), a memory for storing data, and a transmission device for communication function. Those skilled in the art can understand that, Figure 8 The structure shown is only schematic, which does not limit the structure of the above-mentioned electronic device.

[0142] The above program can run in the processor, or can also be stored in the memory, that is, the computer readable medium, the computer readable medium includes permanent and non-permanent, removable and non-removable media can be realized by any method or technology to store information. Information can be computer readable instructions, data structure, program module or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tape, magnetic tape magnetic disk storage or other magnetic storage device or any other non-transmission medium that can be used to store information that can be accessed by a computing device. According to the definition in this paper, computer readable medium does not include temporary computer readable medium, such as modulated data signal and carrier wave. These computer programs can also be loaded into a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate computer implemented processing, so that the instructions executed on the computer or other programmable device provide steps for implementing the processes Figure 1 The function specified in one flow or multiple flows and / or blocks Figure 1 The steps of the function specified in one block or multiple blocks correspond to different method steps, which can be realized by different modules.

[0143] In this embodiment, a device or system is provided, which can be called a local homography based image target matching device, such as Figure 7As shown, comprising: determining acquisition module, for determining background image pair, acquiring the natural feature point pair between the background image pair; wherein the background image pair is composed of source background image and target background image; division calculation module, for dividing the source background image into C1xC2 rectangular grid uniformly, combining the natural feature point pair to calculate the local homography matrix corresponding to each rectangular grid; wherein C1, C2 are the number of the source background image divided in the height direction and the width direction respectively, and C1≥10, C2≥10; identification acquisition module, for identifying image target in source image and target image by using image target detection method, acquiring sub-pixel centroid coordinates of each image target, determining the rectangular grid corresponding to the image target to be matched in the source background image and the local homography matrix corresponding to the rectangular grid; wherein the source image and target image are respectively the image pair to be matched with image target; calculation module, for calculating the mapping point of the image target to be matched in the target image according to the sub-pixel centroid coordinates of the image target in the source image and the local homography matrix of the rectangular grid corresponding thereto; determination judgment module, for determining the mapping area of the mapping point in the target image, judging the number of image targets in the mapping area; when there is only a single image target in the mapping area, determining that the image target is the matching point of the image target to be matched in the target image; when there are multiple image targets in the mapping area, determining that the image target with the minimum Euclidean distance from the mapping point is the matching point of the image target to be matched in the target image.

[0144] The system is used to realize the steps of the image target matching method based on local homography disclosed in the above embodiments, which have been described and will not be repeated here.

[0145] For example, the process of the division calculation module combining the natural feature point pair to calculate the local homography matrix corresponding to each rectangular grid includes the following steps:

[0146] The global homography is used to describe the mapping relationship between the background image pair, and the matrix equation for solving the global homography matrix is established according to the natural feature point pair, specifically:

[0147] s[x′y′ 1] T =H g [x y 1] T Formula (1);

[0148]

[0149] Wherein, s is a scale factor; H g is a global homography matrix; {b(x,y),b′(x′,y′)} is a pair of natural feature points between the background image pair; h=[h 11 ,h12 ,h 13 ,h 21 ,h 22 ,h 23 ,h 31 ,h 32 ,1] T ;h ij (i,j = 1,2,3) is the element in the i-th row and j-th column of the global homography matrix H g ;

[0150] The global homography matrix H g between the background images is calculated by using the direct linear transformation method, and the calculation formula is as follows:

[0151]

[0152] where n is the number of natural feature point pairs; a i ∈R 2×9 is the two-row coefficient of the i-th group of natural feature point pairs {x i ,x i '} in the matrix equation of the global homography matrix, is the optimal solution of the vector h

[0153] According to the natural feature point pairs and the global homography matrix H g , the local homography matrix H * of each rectangular grid center point in the source background image is calculated by using the weighted method, and the calculation formula is as follows:

[0154]

[0155] where h * is a column vector composed of each element in the local homography matrix H * ; the weight changes according to the different image coordinates x, and the calculation formula is as follows:

[0156] w * i = exp(-||x * -x i || 2 / σ 2 ) formula (5);

[0157] In the formula, σ is a proportional parameter; x * and x i are the pixel coordinates of any rectangular grid center point and the i-th natural feature point on the source image, respectively.

[0158] where the local homography matrix H *The calculation formula is described in matrix form as:

[0159]

[0160] W * = diag([w * 1 w * 1 … w * n w * n ]) Formula (7);

[0161] wherein A is the vertical superposition of * ∈R 2n×2n , and consists of a diagonal matrix.

[0162] When the distance between the rectangular grid center point x * and the natural feature point x i exceeds a preset distance threshold, a threshold value γ, 0≤γ≤1 is used to offset the weight, and the weight calculation is adjusted as:

[0163] w * i = max(exp(-||x * -x i || 2 / σ 2 ), γ) Formula (8).

[0164] For another example, the process in which the identification acquisition module acquires the sub-pixel centroid coordinates of each image target includes: using a frame difference method or a background subtraction method based on sequence images to identify image targets in the source image and the target image, and obtaining a region of interest of each image target; for the region of interest of any image target, using a morphological processing method to obtain a connected domain of the image target, and using an average value method to calculate the sub-pixel centroid coordinates of the image target, so as to determine the position of the image target in the source image and the target image; wherein the calculation formula of the sub-pixel centroid coordinates of the image target is as follows:

[0165]

[0166] wherein k is the total number of contour pixel points in the connected domain of the image target, (u i ,v i ) is the image coordinates of the i-th contour pixel point, and (x c ,y c ) is the sub-pixel centroid coordinates of the image target.

[0167] For example, the computing module calculates the process of mapping the to-be-matched image target in the target image, including: according to the sub-pixel centroid coordinates of the to-be-matched image target, searching for the corresponding rectangular grid of the to-be-matched image target in the source background image, and determining the row and column number of the rectangular grid in the rectangular network; according to the row and column number of the rectangular grid in the rectangular network, obtaining the local homography matrix corresponding to the rectangular grid; according to the local homography, calculating the mapping point of the to-be-matched image target in the target image; the row and column number of the to-be-matched image target in the rectangular network is calculated according to the following formula:

[0168]

[0169] The calculation formula of the mapping point (x', y') is as follows:

[0170]

[0171] Wherein, row and col are the row and column number of the to-be-matched image target in the rectangular network, (height, width) is the pixel size of the source image, floor represents the truncation and rounding, is the i-th row vector of the local homography matrix H corresponding to the to-be-matched image target in the source image. *

[0172] The image target matching method and device based on local homography disclosed by the application can solve the defects of poor robustness of the existing image target matching algorithm when there is a significant rotation change, a scale change and a light change between images, and can avoid the problem that cluster targets with similar features and dense distribution are prone to false matching in a complex scene. The local homography matrix is used to weaken the mapping error introduced by the parallax change, improve the image target matching accuracy, reduce the false matching probability of cluster targets with similar features and dense distribution in a complex environment, and significantly improve the image target matching speed.

[0173] Although the application has been disclosed as above with preferred embodiments, it is not intended to limit the application. Those skilled in the art can make various modifications and decorations without departing from the spirit and scope of the application. Therefore, the protection scope of the application shall be subject to the definition of the claims.​

Claims

1. An image target matching method based on local homography, characterized in that, include: Determine background image pairs and obtain natural feature point pairs between the background image pairs; wherein, the background image pair consists of a source background image and a target background image; Divide the source background image evenly into A rectangular grid is used, and the local homography matrix corresponding to each rectangular grid is calculated based on the natural feature point pairs; where... , These represent the number of parts into which the source background image is uniformly divided in the height and width directions, respectively. ≥10、 ≥10; wherein, when calculating the local homography matrix, the weights are determined based on the distance between the center point of the rectangular grid and the natural feature points; and when the distance between the center point of the rectangular grid and the natural feature points exceeds a preset distance threshold, the threshold is used to offset the weights; Image target detection methods are used to identify image targets in source and target images, obtain the sub-pixel centroid coordinates of each image target, and determine the rectangular grid corresponding to the image target to be matched in the source background image and the local homography matrix corresponding to the rectangular grid. The source and target images are pairs of images to be matched, each containing an image target. The process of obtaining the sub-pixel centroid coordinates of each image target includes: identifying image targets in the source and target images using a frame difference method or background subtraction based on sequence images, and obtaining the region of interest (ROI) for each image target; for any image target's ROI, obtaining the connected components of the image target using morphological processing methods, and calculating the sub-pixel centroid coordinates of the image target using an average value method, thereby determining the position of the image target in the source and target images; the formula for calculating the sub-pixel centroid coordinates of the image target is: , Where k is the total number of contour pixels in the connected components of the image target. Let i be the image coordinates of the i-th contour pixel. The sub-pixel centroid coordinates of the image target; Based on the subpixel centroid coordinates of the image target in the source image and the local homography matrix of its corresponding rectangular grid, calculate the mapping point of the image target to be matched in the target image. The mapping region of the mapping point in the target image is determined, and the number of image targets in the mapping region is judged. When there is only a single image target in the mapping region, the image target is determined to be the matching point of the image target to be matched in the target image. When there are multiple image targets in the mapping region, the image target with the smallest Euclidean distance to the mapping point is determined to be the matching point of the image target to be matched in the target image. The mapping region is a region with the mapping point as the center and a radius of r, where the radius r is 0.75 times the diagonal length of the rectangular grid.

2. The image target matching method based on local homography according to claim 1, characterized in that, The process of calculating the local homography matrix corresponding to each rectangular grid by combining the natural feature point pairs involves using the moving direct linear transformation algorithm to calculate the local homography matrix of each rectangular grid in the source background image by combining the natural feature point pairs, including the following steps: 2.1) Global homography is used to describe the mapping relationship between the background image pairs. A matrix equation for solving the global homography matrix is ​​established based on the natural feature point pairs, specifically: Official (1); Official (2); Where s is the scaling factor; This is the global homography matrix; A pair of natural feature points between background image pairs; ; Global homography matrix The element in the i-th row and j-th column; 2.2) Calculate the global homography matrix between the background image pairs using the direct linear transformation method. The calculation formula is as follows: Official (3); Where n is the number of natural feature point pairs; , for the i-th group of natural feature point pairs The coefficients in the two rows of the matrix equation of the global homography matrix. This is the optimal solution for vector h; 2.3) Based on natural feature point pairs and the global homography matrix The local homography matrix of the center points of each rectangular grid in the source background image is calculated using a weighted method. The calculation formula is as follows: Official (4); in, Local homography matrix A column vector composed of each element; weights The formula for calculating it varies depending on the x-coordinate of the image: Official (5); In the formula, It is a proportional parameter; , These are the pixel coordinates of the center point of any rectangular grid and the i-th natural feature point on the source image, respectively.

3. The image target matching method based on local homography according to claim 2, characterized in that, In step 2.3), a weighted method is used to calculate the local homography matrix of the center points of each rectangular grid in the source background image. The calculation formula is described in matrix form as follows: Official (6); Official (7); Where A is Vertical stacking; weights It is composed of a diagonal matrix.

4. The image target matching method based on local homography according to claim 2, characterized in that, When the center point of the rectangular grid With natural feature points When the distance exceeds a preset distance threshold, a threshold is used. The offsetting weights and weight calculations are adjusted as follows: Official (8).

5. The image target matching method based on local homography according to claim 1, characterized in that, The process of calculating the mapping point of the target image in the target image includes: Based on the subpixel centroid coordinates of the target image to be matched, find the rectangular grid corresponding to the target image in the source background image, and determine the number of rows and columns of the rectangular grid in the rectangular network; Based on the number of rows and columns of the rectangular grid in the rectangular network, the local homography matrix corresponding to the rectangular grid is obtained; Based on local homography, the mapping point of the target image in the target image is calculated; The formula for calculating the number of rows and columns of the target image in the rectangular network is as follows: , Official (10); The mapping point The calculation formula is: , Official (11); Where row and col are the number of rows and columns of the target image in the rectangular network, respectively. The pixel size is the source image size, and floor indicates truncation. The local homography matrix of the target image in the source image. The i-th row vector.

6. An image target matching device based on local homography, characterized in that, include: The determination and acquisition module is used to determine background image pairs and acquire natural feature point pairs between the background image pairs; wherein, the background image pair consists of a source background image and a target background image; The partitioning calculation module is used to uniformly divide the source background image into... A rectangular grid is used, and the local homography matrix corresponding to each rectangular grid is calculated based on the natural feature point pairs; where... , These represent the number of parts into which the source background image is uniformly divided in the height and width directions, respectively. ≥10、 ≥10; wherein, when calculating the local homography matrix, the weights are determined based on the distance between the center point of the rectangular grid and the natural feature points; and when the distance between the center point of the rectangular grid and the natural feature points exceeds a preset distance threshold, the threshold is used to offset the weights; The identification and acquisition module is used to identify image targets in a source image and a target image using an image target detection method, acquire the sub-pixel centroid coordinates of each image target, and determine the rectangular grid corresponding to the image target to be matched in the source background image and the local homography matrix corresponding to the rectangular grid; wherein, the source image and the target image are respectively a pair of images to be matched containing image targets; wherein, the process of acquiring the sub-pixel centroid coordinates of each image target includes: identifying image targets in the source image and the target image using a frame difference method or background subtraction method based on sequential images, and obtaining the region of interest of each image target; for any image target's region of interest, obtaining the connected components of the image target using a morphological processing method, and calculating the sub-pixel centroid coordinates of the image target using an average value method, thereby determining the position of the image target in the source image and the target image; wherein, the formula for calculating the sub-pixel centroid coordinates of the image target is: , Where k is the total number of contour pixels in the connected components of the image target. Let i be the image coordinates of the i-th contour pixel. The sub-pixel centroid coordinates of the image target; The calculation module is used to calculate the mapping point of the image target to be matched in the target image based on the sub-pixel centroid coordinates of the image target in the source image and the local homography matrix of its corresponding rectangular grid. The determination and judgment module is used to determine the mapping area of ​​the mapping point in the target image and to determine the number of image targets in the mapping area; when there is only a single image target in the mapping area, the image target is determined to be the matching point of the image target to be matched in the target image; when there are multiple image targets in the mapping area, the image target with the smallest Euclidean distance to the mapping point is determined to be the matching point of the image target to be matched in the target image; wherein, the mapping area is a region with the mapping point as the center and a radius of r, and the radius r is 0.75 times the diagonal length of the rectangular grid.

7. The image target matching device based on local homography according to claim 6, characterized in that, The process by which the partitioning calculation module calculates the local homography matrix corresponding to each rectangular grid in conjunction with the natural feature point pairs includes the following steps: Global homography is used to describe the mapping relationship between the background image pairs. A matrix equation for solving the global homography matrix is ​​established based on the natural feature point pairs, specifically: Official (1); Official (2); Where s is the scaling factor; This is the global homography matrix; A pair of natural feature points between background image pairs; ; Global homography matrix The element in the i-th row and j-th column; The global homography matrix between the background image pairs is calculated using the direct linear transformation method. The calculation formula is as follows: Official (3); Where n is the number of natural feature point pairs; , for the i-th group of natural feature point pairs The coefficients in the two rows of the matrix equation of the global homography matrix. This is the optimal solution for vector h; Based on natural feature point pairs and global homography matrix The local homography matrix of the center points of each rectangular grid in the source background image is calculated using a weighted method. The calculation formula is as follows: Official (4); in, Local homography matrix A column vector composed of each element; weights The formula for calculating it varies depending on the x-coordinate of the image: Official (5); In the formula, It is a proportional parameter; , These are the pixel coordinates of the center point of any rectangular grid and the i-th natural feature point on the source image, respectively.

8. The image target matching device based on local homography according to claim 6, characterized in that, The process by which the calculation module calculates the mapping point of the target image in the target image based on the sub-pixel centroid coordinates of the image target in the source image and the local homography matrix of its corresponding rectangular grid includes: Based on the subpixel centroid coordinates of the target image to be matched, find the rectangular grid corresponding to the target image in the source background image, and determine the number of rows and columns of the rectangular grid in the rectangular network; Based on the number of rows and columns of the rectangular grid in the rectangular network, the local homography matrix corresponding to the rectangular grid is obtained; Based on local homography, the mapping point of the target image in the target image is calculated; The formula for calculating the number of rows and columns of the target image in the rectangular network is as follows: , Official (10); The mapping point The calculation formula is: , Official (11); Where row and col are the number of rows and columns of the target image in the rectangular network, respectively. The pixel size is the source image size, and floor indicates truncation. The local homography matrix of the target image in the source image. The i-th row vector.

9. An electronic device, characterized in that, The method includes a computer program stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of the image target matching method based on local homography as described in any one of claims 1-5.

Images