Image target matching method and device based on local homography

Through the image target matching method based on local homography, the mapping points of the image target are calculated using the local homography matrix, and the problem of poor robustness of rotation, scale and illumination changes between images in the prior art is solved, and the matching accuracy and speed are improved , reduces the probability of mismatch of cluster targets.

CN119027470AActive Publication Date: 2024-11-26ANHUI UNIVERSITY OF TECHNOLOGY

Patent Information

Application Number
CN202411179348.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-27
Publication Date
2024-11-26
Estimated Expiration
2044-08-27

AI Technical Summary

Technical Problem

The existing image target matching algorithms have poor robustness when there are significant rotation changes, scale changes, and lighting changes between images, and cluster target matching with similar characteristics and dense distribution in complex scenarios is poor, making mismatch prone to occur.

Method used

Using the image target matching method based on local homography, the natural feature points of the background image pair are determined, and the source background image is uniformly divided into a rectangular grid, and the local homography matrix is ​​calculated by combining the moving direct linear transformation algorithm to identify the sub-image of the image target. The center of quality coordinates are used to calculate the mapping points of the image target to be matched in the target image using the local holographic matrix, and the number of image targets in the mapping area is judged to determine the matching points.

Benefits of technology

It significantly improves the speed and accuracy of image target matching, reduces the probability of mismatching cluster targets with similar features and dense distribution in complex environments, and improves matching accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027470A_ABST
    Figure CN119027470A_ABST
Patent Text Reader

Abstract

The invention provides an image target matching method and device based on local homography, and relates to the field of image processing. The method comprises the following steps: acquiring natural feature point pairs between background image pairs in advance by using a registration algorithm based on image local features, then uniformly dividing a source background image into a plurality of rectangular grids, and calculating a local homography matrix of each rectangular grid by using a mobile direct linear transformation algorithm; and calculating a mapping point of a to-be-matched image target in the source image in the target image by using the local homography matrix, and taking a unique image target existing in a mapping region or an image target closest to the mapping point in Euclidean distance as a matching point. The local homography matrix is adopted to weaken mapping errors introduced by parallax changes, the image target matching precision is improved, the probability of mismatching of cluster targets with similar features and dense distribution in a complex environment is reduced, and the image target matching speed is remarkably improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image target matching method and device based on local homography. Background Art

[0002] In the field of computer stereo vision geometric measurement, such as target positioning, 3D reconstruction, deformation measurement, etc., image target matching is an indispensable step. Image target matching is a process of extracting target features using existing target image information, matching them with the feature information of the image to be matched, and estimating the target position.

[0003] At present, image target matching methods include template matching-based methods, image registration-based matching methods, and deep learning-based matching methods.

[0004] Template matching methods include grayscale template-based matching methods, feature template-based matching methods, and active wavelet-based matching methods. Grayscale template-based matching algorithms are not invariant to rotation changes, scale changes, affine changes, etc. between images, and it is difficult to have high matching accuracy when the illumination changes dramatically. Feature template-based matching methods generally use directional gradient histogram features HOG and SIFT features, and use support vector machine SVM, AdaBoost and other classification algorithms to identify targets and determine positions. The above classification algorithms have high detection accuracy, but slow execution speed. The matching method based on active wavelet base encodes the target contour with the active wavelet base of the target image, and uses the sum maximum mapping method to determine the target position of the image to be matched during the sliding window matching process. This method is suitable for application scenarios where the target contour is obvious and the rotation scale changes are not large.

[0005] The matching method based on image registration uses the local feature point pairs between the source image and the target image to calculate the geometric transformation between the images, thereby estimating the position of the target in the image to be matched. The geometric transformation between images includes affine transformation and homography transformation, and homography transformation includes global homography transformation and local homography transformation. The prior art proposes a moving direct linear transformation algorithm in the approximate projection transformation algorithm, divides the image into a rectangular grid image, and uses a smooth extrapolation method to obtain the local homography transformation matrix to describe the mapping relationship between overlapping areas between different scene images. The matching method based on image registration makes up for the defect that the template-based matching method does not have rotation change and scale change invariance, and can achieve online matching of single or multiple targets by using the known geometric transformation between images.

[0006] The target matching method based on deep learning first trains a large number of target image samples, and then uses the trained network model to identify and locate the target in the matching image. The matching method based on deep learning has a good effect on the recognition and location of targets in scene images, but the complex training model requires a rich image training set and cannot solve the matching of multiple targets of the same type.

[0007] Clustered image targets are densely distributed and have similar features. Existing target matching algorithms often improve matching accuracy at the expense of time, and do not have a good effect on matching clustered image targets with similar features. Therefore, it is necessary to propose a new image target matching method based on image registration, which estimates the mapping position of the target based on the geometric constraints between scene images and improves the matching accuracy and speed of clustered targets. Summary of the invention

[0008] The purpose of the present invention is to provide an image target matching method and device based on local homography, so as to solve the defect of poor robustness of existing image target matching algorithms when there are significant rotation changes, scale changes, and illumination changes between images, and to avoid the problem of mismatching of cluster targets with similar features and dense distribution in complex scenes.

[0009] To achieve the above object, the present invention proposes the following technical solutions:

[0010] Firstly, a local homography-based image target matching method is proposed, comprising:

[0011] Determine a background image pair, and obtain a natural feature point pair between the background image pair; wherein the background image pair consists of a source background image and a target background image;

[0012] The source background image is evenly divided into C1×C2 rectangular grids, and the local homography matrix corresponding to each rectangular grid is calculated in combination with the natural feature point pairs; wherein C1 and C2 are the number of portions into which the source background image is evenly divided in the height direction and the width direction, respectively, and C1≥10 and C2≥10;

[0013] The image targets in the source image and the target image are identified by using an image target detection method, the sub-pixel centroid coordinates of each of the image targets are obtained, and the rectangular grid corresponding to the image target to be matched in the source background image and the local homography matrix corresponding to the rectangular grid are determined; wherein the source image and the target image are a pair of images to be matched in which the image targets are photographed;

[0014] Calculate the mapping point of the image target to be matched in the target image according to the local homography matrix of the sub-pixel centroid coordinates of the image target in the source image and the corresponding rectangular grid;

[0015] Determine a mapping area of ​​the mapping point in the target image, and judge the number of image targets in the mapping area; when there is only a single image target in the mapping area, determine the image target to be the matching point of the image target to be matched in the target image; when there are multiple image targets in the mapping area, determine the image target with the smallest Euclidean distance to the mapping point to be the matching point of the image target to be matched in the target image.

[0016] Furthermore, the process of calculating the local homography matrix corresponding to each rectangular grid in combination with the natural feature point pairs is to use a moving direct linear transformation algorithm in combination with the natural feature point pairs to calculate the local homography matrix of each rectangular grid in the source background image, including the following steps:

[0017] 2.1) Using global homography to describe the mapping relationship between the background image pairs, a matrix equation for solving the global homography matrix is ​​established based on the natural feature point pairs, specifically:

[0018] s[x′y′ 1] T =H g [xy 1] T Formula (1);

[0019]

[0020] Where s is the proportional factor; H g is the global homography matrix; {b(x,y),b′(x′,y′)} is a pair of natural feature points between the background image pair; h=[h 11 ,h 12 ,h 13 ,h 21 ,h 22 ,h 23 ,h 31 ,h 32 ,1] T ;h ij (i,j=1,2,3) is the global homography matrix H g The element in the i-th row and j-th column of ;

[0021] 2.2) Calculate the global homography matrix H between the background image pair using the direct linear transformation method g , the calculation formula is as follows:

[0022]

[0023] Where n is the number of natural feature point pairs; a i ∈R 2×9 , is the i-th group of natural feature point pairs {x i ,x i'}The two rows of coefficients in the matrix equation of the global homography matrix, is the optimal solution of vector h;

[0024] 2.3) According to the natural feature point pairs and the global homography matrix H g , a weighted method is used to calculate the local homography matrix H of the center points of each rectangular grid in the source background image * , the calculation formula is as follows:

[0025]

[0026] Among them, h * is the local homography matrix H * The column vector of each element in; weight It varies according to the image coordinate x, and its calculation formula is as follows:

[0027] w * i =exp(-||x * -x i || 2 / σ 2 ) formula (5);

[0028] Where σ is the proportional parameter; x * 、x i are the pixel coordinates of the center point of any rectangular grid and the i-th natural feature point on the source image.

[0029] Furthermore, in step 2.3), a weighted method is used to calculate the local homography matrix H of the center points of each rectangular grid in the source background image. * The calculation formula is described in matrix form as:

[0030]

[0031] W * =diag([w * 1 w * 1 … w * n w * n ]) Formula (7);

[0032] Among them, A is Vertical stacking of; weight W * ∈R 2n×2n , and consists of a diagonal matrix.

[0033] Furthermore, when the center point x of the rectangular grid * With natural feature point xi When the distance exceeds the preset distance threshold, the threshold γ is used, 0≤γ≤1 to offset the weight, and the weight calculation is adjusted to:

[0034] w * i =max(exp(-||x * -x i || 2 / σ 2 ),γ) formula (8).

[0035] Furthermore, the process of obtaining the sub-pixel centroid coordinates of each of the image objects includes:

[0036] Using a frame difference method or a background subtraction method based on sequence images to identify image objects in the source image and the target image, and obtaining a region of interest of each image object;

[0037] For any region of interest of the image target, a connected domain of the image target is obtained by using a morphological processing method, and sub-pixel centroid coordinates of the image target are calculated by using an average method, so as to determine the position of the image target in the source image and the target image;

[0038] The sub-pixel centroid coordinate calculation formula of the image target is as follows:

[0039]

[0040] Among them, k is the total number of contour pixels in the connected domain of the image target, (u i ,v i ) is the image coordinate of the i-th contour pixel, (x c ,y c ) is the sub-pixel centroid coordinate of the image object.

[0041] Furthermore, the process of calculating the mapping points of the to-be-matched image target in the target image includes:

[0042] According to the sub-pixel centroid coordinates of the image target to be matched, searching for the rectangular grid corresponding to the image target to be matched in the source background image, and determining the number of rows and columns of the rectangular grid in the rectangular network;

[0043] According to the number of rows and columns of the rectangular grid in the rectangular network, obtaining a local homography matrix corresponding to the rectangular grid;

[0044] Calculating the mapping points of the to-be-matched image target in the target image according to the local homography;

[0045] The formula for calculating the number of rows and columns of the image target to be matched in the rectangular network is:

[0046]

[0047] The calculation formula of the mapping point (x', y') is:

[0048]

[0049] Among them, row and col are the number of rows and columns of the image target to be matched in the rectangular network, (height, width) is the pixel size of the source image, and floor means truncation and rounding. is the local homography matrix H corresponding to the image target to be matched in the source image * The i-th row vector of .

[0050] In a second aspect, an image target matching device based on local homography is proposed, comprising:

[0051] A determination acquisition module is used to determine a background image pair and acquire a natural feature point pair between the background image pair; wherein the background image pair is composed of a source background image and a target background image;

[0052] A division calculation module is used to evenly divide the source background image into C1×C2 rectangular grids, and calculate the local homography matrix corresponding to each rectangular grid in combination with the natural feature point pairs; wherein C1 and C2 are the number of evenly divided portions of the source background image in the height direction and the width direction, respectively, and C1≥10 and C2≥10;

[0053] An identification and acquisition module is used to identify the image targets in the source image and the target image by using an image target detection method, obtain the sub-pixel centroid coordinates of each of the image targets, and determine the rectangular grid corresponding to the image target to be matched in the source background image and the local homography matrix corresponding to the rectangular grid; wherein the source image and the target image are a pair of images to be matched in which the image targets are photographed;

[0054] A calculation module, used for calculating the mapping points of the image target to be matched in the target image according to the local homography matrix of the sub-pixel centroid coordinates of the image target in the source image and the corresponding rectangular grid;

[0055] A determination judgment module is used to determine a mapping area of ​​the mapping point in the target image and to judge the number of image targets in the mapping area; when there is only a single image target in the mapping area, the image target is determined to be a matching point of the image target to be matched in the target image; when there are multiple image targets in the mapping area, the image target with the smallest Euclidean distance to the mapping point is determined to be a matching point of the image target to be matched in the target image.

[0056] Furthermore, the process of the division calculation module calculating the local homography matrix corresponding to each rectangular grid in combination with the natural feature point pairs includes the following steps:

[0057] The global homography is used to describe the mapping relationship between the background image pairs, and a matrix equation for solving the global homography matrix is ​​established according to the natural feature point pairs, specifically:

[0058] s[x′y′ 1] T =H g [xy 1] T Formula (1);

[0059]

[0060] Where s is the proportional factor; H g is the global homography matrix; {b(x,y),b′(x′,y′)} is a pair of natural feature points between the background image pair; h=[h 11 ,h 12 ,h 13 ,h 21 ,h 22 ,h 23 ,h 31 ,h 32 ,1] T ;h ij (i,j=1,2,3) is the global homography matrix H g The element in the i-th row and j-th column of ;

[0061] The global homography matrix H between the background image pairs is calculated using a direct linear transformation method. g , the calculation formula is as follows:

[0062]

[0063] Where n is the number of natural feature point pairs; a i ∈R 2×9 , is the i-th group of natural feature point pairs {x i ,x i '}The two rows of coefficients in the matrix equation of the global homography matrix, is the optimal solution of vector h;

[0064] According to the natural feature point pairs and the global homography matrix H g , a weighted method is used to calculate the local homography matrix H of the center points of each rectangular grid in the source background image * , the calculation formula is as follows:

[0065]

[0066] Among them, h* is the local homography matrix H * The column vector of each element in; weight It varies according to the image coordinate x, and its calculation formula is as follows:

[0067] w * i =exp(-||x * -x i || 2 / σ 2 ) formula (5);

[0068] Where σ is the proportional parameter; x * 、x i are the pixel coordinates of the center point of any rectangular grid and the i-th natural feature point on the source image.

[0069] Furthermore, the calculation module calculates the mapping points of the image target to be matched in the target image according to the local homography matrix of the sub-pixel centroid coordinates of the image target in the source image and the corresponding rectangular grid, including:

[0070] According to the sub-pixel centroid coordinates of the image target to be matched, searching for the rectangular grid corresponding to the image target to be matched in the source background image, and determining the number of rows and columns of the rectangular grid in the rectangular network;

[0071] According to the number of rows and columns of the rectangular grid in the rectangular network, obtaining a local homography matrix corresponding to the rectangular grid;

[0072] Calculating the mapping points of the to-be-matched image target in the target image according to the local homography;

[0073] The formula for calculating the number of rows and columns of the image target to be matched in the rectangular network is:

[0074]

[0075] The calculation formula of the mapping point (x', y') is:

[0076]

[0077] Among them, row and col are the number of rows and columns of the image target to be matched in the rectangular network, (height, width) is the pixel size of the source image, and floor means truncation and rounding. It is the row vector of the i-th row of the local homography matrix H* corresponding to the image target to be matched in the source image.

[0078] In the third aspect, an electronic device is proposed, comprising a computer program, which is stored in a computer-readable storage medium; when a processor of the electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device performs the steps of the above-mentioned local homography-based image target matching method.

[0079] It can be seen from the above technical solutions that the technical solutions of the present invention have the following beneficial effects:

[0080] The present invention discloses an image target matching method and device based on local homography, and the method includes: determining a background image pair, using a registration algorithm based on local image features to obtain a pair of natural feature points between the background image pairs; evenly dividing the source background image into a number of rectangular grids, combining natural feature points, and using a moving direct linear transformation algorithm to calculate the local homography matrix corresponding to each rectangular grid; using an image target detection method to identify image targets in the source image and the target image, and obtaining the sub-pixel centroid coordinates of each of the image targets; using the local homography matrix to calculate the mapping point of the image target to be matched in the source image in the target image, and using the only image target existing in the mapping area or the image target with the closest Euclidean distance to the mapping point as the matching point. The image target matching method proposed by the present invention significantly improves the image target matching speed and matching accuracy.

[0081] The present invention evenly divides the source background image into multiple rectangular grids, and uses the natural feature point pairs between the background image pairs to calculate the local homography matrix, and calculates the mapping position of the image target in the target image; compared with the global homography matrix, the local homography matrix weakens the mapping error introduced by the parallax change, improves the target matching accuracy, and reduces the probability of mismatching of cluster targets with similar features and dense distribution in complex environments; compared with the existing technology, the speed of target matching is also greatly improved.

[0082] It should be appreciated that all combinations of the foregoing concepts, as well as additional concepts described in greater detail below, may be considered to be part of the inventive subject matter of the present disclosure, provided such concepts are not mutually inconsistent.

[0083] The foregoing and other aspects, embodiments and features of the present invention can be more fully understood from the following description in conjunction with the accompanying drawings. Other additional aspects of the present invention, such as the features and / or beneficial effects of the exemplary embodiments, will be apparent from the following description or learned from the practice of the specific embodiments according to the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] The accompanying drawings are not drawn to scale according to actual reference objects. In the accompanying drawings, each identical or nearly identical component shown in various figures may be represented by the same reference numeral. For the sake of clarity, not every component is labeled in each figure. Now, embodiments of various aspects of the present invention will be described by way of example and with reference to the accompanying drawings, in which:

[0085] Figure 1 This is a flow chart of the image target matching method based on local homography of the present invention;

[0086] Figure 2 This is a schematic diagram of image target matching using a local homography matrix in the present invention;

[0087] FIG3( a ) is a distribution diagram of target positions of eight images in a source image of an embodiment;

[0088] FIG3( b ) is a distribution diagram of the positions of eight image targets in the target image of the embodiment;

[0089] Figure 4 is a natural feature point pair between a pair of background images of the embodiment;

[0090] Figure 5 A source background image of an embodiment is divided into a uniform rectangular grid schematic diagram;

[0091] Figure 6 Schematic diagram of 8 image target mapping points and real matching points in the target image;

[0092] Figure 7 It is a structural block diagram of an image target matching device based on local homography disclosed in an embodiment;

[0093] Figure 8 A structural block diagram of an electronic device disclosed in an embodiment. DETAILED DESCRIPTION

[0094] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Unless otherwise defined, the technical terms or scientific terms used herein should be the common meanings understood by people with general skills in the field to which the present invention belongs.

[0095] The words "first", "second" and similar words used in the patent application specification and claims of the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, unless the context clearly indicates otherwise, the singular form of "a", "an" or "the" and similar words do not indicate a quantitative limitation, but indicate the existence of at least one. "Include" or "comprises" and similar words mean that the elements or objects appearing before "include" or "comprises" include the features, wholes, steps, operations, elements and / or components listed after "include" or "comprises", and do not exclude the existence or addition of one or more other features, wholes, steps, operations, elements, components and / or their collections.

[0096] Based on the characteristics of cluster image targets such as dense distribution and similar features, the currently introduced target matching algorithms, such as template matching-based methods, image registration-based matching methods, and deep learning-based matching methods, when applied to cluster image target matching, often sacrifice time to improve matching accuracy, and such algorithms have poor matching effects on cluster image targets with similar features; therefore, the present invention aims to propose an image target matching method and device based on local homography, which can effectively improve not only the target matching accuracy but also the matching speed for matching cluster image targets with similar features.

[0097] Specific as Figure 1 As shown, the image target matching method based on local homography provided by the present invention specifically includes the following steps:

[0098] 1) determining a background image pair, and obtaining a natural feature point pair between the background image pair; wherein the background image pair consists of a source background image and a target background image;

[0099] In the specific implementation, a registration algorithm based on local image features (VL-SIFT) is used to extract natural feature point pairs between background image pairs, and the random sampling consensus algorithm (RANSAC) is used to further purify them.

[0100] 2) The source background image is evenly divided into C1×C2 rectangular grids, and the local homography matrix corresponding to each rectangular grid is calculated in combination with the natural feature point pairs; wherein C1 and C2 are the number of portions into which the source background image is evenly divided in the height direction and the width direction, respectively, and C1≥10 and C2≥10;

[0101] In the embodiment, a moving direct linear transform algorithm (Moving DLT) is used in combination with the natural feature point pairs to calculate the local homography matrix of each rectangular grid in the source background image; specifically, the following steps are included:

[0102] 2.1) Using global homography to describe the mapping relationship between the background image pairs, as shown in formula (1); establishing a matrix equation for solving the global homography matrix based on the natural feature point pairs, as shown in formula (2), specifically:

[0103] s[x′y′ 1] T =H g [xy 1] T Formula (1);

[0104]

[0105] Where s is the proportional factor; H g is the global homography matrix; {b(x,y),b′(x′,y′)} is a pair of natural feature points between the background image pair; h=[h 11 ,h 12 ,h 13 ,h 21 ,h 22 ,h 23 ,h 31 ,h 32 ,1] T ;h ij (i,j=1,2,3) is the global homography matrix H g The element in the i-th row and j-th column of ;

[0106] 2.2) Calculate the global homography matrix H between the background image pair using the direct linear transformation method (DLT) g , the calculation formula is as follows:

[0107]

[0108] Where n is the number of natural feature point pairs; a i ∈R 2×9 , is the i-th group of natural feature point pairs {x i ,x i '}The two rows of coefficients in the matrix equation of the global homography matrix, is the optimal solution of vector h;

[0109] 2.3) According to the natural feature point pairs and the global homography matrix H g , the weighted method is used to calculate the local homography matrix H* of the center points of each rectangular grid in the source background image. The calculation formula is as follows:

[0110]

[0111] Among them, h* is the column vector composed of each element in the local homography matrix H*; the weight It varies according to the image coordinate x, and its calculation formula is as follows:

[0112] w * i =exp(-||x * -x i || 2 / σ 2 ) formula (5);

[0113] Where σ is the proportional parameter; x*, x i are the pixel coordinates of the center point of any rectangular grid and the i-th natural feature point on the source image.

[0114] Optionally, the formula (4) for calculating the local homography matrix H* of the center points of each rectangular grid between the background image pair can be described in matrix form, that is:

[0115]

[0116] W * =diag([w * 1 w * 1 … w * n w * n ]) Formula (7);

[0117] Among them, A is Vertical stacking of; weight W * ∈R 2n×2n , and consists of a diagonal matrix; and formula (6) is a weighted singular value decomposition problem, and its solution is W * The right singular vectors of A. In addition, in order to improve the computational efficiency, the local homography matrix of any rectangular grid center obtained by formula (6) is used as the local homography matrix of all target points in the rectangular grid.

[0118] Formula (6) is not applicable in some scenarios, such as when the center point x of the rectangular grid is * With natural feature point x i When the distance exceeds the preset distance threshold, that is, the distance is far, the value of the weight is not significant, and the solution of formula (6) is unstable; in order to prevent numerical problems in the calculation of the local homography matrix, a threshold γ, 0≤γ≤1 is used to offset the weight, and the weight calculation formula is adjusted to:

[0119] w * i =max(exp(-||x * -x i || 2 / σ 2),γ) formula (8).

[0120] 3) using an image target detection method to identify image targets in the source image and the target image, obtaining the sub-pixel centroid coordinates of each of the image targets, and determining a rectangular grid corresponding to the image target to be matched in the source background image and a local homography matrix corresponding to the rectangular grid; wherein the source image and the target image are a pair of images to be matched in which the image targets are photographed, the background of the source image is the source background image, and the background of the target image is the target background image;

[0121] The process of obtaining the sub-pixel centroid coordinates of each of the image targets includes: using a frame difference method or a background subtraction method based on a sequence of images to identify the image targets in the source image and the target image, and obtaining the region of interest of each image target; for the region of interest of any of the image targets, using a morphological processing method to obtain the connected domain of the image target, and using an average method to calculate the sub-pixel centroid coordinates of the image target, thereby determining the position of the image target in the source image and the target image; wherein the calculation formula of the sub-pixel centroid coordinates of the image target is as follows:

[0122]

[0123] Among them, k is the total number of contour pixels in the connected domain of the image target, (u i ,v i ) is the image coordinate of the i-th contour pixel, (x c ,y c ) is the sub-pixel centroid coordinate of the image object.

[0124] 4) calculating the mapping points of the image target to be matched in the target image according to the local homography matrix of the sub-pixel centroid coordinates of the image target in the source image and the corresponding rectangular grid;

[0125] In a specific implementation, the process of calculating the mapping points of the to-be-matched image target in the target image includes:

[0126] According to the sub-pixel centroid coordinates of the image target to be matched, searching for the rectangular grid corresponding to the image target to be matched in the source background image, and determining the number of rows and columns of the rectangular grid in the rectangular network;

[0127] According to the number of rows and columns of the rectangular grid in the rectangular network, obtaining a local homography matrix corresponding to the rectangular grid;

[0128] Calculating the mapping points of the to-be-matched image target in the target image according to the local homography;

[0129] The formula for calculating the number of rows and columns of the image target to be matched in the rectangular network is:

[0130]

[0131] The calculation formula of the mapping point (x', y') is:

[0132]

[0133] Among them, row and col are the number of rows and columns of the image target to be matched in the rectangular network, (height, width) is the pixel size of the source image, and floor means truncation and rounding. is the local homography matrix H corresponding to the image target to be matched in the source image * The i-th row vector of .

[0134] 5) Determine the mapping area of ​​the mapping point in the target image and judge the number of image targets in the mapping area; when there is only a single image target in the mapping area, determine the image target as the matching point of the image target to be matched in the target image; when there are multiple image targets in the mapping area, determine the image target with the smallest Euclidean distance to the mapping point as the matching point of the image target to be matched in the target image.

[0135] Combination Figure 2 The image target matching process shown in the figure, c1, c2 are the first and second cameras respectively, the first camera shoots to obtain the source image, and the second camera shoots to obtain the target image, l, l' are polar lines, e, e' are poles, b1 is the projection point of target 1 on the source image, b1', b2' are the projection points of target 1 and target 2 on the target image respectively; after determining the position of the image target b1 to be matched in the source image, find the rectangular grid where the image target to be matched is located, and estimate the mapping point m of the image target in the target image according to the local homography matrix of the image target position in the source image and the matrix grid where it is located, and search for the target in the mapping area with the mapping point m as the center and a radius of r, where the radius r is 0.75 times the diagonal length of the rectangular grid; there are two image targets b1' and b2' in the mapping area, and the image target b1' with the closest Euclidean distance to the mapping point is selected as the matching point of target 1 in the target image.

[0136] In the specific implementation process of the image target matching method of the present invention, steps 1)-2) are pre-offline operation processes before image target matching, and steps 3)-5) are online operation processes during image target matching; the time-consuming local homography matrix calculation is an offline operation before image target matching, and the time consumption of image target matching mainly includes target recognition, target mapping point estimation and nearest Euclidean distance search, which effectively improves the image target matching speed.

[0137] The following is a further detailed introduction to the image target matching method based on local homography disclosed in the present invention in conjunction with the embodiments shown in the accompanying drawings.

[0138] like Figure 3(a) to Figure 3(b) As shown, the present invention uses the above steps to match the 8 image target points in the source image and the target image. In this embodiment, the positions of the 8 image target points in the source image and the target image are determined by placing markers, and it is assumed that the sub-pixel centroid coordinates of each image target are obtained by the average method after image target recognition; Figure 3 (a) and Figure 3 (b) are obtained by a pair of binocular cameras with a baseline distance of 30m and a measurement distance of about 100m. The size of the source image and the target image are both 1920 pixels × 1200 pixels. Figure 4 As shown in FIG, the natural feature point pairs between the background image pairs are extracted using the VL-SIFT algorithm, and the source background image is evenly divided into 800 (C1 = 20, C2 = 40) rectangular grids, as shown in FIG. Figure 5 The present invention uses a moving direct linear transformation algorithm (DLT) to estimate the mapping points of 8 image target points in the source image in the target image, and uses the Euclidean distance between the mapping point and the real matching point as a measure of the image target matching accuracy.

[0139] The results are as follows Figure 6 As shown, where: the * points are image target mapping points, and the red dots are real matching points; the data show that the mapping points of the image targets calculated by the method of the present invention in the target image are close to the real matching points, and the root mean square error (RMSE) of the Euclidean distance between the mapping points and the matching points does not exceed 12.9 pixels when the image size is 1920 pixels × 1200 pixels.

[0140] It is particularly noted that the method of the present invention uniformly dividing the source background image into a plurality of rectangular grids and calculating the local homography matrix of the rectangular grids using the moving linear transformation algorithm is suitable for application scenarios where the natural feature points of the background image are very rich. For scenarios where the natural feature points of the background image are sparse, the target mapping position estimated by the local homography matrix is ​​close to the mapping position estimated by the global homography matrix, and the image target matching accuracy is reduced. For application scenarios where the natural feature points of the background image are rich, appropriately increasing the number of uniformly divided rectangular grids is helpful to improve the target matching accuracy.

[0141] In the embodiment disclosed in the present invention, an electronic device is also provided, including a computer program, wherein the computer program is stored in a computer-readable storage medium; when a processor of the electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device performs the steps of the above-mentioned image target matching method based on local homography. Taking an electronic device running on a computer as an example, Figure 8As shown, the electronic device may include one or more (only one is shown in the figure) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory for storing data, and a transmission device for communication functions. A person skilled in the art can understand that Figure 8 The structure shown is for illustration only and does not limit the structure of the above-mentioned electronic device.

[0142] The above program can be run in the processor, or it can be stored in the memory, that is, in the computer-readable medium, which includes permanent and non-permanent, removable and non-removable media. Information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer-readable media does not include temporary computer-readable media, such as modulated data signals and carrier waves. These computer programs can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are performed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps of the functions specified in a block or multiple blocks may be implemented by different modules corresponding to different method steps.

[0143] In this embodiment, such a device or system is provided. The device can be called an image target matching device based on local homography. Figure 7As shown, it includes: a determination acquisition module, which is used to determine a background image pair and obtain a natural feature point pair between the background image pair; wherein the background image pair is composed of a source background image and a target background image; a division calculation module, which is used to evenly divide the source background image into C1×C2 rectangular grids, and calculate the local homography matrix corresponding to each rectangular grid in combination with the natural feature point pair; wherein C1 and C2 are the number of parts into which the source background image is evenly divided in the height direction and the width direction, respectively, and C1≥10, C2≥10; an identification acquisition module, which is used to identify the image targets in the source image and the target image by using an image target detection method, obtain the sub-pixel centroid coordinates of each of the image targets, determine the rectangular grid corresponding to the image target to be matched in the source background image and the corresponding a local homography matrix; wherein the source image and the target image are a pair of images to be matched in which image targets are captured respectively; a calculation module, used to calculate the mapping points of the image target to be matched in the target image according to the local homography matrix of the sub-pixel centroid coordinates of the image target in the source image and its corresponding rectangular grid; a determination and judgment module, used to determine the mapping area of ​​the mapping point in the target image and judge the number of image targets in the mapping area; when there is only a single image target in the mapping area, the image target is determined to be the matching point of the image target to be matched in the target image; when there are multiple image targets in the mapping area, the image target with the smallest Euclidean distance to the mapping point is determined to be the matching point of the image target to be matched in the target image.

[0144] The steps of the system for implementing the local homography-based image target matching method disclosed in the above embodiment have been described and will not be repeated here.

[0145] For example, the process of the division calculation module calculating the local homography matrix corresponding to each rectangular grid in combination with the natural feature point pair includes the following steps:

[0146] The global homography is used to describe the mapping relationship between the background image pairs, and a matrix equation for solving the global homography matrix is ​​established according to the natural feature point pairs, specifically:

[0147] s[x′y′ 1] T =H g [xy 1] T Formula (1);

[0148]

[0149] Where s is the proportional factor; H g is the global homography matrix; {b(x,y),b′(x′,y′)} is a pair of natural feature points between the background image pair; h=[h 11 ,h12 ,h 13 ,h 21 ,h 22 ,h 23 ,h 31 ,h 32 ,1] T ;h ij (i,j=1,2,3) is the global homography matrix H g The element in the i-th row and j-th column of ;

[0150] The global homography matrix H between the background image pairs is calculated using a direct linear transformation method. g , the calculation formula is as follows:

[0151]

[0152] Where n is the number of natural feature point pairs; a i ∈R 2×9 , is the i-th group of natural feature point pairs {x i ,x i '}The two rows of coefficients in the matrix equation of the global homography matrix, is the optimal solution of vector h;

[0153] According to the natural feature point pairs and the global homography matrix H g , a weighted method is used to calculate the local homography matrix H of the center points of each rectangular grid in the source background image * , the calculation formula is as follows:

[0154]

[0155] Among them, h * is the local homography matrix H * The column vector of each element in; weight It varies according to the image coordinate x, and its calculation formula is as follows:

[0156] w * i =exp(-||x * -x i || 2 / σ 2 ) formula (5);

[0157] Where σ is the proportional parameter; x * 、x i are the pixel coordinates of the center point of any rectangular grid and the i-th natural feature point on the source image.

[0158] Among them, the weighted method is used to calculate the local homography matrix H of the center point of each rectangular grid in the source background image *The calculation formula is described in matrix form as:

[0159]

[0160] W * =diag([w * 1 w * 1 … w * n w * n ]) Formula (7);

[0161] Among them, A is Vertical stacking of; weight W * ∈R 2n×2n , and consists of a diagonal matrix.

[0162] And, when the center point x of the rectangular grid * With natural feature point x i When the distance exceeds the preset distance threshold, the threshold γ is used, 0≤γ≤1 to offset the weight, and the weight calculation is adjusted to:

[0163] w * i =max(exp(-||x * -x i || 2 / σ 2 ),γ) formula (8).

[0164] For another example, the process of the recognition acquisition module acquiring the sub-pixel centroid coordinates of each of the image targets includes: using a frame difference method or a background subtraction method based on a sequence of images to identify the image targets in the source image and the target image, and obtaining the region of interest of each image target; for the region of interest of any of the image targets, using a morphological processing method to obtain the connected domain of the image target, and using the average method to calculate the sub-pixel centroid coordinates of the image target, thereby determining the position of the image target in the source image and the target image; wherein the calculation formula of the sub-pixel centroid coordinates of the image target is as follows:

[0165]

[0166] Among them, k is the total number of contour pixels in the connected domain of the image target, (u i ,v i ) is the image coordinate of the i-th contour pixel, (x c ,y c ) is the sub-pixel centroid coordinate of the image object.

[0167] For another example, the process in which the calculation module calculates the mapping points of the image target to be matched in the target image includes: searching for the rectangular grid corresponding to the image target to be matched in the source background image according to the sub-pixel centroid coordinates of the image target to be matched, and determining the number of rows and columns of the rectangular grid in the rectangular network; obtaining the local homography matrix corresponding to the rectangular grid according to the number of rows and columns of the rectangular grid in the rectangular network; calculating the mapping points of the image target to be matched in the target image according to the local homography; the calculation formula for the number of rows and columns of the image target to be matched in the rectangular network is:

[0168]

[0169] The calculation formula of the mapping point (x', y') is:

[0170]

[0171] Among them, row and col are the number of rows and columns of the image target to be matched in the rectangular network, (height, width) is the pixel size of the source image, and floor means truncation and rounding. is the local homography matrix H corresponding to the image target to be matched in the source image * The i-th row vector of .

[0172] The present invention discloses an image target matching method and device based on local homography, which solves the defect of poor robustness of existing image target matching algorithms when there are significant rotation changes, scale changes, and illumination changes between images, and avoids the problem of easy mismatching of cluster targets with similar features and dense distribution in complex scenes. The local homography matrix is ​​used to weaken the mapping error introduced by parallax changes, improve the image target matching accuracy, reduce the probability of mismatching of cluster targets with similar features and dense distribution in complex environments, and significantly improve the image target matching speed.

[0173] Although the present invention has been disclosed as above with preferred embodiments, it is not intended to limit the present invention. A person with ordinary knowledge in the technical field to which the present invention belongs may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be determined by the definition of the claims.

Claims

1. An image target matching method based on local homography, It is characterized in that include: Determine a background image pair, and obtain a natural feature point pair between the background image pair; wherein the background image pair consists of a source background image and a target background image; Divide the source background image evenly into C 1 ×C 2 rectangular grids, and calculate the local homography matrix corresponding to each rectangular grid in combination with the natural feature point pairs; wherein, C 1 ,C 2 are the numbers of portions into which the source background image is evenly divided in the height direction and the width direction, respectively, and C 1 ≥10, C 2 ≥10; The image targets in the source image and the target image are identified by using an image target detection method, the sub-pixel centroid coordinates of each of the image targets are obtained, and the rectangular grid corresponding to the image target to be matched in the source background image and the local homography matrix corresponding to the rectangular grid are determined; wherein the source image and the target image are a pair of images to be matched in which the image targets are photographed; Calculate the mapping point of the image target to be matched in the target image according to the local homography matrix of the sub-pixel centroid coordinates of the image target in the source image and the corresponding rectangular grid; Determine a mapping area of ​​the mapping point in the target image, and judge the number of image targets in the mapping area; when there is only a single image target in the mapping area, determine the image target to be the matching point of the image target to be matched in the target image; when there are multiple image targets in the mapping area, determine the image target with the smallest Euclidean distance to the mapping point to be the matching point of the image target to be matched in the target image.

2. The image target matching method based on local homography according to claim 1, It is characterized in that The process of calculating the local homography matrix corresponding to each rectangular grid in combination with the natural feature point pairs is to use a moving direct linear transformation algorithm in combination with the natural feature point pairs to calculate the local homography matrix of each rectangular grid in the source background image, including the following steps: 2.1) Using global homography to describe the mapping relationship between the background image pairs, a matrix equation for solving the global homography matrix is ​​established based on the natural feature point pairs, specifically: s[x′y′ 1] T = H g [xy 1] T Formula (1): Where s is the proportional factor; H g is the global homography matrix; {b(x, y), b′(x′, y′)} is a pair of natural feature points between the background image pair; h = [h 11 ,h 12 ,h 13 ,h 21 ,h 22 ,h 23 ,h 31 ,h 32 , 1] T ;h ij (i, j = 1, 2, 3) is the global homography matrix H g The element in the i-th row and j-th column of ; 2.2) Calculate the global homography matrix H between the background image pair using the direct linear transformation method g , the calculation formula is as follows: Where n is the number of natural feature point pairs; a i ∈R 2×9 , is the i-th group of natural feature point pairs {x i , x i ′} are the two row coefficients in the matrix equation of the global homography matrix, is the optimal solution of vector h; 2.3) According to the natural feature point pairs and the global homography matrix H g , a weighted method is used to calculate the local homography matrix H of the center points of each rectangular grid in the source background image * , the calculation formula is as follows: Among them, h * is the local homography matrix H * The column vector of each element in; weight It varies according to the image coordinate x, and its calculation formula is as follows: w * i =exp(-||x * -x i || 2 / σ 2 ) formula (5); Where σ is the proportional parameter; x * 、x i are the pixel coordinates of the center point of any rectangular grid and the i-th natural feature point on the source image.

3. The image target matching method based on local homography according to claim 2, It is characterized in that In the step 2.3), a weighted method is used to calculate the local homography matrix H of the center points of each rectangular grid in the source background image. * The calculation formula is described in matrix form as: W * = diag([w * 1 w * 1 ... w * n w * n ) Formula (7); Among them, A is Vertical stacking of; weight W * ∈R 2n×2n , and consists of a diagonal matrix.

4. The image target matching method based on local homography according to claim 2, It is characterized in that When the center point x of the rectangular grid * With natural feature point x i When the distance exceeds the preset distance threshold, the threshold γ is used, 0≤γ≤1 to offset the weight, and the weight calculation is adjusted to: w * i =max(exp(-||x * -x i || 2 / σ 2 ), γ) formula (8).

5. The image target matching method based on local homography according to claim 1, It is characterized in that The process of obtaining the sub-pixel centroid coordinates of each image target includes: Using a frame difference method or a background subtraction method based on sequence images to identify image objects in the source image and the target image, and obtaining a region of interest of each image object; For any region of interest of the image target, a connected domain of the image target is obtained by using a morphological processing method, and sub-pixel centroid coordinates of the image target are calculated by using an average method, so as to determine the position of the image target in the source image and the target image; The sub-pixel centroid coordinate calculation formula of the image target is as follows: Among them, k is the total number of contour pixels in the connected domain of the image target, (u i , v i ) is the image coordinate of the i-th contour pixel, (x c ,y c ) is the sub-pixel centroid coordinate of the image object.

6. The image target matching method based on local homography according to claim 1, It is characterized in that The process of calculating the mapping points of the to-be-matched image target in the target image includes: According to the sub-pixel centroid coordinates of the image target to be matched, searching for the rectangular grid corresponding to the image target to be matched in the source background image, and determining the number of rows and columns of the rectangular grid in the rectangular network; According to the number of rows and columns of the rectangular grid in the rectangular network, obtaining a local homography matrix corresponding to the rectangular grid; Calculating the mapping points of the to-be-matched image target in the target image according to the local homography; The formula for calculating the number of rows and columns of the image target to be matched in the rectangular network is: The calculation formula of the mapping point (x', y') is: Among them, row and col are the number of rows and columns of the image target to be matched in the rectangular network, (height, width) is the pixel size of the source image, and floor means truncation and rounding. is the local homography matrix H corresponding to the image target to be matched in the source image * The i-th row vector of .

7. An image target matching device based on local homography, It is characterized in that include: A determination acquisition module is used to determine a background image pair and acquire a natural feature point pair between the background image pair; wherein the background image pair is composed of a source background image and a target background image; The division calculation module is used to evenly divide the source background image into C 1 ×C 2 rectangular grids, and calculate the local homography matrix corresponding to each rectangular grid in combination with the natural feature point pairs; wherein, C 1 ,C 2 are the numbers of portions into which the source background image is evenly divided in the height direction and the width direction, respectively, and C 1 ≥10, C 2 ≥10; An identification and acquisition module is used to identify the image targets in the source image and the target image by using an image target detection method, obtain the sub-pixel centroid coordinates of each of the image targets, and determine the rectangular grid corresponding to the image target to be matched in the source background image and the local homography matrix corresponding to the rectangular grid; wherein the source image and the target image are a pair of images to be matched in which the image targets are photographed; A calculation module, used for calculating the mapping points of the image target to be matched in the target image according to the local homography matrix of the sub-pixel centroid coordinates of the image target in the source image and the corresponding rectangular grid; A determination judgment module is used to determine a mapping area of ​​the mapping point in the target image and to judge the number of image targets in the mapping area; when there is only a single image target in the mapping area, the image target is determined to be a matching point of the image target to be matched in the target image; when there are multiple image targets in the mapping area, the image target with the smallest Euclidean distance to the mapping point is determined to be a matching point of the image target to be matched in the target image.

8. The image target matching device based on local homography according to claim 7, It is characterized in that The process of the division calculation module calculating the local homography matrix corresponding to each rectangular grid in combination with the natural feature point pairs includes the following steps: The global homography is used to describe the mapping relationship between the background image pairs, and a matrix equation for solving the global homography matrix is ​​established according to the natural feature point pairs, specifically: s[x′ y′ 1] T = H g [xy 1] T Formula (1): Where s is the proportional factor; H g is the global homography matrix; {b(x, y), b′(x′, y′)} is a pair of natural feature points between the background image pair; h = [h 11 ,h 12 ,h 13 ,h 21 ,h 22 ,h 23 ,h 31 ,h 32 , 1] T ;h ij (i, j = 1, 2, 3) is the global homography matrix H g The element in the i-th row and j-th column of ; The global homography matrix H between the background image pairs is calculated using a direct linear transformation method. g , the calculation formula is as follows: Where n is the number of natural feature point pairs; a i ∈R 2×9 , is the i-th group of natural feature point pairs {x i , x i ′} are the two row coefficients in the matrix equation of the global homography matrix, is the optimal solution of vector h; According to the natural feature point pairs and the global homography matrix H g , a weighted method is used to calculate the local homography matrix H of the center points of each rectangular grid in the source background image * , the calculation formula is as follows: Among them, h * is the local homography matrix H * The column vector of each element in; weight It varies according to the image coordinate x, and its calculation formula is as follows: w * i =exp(-||x * -x i || 2 / σ 2 ) formula (5); Where σ is the proportional parameter; x * 、x i are the pixel coordinates of the center point of any rectangular grid and the i-th natural feature point on the source image.

9. The image target matching device based on local homography according to claim 7, It is characterized in that The process of the calculation module calculating the mapping points of the image target to be matched in the target image according to the local homography matrix of the sub-pixel centroid coordinates of the image target in the source image and the corresponding rectangular grid includes: According to the sub-pixel centroid coordinates of the image target to be matched, searching for the rectangular grid corresponding to the image target to be matched in the source background image, and determining the number of rows and columns of the rectangular grid in the rectangular network; According to the number of rows and columns of the rectangular grid in the rectangular network, obtaining a local homography matrix corresponding to the rectangular grid; Calculating the mapping points of the to-be-matched image target in the target image according to the local homography; The formula for calculating the number of rows and columns of the image target to be matched in the rectangular network is: The calculation formula of the mapping point (x', y') is: Among them, row and col are the number of rows and columns of the image target to be matched in the rectangular network, (height, width) is the pixel size of the source image, and floor means truncation and rounding. is the local homography matrix H corresponding to the image target to be matched in the source image * The i-th row vector of .

10. An electronic device, It is characterized in that It includes a computer program, which is stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device performs the steps of the image target matching method based on local homography as described in any one of claims 1 to 6.

Citation Information

Patent Citations

  • Multi-target object image matching method

    CN105354578A

  • Large viewing angle image fast recognition method based on ORB algorithm

    CN106919944A

  • Improved approximate projection transformation image splicing method

    CN107016646A

  • Parallax image splicing method and system based on template matching and feature clustering

    CN110544202A

  • Image splicing method and splicing system based on grid optimization and readable storage medium

    CN112801870A

Cited By

  • Image template matching method, monitoring system, device and storage medium

    CN119888279A

  • Zoom camera focal length error correction method based on image processing

    CN122289403A