Image stitching matching method based on neighborhood topological similarity and prior probability sampling

Abstract

This invention discloses an image stitching and matching method based on neighborhood topological similarity and prior probability sampling. The method includes obtaining initial feature points in two images to be stitched; selecting calibration points from the initial feature points through brute-force matching to obtain calibration point pairs; performing a triangular network topology on the calibration points to obtain a triangular topological network of the image; obtaining neighborhood topological vectors of the calibration points through the triangular topological network; combining the neighborhood topological vectors of all calibration points in the same image to be stitched into a neighborhood topological matrix; calculating the prior probability and sampling probability of each calibration point pair using the neighborhood topological matrix; determining the optimal transformation matrix based on the calibration point pair with the higher sampling probability; and stitching the two images to be stitched together using the optimal transformation matrix. This invention introduces the concepts of prior probability and sampling probability, making calibration points with a high probability of mismatch less likely to be selected, effectively improving the efficiency and accuracy of mismatch removal of calibration points.

Description

Technical Field

[0001] This invention relates to image stitching and matching methods in the field of machine vision, and in particular to an image stitching and matching method based on neighborhood topological similarity and prior probability sampling. Background Technology

[0002] Point matching in images is a crucial aspect of computer image processing and a significant research direction in machine vision. It is widely used in target recognition, image registration, target tracking, and target pose determination. Correct point matching is fundamental to accurately identifying targets and determining the correct relationships between images. However, almost no matching method can produce perfectly correct matches, making mismatch removal crucial and a key research focus for scholars both domestically and internationally. Existing classic mismatch removal methods can be categorized into sampling-based and geometric constraint-based methods. Sampling-based methods are exemplified by the Random Sample Consensus Algorithm (RANSAC). Its principle involves randomly selecting several datasets for model calculation (here, the model is the transformation matrix between two images, which can be an essential matrix or a homography matrix). Then, it incorporates other data to calculate the error (here, the reprojection error), repeating the random sampling process until the model that best matches the most data is found. The interior points of this model are considered correct point matches, and the exterior points are removed. However, RANSAC is unsuitable when there are a large number of mismatches because it is difficult to sample correct data simultaneously, affecting localization accuracy. In recent years, many improved methods have emerged, such as DL-RANSAC, which introduces the idea of ​​descending likelihood to reduce the randomness of sampling and enable faster convergence of sampling results. Gao et al. also proposed improvements to RANSAC, accelerating detection efficiency through pre-validation and resampling during iteration. J. Bian et al. proposed GMS, which segments the initial point set before sampling to improve convergence and efficiency. Many scholars have also proposed point-based geometric constraint methods in recent years. These methods are less affected by the number of mismatches. A typical example is Luo et al.'s method of measuring the relationship between feature points using Euclidean distance and then correcting erroneous points using angular cosine. Aguilar et al. proposed GTM, which constructs an undirected graph between calibration points and uses information computation from the undirected graph to detect and remove mismatches. Zhao et al. proposed a mismatch removal method based on distance constraints between calibration points.

[0003] These geometry-based and sampling-based methods generally achieve good removal results, but they have limitations when the number of mismatches is large, especially when matching images with repeated patterns and textures. Existing technologies lack a new way to adapt to mismatch removal under conditions of large mismatch rates. Summary of the Invention

[0004] To address the shortcomings of existing methods when the mismatch rate is relatively high, this invention proposes an image stitching and matching method based on neighborhood topological similarity and prior probability sampling. It also proposes a new method for calculating the correct probability of calibration points based on triangular topological networks and weights it during the random sampling process, making it easier to select the correct calibration point pairs and meet the needs of practical applications.

[0005] The technical solution of this invention is as follows:

[0006] like Figure 1 As shown, the technical solution of the present invention includes the following steps:

[0007] This invention includes the following steps:

[0008] Step 1: Select two images that overlap in shooting space to be stitched together. One of the two images is a reference image and the other is a test image. Then, extract feature points from the two images to obtain the initial feature points of each image. Use brute force matching to filter out calibration points from the initial feature points and obtain several sets of calibration point pairs. Each set of calibration point pairs includes a calibration point from the reference image and a calibration point from the test image.

[0009] Step 2: Construct a triangular topology network

[0010] The triangular network topology of the two images to be stitched together will be obtained by referring to the calibration points in the reference image and the calibration points in the test image respectively.

[0011] Step 3: Obtain the neighborhood topology vectors of the calibration points through the triangular topology network, and form a neighborhood topology matrix by combining the neighborhood topology vectors of all calibration points on the same image to be stitched.

[0012] Step 4: Use the neighborhood topology matrix to obtain the prior probability of each pair of calibration points, and use the prior probability of the calibration point pairs to obtain the sampling probability of the calibration point pairs.

[0013] Step 5: Select four or seven sets of calibration point pairs with higher sampling probabilities as a sampling scheme, calculate the transformation matrix under different sampling schemes, determine the optimal transformation matrix, and use the optimal transformation matrix to stitch together the reference image and the test image.

[0014] The specific method for selecting calibration points and obtaining calibration point pairs through brute-force matching in step 1 is as follows:

[0015] Each initial feature point in the reference image will be matched with each initial feature point in the test image using feature matching. The initial feature points that fail to match will be deleted from the two images to be stitched together, and the remaining initial feature points will be used as calibration points. The two calibration points that match successfully will be combined into a calibration point pair.

[0016] Each triangular topology network in step 2 consists of all the calibration points in the same graph and the straight line segment connecting any two calibration points in the graph. The straight line segment connecting any two calibration points in the same graph is used as the topological line segment of the triangular topology network.

[0017] Step 3 specifically includes:

[0018] Step 3.1: Calculate the first-order neighborhood topology matrix

[0019] For calibration point P i and calibration point Q i Establish their respective first-order neighborhood topology vectors PV i and the first-order neighborhood topological vector QV i :

[0020] PV i =[pa1 pa2 pa3…pa j …pa m ] T

[0021] QV i =[qa1 qa2 qa3…qa j …qa m ] T

[0022] Among them, P i and Q i These refer to the calibration point numbers in the reference image and the calibration point numbers in the test image, respectively. j and qa j These are the first-order neighborhood topological vectors PV. i The elements in and the first-order neighborhood topological vector QV i In the elements, i and j both represent the ordinal numbers of the calibration points, and m represents the number of calibration points in each image;

[0023] The first-order neighborhood topological vector PV is determined by the topological segments in the triangular topological network. i each element pa j The possible values ​​of:

[0024] When j = i, then element pa j Set to 0;

[0025] When j ≠ i and the calibration point P iWith calibration point P j If they are connected by only one topological line segment, then a j Take 1;

[0026] When j ≠ i and the calibration point P i With calibration point P j If they are not connected by a single topological line segment, then a j Set to 0;

[0027] The first-order neighborhood topological vector QV is determined by the topological segments in the triangular topological network. i Each element qa j The possible values ​​of:

[0028] When j = i, then element qa j Set to 0;

[0029] When j≠i and the calibration point Q i With calibration point Q j If they are connected by only one topological line segment, then a j Take 1;

[0030] When j≠i and the calibration point Q i With calibration point Q j If they are not connected by a single topological line segment, then a j Set to 0;

[0031] The first-order neighborhood topology vector PV of all calibration points in the image will be referenced. i Combined into a first-order neighborhood topological matrix M p The first-order neighborhood topology vector QV of all calibration points in the test image is... i Combined into a first-order neighborhood topological matrix M q :

[0032] M p =[PV1 PV2 PV3…PV m ]

[0033] M q =[QV1 QV2 QV3…QV m ]

[0034] Step 3.2: Calculate the second-order neighborhood topology matrix

[0035] The calibration point P is calculated using the following formula. i The second-order neighborhood topological vector PV' i and calibration point Q i The second-order neighborhood topological vector QV' i :

[0036] PV' i =(E+M) P )PVi

[0037] QV' i =(E+M) q QV i

[0038] Where E is an m-order unit matrix;

[0039] The second-order neighborhood topology vector PV' of all calibration points in the image will be referenced. i Combined into a second-order neighborhood topology matrix M' p The second-order neighborhood topology vector QV' of all calibration points in the test image is... i Combined into a second-order neighborhood topology matrix M' q :

[0040] M' p =[PV'1 PV'2 PV'3 … PV' m ]

[0041] M' q =[QV'1 QV'2 QV'3 … QV' m ].

[0042] Step 4 specifically includes:

[0043] Step 4.1: Based on the second-order neighborhood topology matrix M' p and the second-order neighborhood topology matrix M' q Calculate the topological similarity matrix M using the following formula. pq :

[0044] M pq (a,b)=M' p (ap,bp)*M' q (aq, bq)

[0045] Among them, M pq (a,b) represents the topological similarity matrix M. pq The element in row a, column b of the array; M' p (ap,bp) represents the second-order neighborhood topology matrix M' p The element in row ap and column bp of the middle element; M' q (aq, bq) represents the second-order neighborhood topology matrix M' q The element in the aq-th row and bq-th column of the matrix; a and b represent the topological similarity matrix M. pq The ordinal numbers of the row vectors and column vectors; ap and bp represent the second-order neighborhood topological matrix M', respectively. p The ordinal numbers of the row vectors and column vectors are given, and aq and bq represent the second-order neighborhood topological matrix M', respectively. qThe ordinal numbers of row vectors and column vectors;

[0046] For topological similarity matrix M pq Sum each column in the matrix and sum the results in the topological similarity matrix M. pq The sum of the i-th column is denoted as S. i ;

[0047] Calculate the calibration point pair P using the following formula. i -Q i The prior probability p i :

[0048]

[0049] μ = average(Si)

[0050] σ = varp(Si)

[0051] Among them, P i -Q i The calibration point pair is designated as the number of the calibration point pair P. i -Q i From calibration point P i and calibration point Q i The formula consists of average(), varp(), and e, where e is the natural constant.

[0052] Step 4.2: Based on the prior probability p i Calculate the calibration point pair P using the following formula. i -Q i sampling probability p' i :

[0053]

[0054] Step 5 specifically includes:

[0055] Step 5.1: Select sampling probability p' i Larger sets of four or seven calibration points P i -Q i As a sampling scheme, based on each set of calibration points in the sampling scheme, P i -Q i The location information of the calibration point is used to establish calibration point P. i coordinate vector PR i and calibration point Q i coordinate vector QR i :

[0056] PR i =[px i py i w]T

[0057] QR i =[qx i qy i w] T

[0058] Where, px i py i These are the calibration points P. i The coordinate values ​​of qx in the X and Y directions of the Cartesian coordinate system. i qy i The calibration points Q are respectively i The coordinate values ​​in the X and Y directions of the Cartesian coordinate system, where w is a preset constant;

[0059] Select each set of calibration points to P i -Q i Substituting the coordinate vectors of the reference point into the following formula yields a 3×3 third-order transformation matrix H:

[0060] QR i =H PR i

[0061] All calibration points Q are calculated based on the transformation matrix H. i Coordinate prediction vector QR' i :

[0062] QR' i =H PR i =[qx' i qy' i w] T

[0063] Calibration point Q i Coordinate prediction vector QR' i and coordinate vector QR i Compare and determine the calibration point relative to P. i -Q i Check if the calibration point pair is correct and remove mismatches:

[0064] If the relation is satisfied If d is the preset calibration point distance threshold, then it indicates that the calibration point is relative to P. i -Q i To match the correct calibration point pair, retain the calibration point pair P. i -Q i As a valid calibration point pair;

[0065] Otherwise, it means that the calibration point is relative to P. i -Q iTo match incorrect calibration point pairs, the calibration point pair P is... i -Q i Remove;

[0066] Record the number of valid calibration point pairs obtained based on the transformation matrix H;

[0067] Step 5.2: Repeat step 5.1 multiple times to obtain the transformation matrix H under different sampling schemes. Select the transformation matrix H corresponding to the largest number of effective calibration point pairs as the optimal transformation matrix. Use the optimal transformation matrix to perform coordinate transformation on all pixels in the reference image. Then, stitch the two images together based on the coordinate-transformed reference image and the test image.

[0068] This invention introduces the concept of domain topological similarity into sampling, combining topological stability and local topological stability phenomena to quantify and remove mismatches. A topological network is constructed in both the reference image P and the test image Q. Neighborhood topological vectors are used to describe the topological relationships of the first-order or second-order neighbors of each calibration point. A topological similarity matrix is ​​used to compare and calculate the similarity of the neighborhood topological relationships of the two calibration points in each pair in their respective images, thereby quantifying the probability of a calibration point being a mismatch. The higher the similarity between the two calibration points in each pair in the reference and test images, the more likely they are to be a correctly matched calibration point pair, and the lower the probability of a mismatch. Subsequently, the probability of each pair of calibration points being a mismatch is incorporated into sampling consistency, making points with a high probability of mismatch less likely to be selected by the random sampling process. Through continuous sampling, multiple transformation matrices with better performance are obtained. To improve the accuracy of the results, regression is finally performed on these multiple transformation matrices to obtain an optimal transformation matrix that is more accurate than the sampling results, thereby filtering out the correct matches. This method can effectively improve the efficiency and accuracy of mismatch removal.

[0069] The beneficial effects of this invention are as follows:

[0070] 1. This invention solves the problem of low efficiency and low success rate of existing mismatch removal methods when the feature point error rate is relatively high.

[0071] 2. This invention proposes local topological stability and topological similarity of point clouds. Topological similarity is used to reflect the local topological relationships retained by point clouds in higher-dimensional transformations, and topological similarity is used to judge the correctness of calibration points, avoiding the complex calculations of angle or distance constraints in traditional geometric methods.

[0072] 3. The present invention combines the sampling method with topological operations. After performing topological relationship operations, it does not directly remove or filter points, but assigns a probability to each point, thus combining the advantages of sampling-based mismatch removal and geometric constraint-based methods.

[0073] 4. This invention extends the removal of image mismatches to point matching. This invention is still applicable to the matching of two-dimensional point clouds that are not images. Attached Figure Description

[0074] Figure 1 This is a flowchart of the method of the present invention;

[0075] Figure 2 This is a schematic diagram of the neighborhood of the calibration point;

[0076] Figure 3 Images for example embodiments;

[0077] Figure 4 Example point matching diagram;

[0078] Figure 5 The diagram shows the result of constructing a triangular topology network as an example.

[0079] Figure 6 This is a schematic diagram of the neighborhood topology.

[0080] Figure 7 This is the result image after removing mismatches; Detailed Implementation

[0081] The present invention will be further described below with reference to the accompanying drawings and embodiments. The flowchart of the present invention is as follows: Figure 1 As shown.

[0082] The implementation process of the embodiments of the present invention is as follows:

[0083] Step 1: Select two images that overlap in their shooting space to be stitched together. One image is the reference image, and the other is the test image. That is, the objects captured in the two images have overlapping areas, such as... Figure 3 As shown, scale-invariant feature transformation (SIFT) is then used to extract feature points from the two images to be stitched, obtaining initial feature points for each image. Calibration points are then selected from the initial feature points using brute-force matching, resulting in several pairs of calibration points. Each pair contains one calibration point from the reference image and one from the test image. Specifically, the reference image and the test image are selected from the Mikolajczyk VGG dataset, which is widely used in mismatch removal. The brute-force matching results are shown below. Figure 4 As shown, after brute-force matching, 320 sets of calibration point pairs are obtained. Both white and black lines are used to connect the two calibration points in the calibration point pairs.

[0084] The specific method for selecting calibration points and obtaining calibration point pairs through brute-force matching in step 1 is as follows:

[0085] Each initial feature point in the reference image will be matched with each initial feature point in the test image using feature matching. The initial feature points that fail to match will be deleted from the two images to be stitched together, and the remaining initial feature points will be used as calibration points. The two calibration points that match successfully will be combined into a calibration point pair.

[0086] Step 2: Construct a triangular topology network

[0087] The triangular network topology of the two images to be stitched together will be obtained by referring to the calibration points in the reference image and the calibration points in the test image respectively.

[0088] Each triangular topology network in step 2 consists of all the calibration points in the same image and the straight line segments connecting any two calibration points in the image. The straight line segments connecting any two calibration points in the same image are used as the topological segments of the triangular topology network. The triangular topology network is used to observe the topological connections between calibration points. The triangular topology networks of the two images to be stitched together are as follows: Figure 5 As shown.

[0089] Step 3: Obtain the neighborhood topology vectors of the calibration points through the triangular topology network, and form a neighborhood topology matrix by combining the neighborhood topology vectors of all calibration points on the same image to be stitched.

[0090] Step 3 specifically involves:

[0091] Step 3.1: Calculate the first-order neighborhood topology matrix

[0092] For calibration point P i and calibration point Q i Establish their respective first-order neighborhood topology vectors PV i and the first-order neighborhood topological vector QV i :

[0093] PV i =[pa1 pa2 pa3…pa j …pa m ]

[0094] QV i =[qa1 qa2 qa3…qa j …qa m ]

[0095] Among them, P i and Q i These refer to the calibration point numbers in the reference image and the calibration point numbers in the test image, respectively. j and qa j These are the first-order neighborhood topological vectors PV. i The elements in and the first-order neighborhood topological vector QV iIn the elements, i and j both represent the ordinal numbers of the calibration points, and m represents the number of calibration points in each image;

[0096] The first-order neighborhood topological vector PV is determined by the topological segments in the triangular topological network. i each element pa j The possible values ​​of:

[0097] When j = i, then element pa j Set to 0;

[0098] When j ≠ i and the calibration point P i With calibration point P j If they are connected by only one topological line segment, then a j Take 1;

[0099] When j ≠ i and the calibration point P i With calibration point P j If they are not connected by a single topological line segment, then a j Set to 0;

[0100] The first-order neighborhood topological vector QV is determined by the topological segments in the triangular topological network. i Each element qa j The possible values ​​of:

[0101] When j = i, then element qa j Set to 0;

[0102] When j≠i and the calibration point Q i With calibration point Q j If they are connected by only one topological line segment, then a j Take 1;

[0103] When j≠i and the calibration point Q i With calibration point Q j If they are not connected by a single topological line segment, then a j Set to 0;

[0104] Two calibration points not connected by a single topological segment include two calibration points that are not connected at all or that are connected by multiple topological segments.

[0105] like Figure 2 As shown, the first-order neighborhood topological vector of calibration point P5 is V5 = [0 0 1 1 0 1 1 1 1 0];

[0106] The first-order neighborhood topology vector PV of all calibration points in the image will be referenced. i Combined into a first-order neighborhood topological matrix M p The first-order neighborhood topology vector QV of all calibration points in the test image is... iCombined into a first-order neighborhood topological matrix M q :

[0107] M p =[PV1 PV2 PV3…PV m ]

[0108] M q =[QV1 QV2 QV3…QV m ]

[0109] Step 3.2: Calculate the second-order neighborhood topology matrix

[0110] The calibration point P is calculated using the following formula. i The second-order neighborhood topological vector PV' i and calibration point Q i The second-order neighborhood topological vector QV' i :

[0111] PV' i =(E+M) P )PV i

[0112] QV' i =(E+M) q QV i

[0113] Where E is an m-order unit matrix;

[0114] The second-order neighborhood topology vector PV' of all calibration points in the image will be referenced. i Combined into a second-order neighborhood topology matrix M' p The second-order neighborhood topology vector QV' of all calibration points in the test image is... i Combined into a second-order neighborhood topology matrix M' q :

[0115] M' p =[PV'1 PV'2 PV'3 … PV' m ]

[0116] M' q =[QV'1 QV'2 QV'3 … QV' m ].

[0117] In practice, each image can obtain a 320-dimensional neighborhood topology matrix, such as... Figure 6As shown in the figure, the triangular topology network represents the topological connections between calibration points, i.e., the neighborhood topology. White topological line segments indicate that the calibration points at both ends of the segment have consistent neighborhood topology relationships in both figures, while black topological line segments indicate that the calibration points at both ends of the segment have inconsistent neighborhood topology relationships in both figures. The figure shows that most of the retained calibration points are correct.

[0118] The consistency of the neighborhood topology between two calibration points in a pair is crucial to the accuracy of the calibration. Therefore, it is necessary to quantify the neighborhood topology using neighborhood topology vectors. This invention uses topological line segments to record the connectivity between a calibration point and its surrounding neighboring points in both images to determine consistency. The more consistent the number of topological line segments, the more similar the connectivity of the local region where the calibration point is located is in the two images, thus quantitatively representing the neighborhood similarity between the two calibration points. If the neighborhood topology vectors of the two calibration points in a pair are closer in their respective images, it indicates that the two calibration points do not change much in their topological connectivity and conform to the same transformation, thus making them more likely to be correct points.

[0119] The neighborhood topology vector represents the topological connections between each calibration point and all surrounding calibration points. A calibration point's first-order neighbors are the other calibration points directly connected to it via a topological segment. The more common first-order neighbors two calibration points in a pair have, the higher their first-order neighborhood similarity (matching degree). A calibration point's second-order neighbors are the other calibration points connected to it via two topological segments. The more common second-order neighbors two calibration points in a pair have, the higher their second-order neighborhood similarity (matching degree).

[0120] Step 4: Use the neighborhood topology matrix to obtain the prior probability of each pair of calibration points, and use the prior probability of the calibration point pairs to obtain the sampling probability of the calibration point pairs.

[0121] Step 4 specifically involves:

[0122] Step 4.1: Based on the second-order neighborhood topology matrix M' p and the second-order neighborhood topology matrix M' q Calculate the topological similarity matrix M using the following formula. pq :

[0123] M pq (a,b)=M' p (ap,bp)*M' q (aq, bq)

[0124] Among them, M pq(a,b) represents the topological similarity matrix M. pq The element in row a, column b of the array; M' p (ap,bp) represents the second-order neighborhood topology matrix M' p The element in row ap and column bp of the middle element; M' q (aq, bq) represents the second-order neighborhood topology matrix M' q The element in the aq-th row and bq-th column of the matrix; a and b represent the topological similarity matrix M. pq The ordinal numbers of the row vectors and column vectors; ap and bp represent the second-order neighborhood topological matrix M', respectively. p The ordinal numbers of the row vectors and column vectors are given, and aq and bq represent the second-order neighborhood topological matrix M', respectively. q The ordinal numbers of row vectors and column vectors;

[0125] For topological similarity matrix M pq Sum each column in the matrix and sum the results in the topological similarity matrix M. pq The sum of the i-th column is denoted as S. i S i S represents the degree of matching between two calibration points in a calibration point pair. i The larger the value, the higher the value of the calibration point relative to P. i -Q i The higher the degree of matching between two calibration points, the better the calibration point P is. i and calibration point Q i It is more likely to be the correct calibration point;

[0126] Calculate the calibration point pair P using the following formula. i -Q i The prior probability p i :

[0127]

[0128] μ = average(Si)

[0129] σ = varp(Si)

[0130] Among them, P i -Q i The calibration point pair is numbered as P. i -Q i From calibration point P i and calibration point Q i The formula consists of average(), varp(), and e, where e is the natural constant.

[0131] Calibration point P i -Q i The prior probability pi The larger the value, the higher the value of the calibration point P. i and calibration point Q i The more similar the connection relationships, i.e., the higher the neighborhood similarity (matching degree), the higher the probability that the point is the correct calibration point. σ represents the second moment, used to normalize the input parameters. When the triangular topological network of the image is relatively sparse, the calibration point is more likely to be a correct calibration point for P. i -Q i If the neighborhood similarity between two calibration points differs significantly, the value of σ will increase accordingly, and vice versa.

[0132] Step 4.2: Based on the prior probability p i Calculate the calibration point pair P using the following formula. i -Q i sampling probability p' i :

[0133]

[0134] Sampling probability p' i With prior probability p i The sampling probability p' increases as the value increases, and the sampling probability is chosen when sampling calibration points. i A larger pair of calibration points can be considered as having chosen the prior probability p. i Larger calibration points, because the accuracy of the calibration points increases with the prior probability p. i The probability increases with the increase of , therefore it is believed that by selecting the sampling probability p' i With a larger set of calibration points, the probability of selecting the correct calibration point is greater.

[0135] Step 5: Select four or seven sets of calibration point pairs with higher sampling probabilities as a sampling scheme, calculate the transformation matrix under different sampling schemes, determine the optimal transformation matrix, and use the optimal transformation matrix to stitch together the reference image and the test image.

[0136] Step 5 specifically involves:

[0137] Step 5.1: Select sampling probability p' i Larger sets of four or seven calibration points P i -Q i As a sampling scheme, based on each set of calibration points in the sampling scheme, P i -Q i The location information of the calibration point is used to establish calibration point P. i coordinate vector PR i and calibration point Q i coordinate vector QR i :

[0138] PR i =[pxi py i w] T

[0139] QR i =[qx i qy i w] T

[0140] Where, px i py i These are the calibration points P. i The coordinate values ​​of qx in the X and Y directions of the Cartesian coordinate system. i qy i The calibration points Q are respectively i The coordinate values ​​in the X and Y directions of the Cartesian coordinate system, where w is a preset constant;

[0141] Select each set of calibration points to P i -Q i Substituting the coordinate vectors of the reference point into the following formula yields a 3×3 third-order transformation matrix H:

[0142] QR i =H PR i

[0143] All calibration points Q are calculated based on the transformation matrix H. i Coordinate prediction vector QR' i :

[0144] QR' i =H PR i =[qx' i qy' i w] T

[0145] Calibration point Q i Coordinate prediction vector QR' i and coordinate vector QR i Compare and determine the calibration point relative to P. i -Q i Check if the calibration point pair is correct and remove mismatches:

[0146] If the relation is satisfied If d is the preset calibration point distance threshold, then it indicates that the calibration point is relative to P. i -Q i To match the correct calibration point pair, retain the calibration point pair P. i -Q i As a valid calibration point pair;

[0147] Otherwise, it means that the calibration point is relative to P. i-Q i To match incorrect calibration point pairs, the calibration point pair P is... i -Q i Remove;

[0148] Record the number of valid calibration point pairs obtained based on the transformation matrix H;

[0149] Step 5.2: Repeat step 5.1 multiple times to obtain the transformation matrix H under different sampling schemes. Select the transformation matrix H corresponding to the largest number of effective calibration point pairs as the optimal transformation matrix. Use the optimal transformation matrix to perform coordinate transformation on all pixels in the reference image. Then, stitch the two images together based on the coordinate-transformed reference image and the test image.

[0150] The specific method for pixel transformation and stitching two images to be stitched is as follows: multiply the optimal transformation matrix by the coordinates of each pixel in the reference image to obtain the position of each pixel in the reference image in the coordinate system of the test image. Then, the new reference image and the test image in the coordinate system of the test image are fused together to achieve the stitching of the reference image and the test image.

[0151] When sampling each pair of calibration points, the higher the prior probability of the calibration point pair, the greater the probability of it being sampled. The transformation matrix H includes the homography matrix F and the fundamental matrix. Four or seven pairs of calibration points are sampled each time, used to calculate the homography matrix F or the fundamental matrix between the two images, respectively. Through continuous sampling, the homography matrix F or the fundamental matrix with the most valid calibration point pairs is found. Generally, the homography matrix F is chosen for calculation. For images with a slightly larger perspective transformation, seven pairs of calibration points can be sampled each time to calculate the fundamental matrix between the two images.

[0152] Since the images used in this invention strictly conform to affine transformation, four sets of sampling probabilities p' are selected in this embodiment. i The largest calibration point pair P i -Q i Calculate the 3×3 homography matrix F:

[0153]

[0154] To verify the accuracy of the method of the present invention, both images in the embodiments are selected from an image comparison database. Based on the database, the correct reference homography matrix for these two images to be stitched is as follows:

[0155]

[0156] The homography matrix F obtained by the method of this invention and the error of each element of the reference homography matrix are both within 1%, indicating that the method has achieved good results. After screening using this invention, 232 valid calibration point pairs were obtained, exhibiting very high precision and recall (close to 100%). Figure 7 As shown.

[0157] This invention utilizes neighborhood topology vectors to quantitatively describe the neighborhood topology of each calibration point, and uses a topological similarity matrix to describe the similarity of the neighborhood topology between two calibration points in each pair, thereby quantifying the probability of a calibration point being a mismatch. By introducing the concepts of prior probability and sampling probability, calibration points with a high probability of mismatch are less likely to be selected in the sampling, increasing the probability of selecting the correct calibration point and thus effectively improving the efficiency and accuracy of mismatch removal.

[0158] While the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the invention. Those skilled in the art can make various modifications and refinements without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention shall be determined by the claims.

Claims

1. An image stitching and matching method based on neighborhood topological similarity and prior probability sampling, characterized in that, Includes the following steps: Step 1: Select two images that overlap in shooting space to be stitched. One of the two images is the reference image and the other is the test image. Then, extract feature points from the two images to obtain the initial feature points of each image. Use brute force matching to filter out calibration points from the initial feature points and obtain several sets of calibration point pairs. Each set of calibration point pairs contains a calibration point from the reference image and a calibration point from the test image. Step 2: Construct a triangular topology network The triangular network topology of the two images to be stitched together will be obtained by referring to the calibration points in the reference image and the calibration points in the test image respectively. Step 3: Obtain the neighborhood topology vectors of the calibration points through the triangular topology network, and form a neighborhood topology matrix by combining the neighborhood topology vectors of all calibration points on the same image to be stitched. Step 3 specifically includes: Step 3.1: Calculate the first-order neighborhood topology matrix For calibration points P i and calibration points Q i Establish their respective first-order neighborhood topological vectors PV i and first-order neighborhood topological vector QV i : PV i =[p a 1 p a 2 p a 3 … p a j …p a m ] T QV i =[q a 1 q a 2 q a 3 … q a j … q a m ] T in, P i and Q i These refer to the calibration point numbers in the reference image and the calibration point numbers in the test image, respectively. a j and q a j They are respectively first-order neighborhood topological vectors PV i Elements and first-order neighborhood topological vectors QV i The elements in i and j Both represent the ordinal numbers of the calibration points. m This indicates the number of calibration points in each image; The first-order neighborhood topological vector P is determined by the topological segments in the triangular topological network. V i Each element p in a j The possible values ​​of: when j = i When, then element p a j Set to 0; when j ≠ i And calibration points P i With calibration point P j If they are connected by only one topological line segment, then p a j Take 1; when j ≠ i And calibration points P i With calibration point P j If they are not connected by a single topological line segment, then p a j Set to 0; The first-order neighborhood topological vector Q is determined by the topological segments in the triangular topological network. V i Each element q a j The possible values ​​of: when j = i When, then element q a j Set to 0; when j ≠ i And calibration points Q i With calibration point Q j If they are connected by only one topological line segment, then q a j Take 1; when j ≠ i And calibration points Q i With calibration point Q j If they are not connected by a single topological line segment, then q a j Set to 0; The first-order neighborhood topological vectors of all calibration points in the image will be referenced. PV i Combined into a first-order neighborhood topological matrix M p The first-order neighborhood topological vectors of all calibration points in the test image are generated. QV i Combined into a first-order neighborhood topological matrix M q : M p =[ PV 1 PV 2 PV 3 … PV m ] M q =[ QV 1 QV 2 QV 3 … QV m ] Step 3.2: Calculate the second-order neighborhood topology matrix The calibration point is calculated using the following formula. P i Second-order neighborhood topological vector PV' i and calibration points Q i Second-order neighborhood topological vector QV' i : PV' i = ( E +M P ) PV i QV' i =( E +M q ) QV i Where E is an m-order unit matrix; The second-order neighborhood topological vectors of all calibration points in the reference image will be used. PV' i Combined into a second-order neighborhood topology matrix M' p The second-order neighborhood topological vectors of all calibration points in the test image are... QV' i Combined into a second-order neighborhood topology matrix M' q : M’ p =[ PV' 1 PV' 2 PV' 3 … PV' m ] M’ q =[ QV' 1 QV' 2 QV' 3 … QV' m ]； Step 4: Use the neighborhood topology matrix to obtain the prior probability of each pair of calibration points, and use the prior probability of the calibration point pairs to obtain the sampling probability of the calibration point pairs. Step 4 specifically includes: Step 4.1: Based on the second-order neighborhood topology matrix M' p and the second-order neighborhood topology matrix M' q Calculate the topological similarity matrix M using the following formula. pq : M pq (a,b)= M’ p (ap,bp)*M’ q (aq,bq) Among them, M pq (a,b) represents the topological similarity matrix M. pq The element in row a, column b of the array; M' p (ap,bp) represents the second-order neighborhood topology matrix M' p The element in row ap and column bp of the middle element; M' q (aq, bq) represents the second-order neighborhood topology matrix M' q The element in the aq-th row and bq-th column of the matrix; a and b represent the topological similarity matrix M. pq The ordinal numbers of the row vectors and column vectors; ap and bp represent the second-order neighborhood topological matrix M', respectively. p The ordinal numbers of the row vectors and column vectors are given, and aq and bq represent the second-order neighborhood topological matrix M', respectively. q The ordinal numbers of row vectors and column vectors; For topological similarity matrix M pq Sum each column in the matrix and sum the results in the topological similarity matrix M. pq The Middle i The sum of the columns is denoted as S i ; Calculate the calibration point pair using the following formula. P i - Q i Prior probability p i : ； in, P i - Q i The calibration point pair is a numbered pair. P i - Q i From the calibration point P i and calibration points Q i The formula consists of average(), varp(), and e, where e is the natural constant. Step 4.2: Based on prior probabilities p i Calculate the calibration point pair using the following formula. P i - Q i sampling probability p’ i : ； Step 5: Select four or seven sets of calibration point pairs with higher sampling probabilities as a sampling scheme, calculate the transformation matrix under different sampling schemes, determine the optimal transformation matrix, and use the optimal transformation matrix to stitch together the reference image and the test image.

2. The image stitching and matching method based on neighborhood topological similarity and prior probability sampling according to claim 1, characterized in that: The specific method for selecting calibration points and obtaining calibration point pairs through brute-force matching in step 1 is as follows: Each initial feature point in the reference image will be matched with each initial feature point in the test image using feature matching. The initial feature points that fail to match will be deleted from the two images to be stitched together, and the remaining initial feature points will be used as calibration points. The two calibration points that match successfully will be combined into a calibration point pair.

3. The image stitching and matching method based on neighborhood topological similarity and prior probability sampling according to claim 1, characterized in that: Each triangular topology network in step 2 consists of all the calibration points in the same graph and the straight line segment connecting any two calibration points in the graph. The straight line segment connecting any two calibration points in the same graph is used as the topological line segment of the triangular topology network.

4. The image stitching and matching method based on neighborhood topological similarity and prior probability sampling according to claim 1, characterized in that: Step 5 specifically includes: Step 5.1: Select sampling probability p’ i Larger sets of four or seven calibration point pairs P i - Q i As a sampling scheme, based on each set of calibration points in the sampling scheme... P i - Q i The location information of the calibration points is used to establish calibration points. P i coordinate vector PR i and calibration points Q i coordinate vector QR i : PR i =[p x i p y i w ] T QR i =[q x i q y i w ] T Where, p x i p y i These are the calibration points. P i The coordinate values ​​q in the X and Y directions of the Cartesian coordinate system. x i q y i The calibration points Q are respectively i The coordinate values ​​in the X and Y directions of the Cartesian coordinate system. w These are preset constants; Select each set of calibration points P i - Q i Substituting the coordinate vectors of the reference point into the following formula yields a 3×3 third-order transformation matrix H: QR i =H PR i All calibration points are calculated based on the transformation matrix H. Q i Coordinate prediction vector QR' i : QR' i =H PR i =[q x’ i q y’ i w ] T Calibration point Q i Coordinate prediction vector QR' i and coordinate vector QR i Compare and determine the calibration point pair P i - Q i Check if the calibration point pair is correct and remove mismatches: If the relation is satisfied ≤d, where d is a preset calibration point distance threshold, indicates that the calibration point pair P i - Q i To match the correct calibration point pairs, retain the calibration point pairs. P i - Q i As a valid calibration point pair; Otherwise, it indicates that the calibration point is correct. P i - Q i To match incorrect calibration point pairs, the calibration point pairs will be... P i - Q i Remove; Record the number of valid calibration point pairs obtained based on the transformation matrix H; Step 5.2: Repeat step 5.1 multiple times to obtain the transformation matrix H under different sampling schemes. Select the transformation matrix H corresponding to the largest number of effective calibration point pairs as the optimal transformation matrix. Use the optimal transformation matrix to perform coordinate transformation on all pixels in the reference image. Then, stitch the two images together based on the coordinate-transformed reference image and the test image.

Images