Fast image stitching method based on grid partition feature point reduction

Abstract

The invention discloses a fast image splicing method based on grid partition feature point reduction, including reading the original image to be spliced, performing denoising preprocessing on the image; extracting SIFT features of all images; purifying the inliers between matching images by RANSAC algorithm; judging the image matching relationship by the quantitative relationship between the inliers and the original feature points according to Bayesian estimation; dividing the image into dense grids by grid partitioning method, reducing the inliers in each grid; calculating the transformation matrix between any two matching images as the optimization initial value according to the reduced feature inliers; calculating the transformation matrix of each image according to the reduced inlier matching relationship by iterative optimization; and obtaining the final splicing effect image by the average value fusion method. The invention can effectively reduce the time required for splicing multiple images while ensuring the splicing quality, so that it can meet the requirements of industrial real-time splicing.

Description

Technical Field

[0001] The invention relates to the technical field of image processing, and in particular to a fast image splicing method based on grid partition feature point reduction. Background Art

[0002] Image stitching is a technology that fuses two or more narrow-angle, low-resolution images with a certain overlapping area to form a wide-angle, high-resolution image. It is widely used in practical scenarios, such as drone aerial photography, virtual reality, and medical image processing. In many types of practical scene applications, such as the actual battlefield monitoring process, it is necessary to complete image stitching in real time during drone aerial photography, return to the real battlefield scene immediately, and assist in battlefield decision-making. Therefore, fast image stitching is an extremely important requirement for practical application scenarios. The main operation steps of image stitching technology can be summarized into four parts, namely image acquisition, image preprocessing, image registration, and image fusion. The most critical step is image registration. In this step, the feature information of all images is first extracted, such as SIFT, SURF, ORB or Harris corner points, and then the registration matrix of all images is iteratively optimized and calculated based on the position information of all feature points using the Bundle Adjustment method to complete the alignment of the geometric positions of the images. However, if feature points are directly extracted, there are too many feature points on each image, the optimization matrix scale is too large during iterative optimization, and the calculation time is very serious, which cannot meet the needs of real-time stitching in practical application scenarios.

[0003] Therefore, it is necessary to design a multi-image stitching method with strong real-time performance and low computational complexity. The present invention reduces the number of directly extracted feature points based on grid partitioning, reduces the scale of optimization matrix calculation, and can still obtain better stitching effects on the basis of meeting fast stitching requirements. Summary of the invention

[0004] In order to overcome the shortcomings and deficiencies of the prior art, the present invention provides a fast image stitching method based on grid partition feature point reduction.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] A fast image stitching method based on grid partition feature point reduction, comprising:

[0007] Obtaining images to be stitched;

[0008] Extract the feature points of each image separately;

[0009] Obtain feature inliers between matching images based on the extracted feature points;

[0010] Obtain image matching pairs based on feature internal points;

[0011] The matched feature internal points are reduced by using the grid differentiation method;

[0012] Calculate the transformation matrix between each two matching images based on the reduced feature inliers;

[0013] Calculate the transformation matrix of each image based on the reduced inlier matching relationship;

[0014] According to the calculated transformation matrix of each image, the relative positions of all images are obtained, and the final spliced ​​image is obtained through the image fusion step.

[0015] Furthermore, the step of obtaining the images to be stitched further includes preprocessing the images to be stitched, and the preprocessing includes removing noise.

[0016] Further, the method of obtaining the feature inliers between the matching images according to the extracted feature points is specifically as follows:

[0017] Directly match the feature points between the two images, purify the directly matched feature points through the RANSAC algorithm, and obtain the correctly matched feature inliers. Suppose the set of directly matched feature point pairs is V, where the number of feature point matching pairs is m;

[0018] Randomly select q pairs of matching feature points from the set V and further calculate the homography transformation matrix H;

[0019] Calculate the number k of feature points that satisfy the transformation matrix H in (mq);

[0020] Repeat the above steps n times, and select the feature matching point with the largest number k as the final feature inlier point.

[0021] Furthermore, the image matching pairs are obtained according to the feature inner points. Specifically, whether any two images match is determined by Bayesian estimation to obtain all image matching pairs. The determination process is as follows:

[0022] The total number of feature matching pairs is m, and the number of feature inliers obtained after purification by the RANSAC algorithm is n i , if n i >8+0.3·m, the two images can be judged to match.

[0023] Furthermore, the matched feature interior points are reduced by the grid differentiation method, specifically:

[0024] Divide the matching image into C1×C2 grids;

[0025] Calculate the number of feature points in each grid;

[0026] Use the k-means clustering algorithm to cluster the feature points in each grid. If the number of inliers in the grid exceeds 1 / 1 of the total feature points, the clustering is divided into 2 categories. If it is less than 1 / 1 of the total feature points, the clustering is divided into 4 categories.

[0027] The inliers of the clusters in the grid were sampled, and 60% of the inliers in each class were randomly selected;

[0028] Delete the remaining feature inliers.

[0029] Furthermore, the transformation matrix between each two matching images is calculated based on the reduced feature interior points, specifically:

[0030] Assume that a pair of feature inliers in the two matching images are u1=[x1 y1] T and u2=[x2 y2] T , the transformation matrix adopts the projective transformation, and the calculation method is as follows:

[0031]

[0032]

[0033] Where N is the number of SIFT inliers extracted from the image, and the matrix A is the matrix calculated based on all the inliers. i The matrix size is 2N×9, and the singular value decomposition algorithm is used for calculation. The final matrix h to be solved is the smallest right singular vector of matrix A.

[0034] Furthermore, the transformation matrix of each image is calculated according to the reduced interior point matching relationship, specifically:

[0035] The error between two matching images i and j is defined as the sum of the distances of all feature inliers after reduction after transformation by the transformation matrix. The calculation method is as follows:

[0036]

[0037] in, and are the kth matching features in images i, j, respectively. H represents all SIFT internal points of image i, j after being purified by the RANSAC algorithm. ij represents the homography transformation matrix between images i, j. The cumulative error of all images is the sum of the distances between the reduced feature inliers of all images and their matching images after passing through the transformation matrix. The calculation method is as follows:

[0038]

[0039] Where I(i) represents all images that correctly match image i, and then it is iteratively optimized according to the Levenberg-Marquardt algorithm to calculate all transformation matrices H ij .

[0040] Furthermore, the average value fusion method is used to achieve image fusion.

[0041] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the image fast stitching method when executing the computer program.

[0042] A storage medium stores a computer program, wherein the computer program implements any one of the methods for rapid image stitching when executed by a processor.

[0043] Beneficial effects of the present invention:

[0044] (1) The method proposed in the present invention uses grid partitioning to reduce feature points and then calculates the transformation matrix of each image in the image registration process. Compared with the method of directly using the original feature points to calculate the transformation matrix, the scale of the iterative optimization matrix can be reduced, the amount of calculation can be greatly reduced, the stitching process can be accelerated, the stitching time can be shortened, and real-time stitching can be achieved.

[0045] (2) The present invention reduces the number of feature points by means of grid partitioning, and does not reduce the feature points by means of randomly removing certain feature points. This can avoid the situation where the registration error increases due to random removal of feature points, and ensure that the stitching effect is similar to the stitching effect using the original feature points. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a work flow chart of the present invention;

[0047] Figure 2(a) is the original image to be stitched, Figure 2(b) is the stitching effect diagram using the directly extracted feature points to calculate the transformation matrix, Figure 2(c) is the stitching effect diagram using the transformation matrix calculated after randomly removing feature points, and Figure 2(d) is the stitching effect diagram after processing using the method described in this embodiment.

[0048] Figure 3 It is a schematic diagram of the feature point reduction method of the present invention. DETAILED DESCRIPTION

[0049] The present invention will be further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0050] Example 1

[0051] A fast image stitching method based on grid-differentiated feature point reduction is proposed. In order to speed up the stitching speed of multiple images while obtaining a better stitching effect and meet the needs of actual industrial applications, the method includes extracting feature points in all images to be stitched, dividing the feature points in the image into multiple regions according to the grid partitioning method, randomly selecting 60% of the feature points in each region for retention, and removing the remaining feature points, so that the optimization matrix size in the iterative optimization process is greatly reduced and the stitching speed is greatly accelerated. At the same time, due to the use of the grid partitioning method, it is ensured that the feature points after reduction can still maintain the original feature point position distribution more evenly, and the feature point position distribution will not be abnormal due to the random removal of feature points, thereby increasing the registration error. This method ensures the stitching effect while greatly reducing the time required for stitching multiple images.

[0052] Flowchart as Figure 1 As shown, specifically:

[0053] S1 reads all the original images to be stitched, preprocesses the original images, and removes the noise;

[0054] S2 extracts and saves the SIFT feature points of each image respectively;

[0055] S3 directly matches the feature points between every two images, and purifies the directly matched feature points through the RANSAC (Random sample consensus) algorithm, removes the incorrectly matched feature points, and retains the correctly matched feature points. Assuming that the feature point pairs for direct matching are known, they are symbolically represented as a feature point pair set V, where the number of feature point matching pairs is m. The specific steps are as follows:

[0056] S3.1 arbitrarily select q pairs of matching feature points from the set V;

[0057] S3.2 calculates the homography transformation matrix H based on the selected q pairs of matching feature points;

[0058] S3.3 calculates the number k of the remaining (mq) pairs of feature points that satisfy the transformation matrix H;

[0059] S3.4 Repeat the above steps n times, and select the matching feature point with the largest k value as the final inlier point.

[0060] S4 obtains image matching pairs based on feature inliers, specifically:

[0061] Through Bayesian estimation, we determine whether any two images match based on the extracted feature point information, and obtain all image matching pairs. The determination method is as follows:

[0062] Assume that the total number of SIFT feature matching pairs directly extracted is m, and the number of inliers obtained after purification by the RANSAC algorithm is n. i If n i >8+0.3·n f , we can judge that the two images match;

[0063] S5 reduces the matched feature internal points by using the grid differentiation method, such as Figure 3 As shown, specifically, the matching image is divided into C1×C2 grids;

[0064] Calculate the number of feature points in each grid;

[0065] Use the k-means clustering algorithm to cluster the feature points in each grid. If the number of inliers in the grid exceeds 1 / C1 of the total feature points, the clustering is divided into 2 categories. If it is less than 1 / C1 of the total feature points, the clustering is divided into 4 categories.

[0066] The inliers of the clusters in the grid were sampled, and 60% of the inliers in each class were randomly selected;

[0067] Delete the remaining feature inliers.

[0068] In this step, the internal point selection is 60%, which reduces the amount of calculation and achieves the best stitching effect.

[0069] S6 calculates the transformation matrix between every two matching images based on the reduced feature inliers, providing an initial value for the iterative optimization of the BundleAdjustment method. The specific calculation steps are as follows:

[0070] Assume that one pair of inner points in the two matching images is u1=[x1 y1] T and u2=[x2 y2] T , the transformation matrix adopts the projective transformation, and the calculation method is as follows:

[0071]

[0072]

[0073] Where N is the number of SIFT inliers extracted from the image, and the matrix A is the matrix calculated based on all the inliers. i The matrix size is 2N × 9. The singular value decomposition algorithm can be used for calculation here, and the final matrix h to be solved is the smallest right singular vector of matrix A;

[0074] S7 iteratively optimizes the transformation matrix of each image by combining the interior point pairs between all matching images according to the reduced interior point matching relationship. The specific implementation steps are as follows:

[0075] The error between two correctly matched images i and j is defined as the sum of the distances of all feature inliers after reduction after transformation by the transformation matrix, and is calculated as follows:

[0076]

[0077] in, and are the kth matching features in images i, j, respectively. H represents all SIFT internal points of image i, j after being purified by the RANSAC algorithm. ij Represents the homography transformation matrix between images i, j. The cumulative error of all images is the sum of the distances between all images and their matching images after the reduced feature inliers pass through the transformation matrix. The calculation method is as follows:

[0078]

[0079] Where I(i) represents all images that correctly match image i. Then, the Levenberg-Marquardt algorithm is used to iteratively optimize and calculate all transformation matrices H ij ;

[0080] S8 calculates the relative positions of all images according to the calculated transformation matrix, realizes image fusion according to the average value fusion algorithm, and obtains the final spliced ​​image.

[0081] Figure 2(a) is the original image to be stitched, Figure 2(b) is the stitching effect diagram of the transformation matrix calculated by directly extracting feature points, Figure 2(c) is the stitching effect diagram of the transformation matrix calculated by randomly removing feature points, and Figure 2(d) is the stitching effect diagram after being processed by the method described in this embodiment. The method of directly using the extracted feature points to calculate the transformation matrix has the smallest registration error and the best image stitching quality, but the running time is very long and cannot meet the actual industrial application requirements; the method of randomly removing feature points to calculate the transformation matrix has a shorter running time, but the registration error is large, there is obvious ghosting and blurring, and the stitching quality is poor; the method proposed in this embodiment has a better stitching effect, a smaller image registration error, and the time required is significantly reduced compared to the method of using all feature points for calculation, indicating that this embodiment is more in line with actual application requirements than the existing algorithm.

[0082] Example 2

[0083] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the image fast stitching method when executing the computer program.

[0084] The image rapid splicing method comprises:

[0085] A fast image stitching method based on grid partition feature point reduction, comprising:

[0086] Obtaining images to be stitched;

[0087] Extract the feature points of each image separately;

[0088] Obtain feature inliers between matching images based on the extracted feature points;

[0089] Obtain image matching pairs based on feature internal points;

[0090] The matched feature internal points are reduced by using the grid differentiation method;

[0091] Calculate the transformation matrix between each two matching images based on the reduced feature inliers;

[0092] Calculate the transformation matrix of each image based on the reduced inlier matching relationship;

[0093] According to the calculated transformation matrix of each image, the relative positions of all images are obtained, and the final spliced ​​image is obtained through the image fusion step.

[0094] Example 3

[0095] A storage medium stores a computer program, wherein the computer program implements any one of the methods for rapid image stitching when executed by a processor.

[0096] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A fast image stitching method based on grid partition feature point reduction, characterized in that: include: Obtaining images to be stitched; Extract the feature points of each image separately; Obtain feature inliers between matching images based on the extracted feature points; Obtain image matching pairs based on feature internal points; The matched feature internal points are reduced by using the grid differentiation method; Calculate the transformation matrix between each two matching images based on the reduced feature inliers; Calculate the transformation matrix of each image based on the reduced inlier matching relationship; According to the calculated transformation matrix of each image, the relative positions of all images are obtained, and the final spliced ​​image is obtained through the image fusion step; The grid differentiation method is used to reduce the matched feature internal points, specifically: Divide the matching image into C1×C2 grids; Calculate the number of feature points in each grid; Use the k-means clustering algorithm to cluster the feature points in each grid. If the number of inliers in the grid exceeds 1 / C1 of the total feature points, the clustering is divided into 2 categories. If it is less than 1 / C1 of the total feature points, the clustering is divided into 4 categories. The inliers of the clusters in the grid were sampled, and 60% of the inliers in each class were randomly selected; Delete the remaining feature internal points; The transformation matrix between each two matching images is calculated based on the reduced feature interior points, specifically: Assume that a pair of feature inliers in the two matching images are u1=[x1y1] T and u2=[x2y2] T , the transformation matrix adopts the projective transformation, and the calculation method is as follows: Where N is the number of SIFT inliers extracted from the image, and the matrix A is the matrix calculated based on all the inliers. i The matrix size is 2N×9, and the singular value decomposition algorithm is used for calculation. The final matrix to be solved is That is, the smallest right singular vector of matrix A; The transformation matrix of each picture is calculated according to the reduced interior point matching relationship, specifically: The error between two matching images i and j is defined as the sum of the distances of all feature inliers after reduction after transformation by the transformation matrix. The calculation method is as follows: in, and are the kth matching features in images i and j respectively, H represents all SIFT internal points of image i, j after being purified by the RANSAC algorithm. ij Represents the homography transformation matrix between images i and j. The cumulative error of all images is the sum of the distances between the reduced feature inliers of all images and their matching images after passing through the transformation matrix. The calculation method is as follows: Where I(i) represents all images that correctly match image i, and then it is iteratively optimized according to the Levenberg-Marquardt algorithm to calculate all transformation matrices H ij .

2. The method for rapid image stitching according to claim 1, characterized in that: The step of obtaining the images to be stitched further includes preprocessing the images to be stitched, wherein the preprocessing includes removing noise.

3. The method for rapid image stitching according to claim 1, characterized in that: The method of obtaining the feature inliers between the matching images according to the extracted feature points is specifically as follows: Directly match the feature points between the two images, purify the directly matched feature points through the RANSAC algorithm, and obtain the correctly matched feature inliers. Suppose the set of directly matched feature point pairs is V, where the number of feature point matching pairs is m; Randomly select q pairs of matching feature points from the set V and further calculate the homography transformation matrix H; Calculate the number k of feature points that satisfy the transformation matrix H in (mq); Repeat the above steps n times, and select the feature matching point with the largest number k as the final feature inlier point.

4. The method for rapid image stitching according to claim 1, characterized in that: The image matching pairs are obtained based on the feature inner points. Specifically, whether any two images match is determined by Bayesian estimation to obtain all image matching pairs. The judgment process is as follows: The total number of feature matching pairs is m, and the number of feature inliers obtained after purification by the RANSAC algorithm is n i , if n i >8+0.3·m, the two images can be judged to match.

5. The method for rapid image stitching according to claim 1, characterized in that: The average value fusion method is used to achieve image fusion.

6. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for rapid image stitching according to any one of claims 1 to 5 is implemented.

7. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for rapid image stitching described in any one of claims 1 to 5 is implemented.

Images