An image stitching method

Abstract

The present invention discloses an image stitching method, which includes the following steps: using SURF to provide feature points of a reference image and an image to be stitched; using knn to pair the feature points of the reference image and the feature points of the image to be stitched; filtering out incorrect matching items in the matching feature points according to the relative distance of knn; using the improved RANSAC method to calculate a projection matrix; projecting the matrix to be stitched onto the reference image plane to complete image fusion. The method of the present invention improves the binary weight accumulation method of the traditional RANSAC algorithm to multiplying the original weight by the inlier distribution area, which can effectively improve the robustness problem of image stitching by drones in areas without sufficient feature points such as grasslands and rivers.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an improved image stitching method. Background Art

[0002] Image stitching is a method of combining multiple overlapping images into a larger image. It is of great significance in medical imaging, computer vision, satellite data, automatic target recognition, and other fields. The output of image stitching is the union of the two input images.

[0003] However, existing image stitching methods have the following disadvantages when used for UAV aerial photography:

[0004] (1) It is impossible to extract effective feature points from specific scenes photographed by drones, such as grasslands. There are too few feature points or the feature points are too concentrated to effectively reflect the image information.

[0005] (2) Small parts of the images captured by drones are prone to abnormalities such as blurring, distortion, or tearing.

[0006] (3) The number of images captured by the drone is large, and repeated stitching operations will accumulate errors, causing image drift. Summary of the Invention

[0007] In view of this, the present invention proposes an improved image stitching method, aiming to solve the above-mentioned problem of poor stitching effect when the feature points of the image are not uniform.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is:

[0009] An image stitching method comprises the following steps:

[0010] (1) Use the SURF method to extract feature points of the reference image and the image to be stitched;

[0011] (2) Use the K nearest neighbor method to pair the feature points of the reference image with the feature points of the image to be stitched;

[0012] (3) Filter out the incorrect matching items in the matching feature points based on the relative distance obtained by the K nearest neighbor method to obtain the matching point set;

[0013] (4) Calculate the projection matrix using the improved RANSAC method;

[0014] (5) Copy the calculated projection matrix to the reference image plane to complete image fusion.

[0015] Furthermore, the step (1) specifically includes the following steps:

[0016] (1.1) Calculate a Hessian matrix for each pixel point. When the discriminant of the Hessian matrix obtains a local maximum value, determine that the current point is a point that is brighter or darker than other points in the surrounding neighborhood, thereby locating the position of the key point;

[0017] (1.2) Compare each pixel point processed by the Hessian matrix with the points in the two-dimensional image space and the scale space neighborhood. Locate the key points from the positions of the key points, and then filter out the key points with energy lower than the threshold to screen out the final stable key points, that is, feature points.

[0018] Further, the step (2) specifically includes the following steps:

[0019] (2.1) Calculate the distance between the points in the reference image feature point dataset and the midpoint of the feature points of the image to be stitched;

[0020] (2.2) Record the pairing situation and distance between the feature points.

[0021] Further, the step (3) specifically includes the following steps:

[0022] For the one-to-many situation existing in the feature point matching calculated by the K-nearest neighbor method, compare the distance of each matching point calculated by the K-nearest neighbor method with the distance of the midpoint described in step (2.1), and screen out the distance with the smallest difference to obtain the matching point set.

[0023] Further, the step (4) specifically includes the following steps:

[0024] (4.1) Randomly extract multiple sample data from the matching point set. The extracted samples are not collinear. Use the RANSAC method to calculate the transformation matrix H, and record the transformation matrix H as the model M;

[0025] (4.2) Calculate the projection error of all data in the reference image feature point dataset and the model M, and add the data with an error less than the threshold to the inlier set I;

[0026] (4.3) If the product of the number of elements N of the current inlier set I and the area R of the polygon formed by the inliers on the reference image is greater than the optimal inlier set description value A, update A = N * R, and update the optimal inlier set Best_I to the inlier set I. The initial value of the optimal inlier set description value A is 0;

[0027] (4.4) Repeat steps (4.1), (4.2), and (4.3) until the number of iterations reaches the preset value.

[0028] The beneficial effects of the present invention are as follows:

[0029] 1. The present invention converts whether the inlier distribution is uniform into the solution of the area of the polygon formed by the inliers, thus not only referring to the number of inliers but also considering the problem of over-concentration of inliers, making the result of image stitching more robust.

[0030] 2. The present invention improves the binary weight accumulation method of the traditional RANSAC algorithm to multiply the original weight by the inlier distribution area. By this method, the problem of image stitching for drones in areas without enough feature points such as grasslands and rivers can be effectively solved.

[0031] 3. Existing image stitching based on feature point methods requires a large overlapping area between two adjacent images to obtain a large number of feature matches, uses a large number of redundant observations to improve the reliability of the orientation parameter solution and presents errors to ensure the quality of image stitching. However, when there are few feature points or the feature points are too concentrated in the image, there will be a large error after pairing the feature points, and the image information cannot be effectively reflected. The present invention can effectively solve the above problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is the overall framework diagram of the method of the embodiment of the present invention.

[0033] Figure 2 is the existing image stitching flowchart.

[0034] Figure 3 is the existing RANSAC method flowchart.

[0035] Figure 4 is the flowchart of the improved RANSAC method in the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0036] The present invention will be further described below in conjunction with the drawings and specific embodiments.

[0037] As Figure 1 shown, an image stitching method includes the following steps:

[0038] (1) Process the image processing of various parameters from different models of drones into a unified format through a preprocessing factory.

[0039] (2) Process each frame of the image coming out of the preprocessing factory through the front end.

[0040] (3) Correct the errors between and within the image sequences through the back end. To achieve error correction in both cases of between-image sequences and within-image sequences, we can borrow the method of BA (Bundle Adjustment) correction, and perform BA correction between image sequences (seq - between) and within image sequences (seq - within) for all pictures.

[0041] (4) The observer immediately generates multiple preview images based on the results of the backend.

[0042] Wherein, step (1) specifically includes the following steps:

[0043] The preprocessing factory preprocesses the data and finally constructs the output into a unified format, that is, the smallest unit of drone output - image frame - is generated in factory mode.

[0044] Step (2) specifically includes the following steps:

[0045] (2.1) The front end mainly completes the processing of each frame of the image, including calculating the image feature points and the rigid transformation between two frames.

[0046] (2.2) Discard sporadic abnormal frames to ensure the correctness of the splicing results.

[0047] (2.3) The non-convergent discontinuity points are extracted and a separate image sequence (seq) is generated, in the hope that the back-end will obtain the result with the minimum error according to the least squares method.

[0048] Step (3) specifically includes the following steps:

[0049] (3.1) When performing BA correction between image sequences (between seq), the backend can selectively input image sequences whose errors have not been correctly corrected, and input a rigid transformation as the correction variable.

[0050] (3.2) When performing BA correction within an image sequence (within seq), the image sequence is split into N frames and then input into the BA calculation module.

[0051] (3.3) After completing the correction, the backend will notify the observer to generate a new preview.

[0052] Step (4) specifically includes the following steps:

[0053] (4.1) The observer listens for notifications from the backend.

[0054] (4.2) After the observer receives the notification, it can use the observer pattern: when the state of the observed object changes, it can notify through broadcasting, so that all objects that depend on it (observer objects) will trigger corresponding actions to update the preview image.

[0055] (4.3) After receiving the notification, the observer will obtain the optimization information of the backend and generate an image for preview based on the information.

[0056] Figure 2The following shows an existing image stitching method. In this solution, the image stitching method mainly includes the following steps:

[0057] (1) Use SURF to provide the feature points of the reference image and the image to be stitched.

[0058] Specifically, it includes the following steps:

[0059] (1.1) Calculate a Hessian matrix for each pixel point. When the discriminant of the Hessian matrix obtains a local maximum value, it is determined that the current point is a point that is brighter or darker than other points in the surrounding neighborhood, thereby locating the position of the key point.

[0060] (1.2) Compare each pixel point processed by the Hessian matrix with the points in the two-dimensional image space and the scale space neighborhood, initially locate the key points, and then filter out the key points with weak energy and the mislocated key points to screen out the final stable feature points.

[0061] (2) Use knn to pair the feature points of the reference image and the feature points of the image to be stitched.

[0062] Specifically, it includes the following steps:

[0063] (2.1) Calculate the distance between the points in the reference image feature point dataset and the points in the image to be stitched feature points.

[0064] (2.2) Record the pairing situation and distance between the feature points.

[0065] (3) Filter out the incorrect matching items in the matching feature points according to the relative distance of knn.

[0066] Specifically, it includes the following steps:

[0067] (3.1) The feature point matching calculated by the knn algorithm may have a one-to-many situation. At this time, it needs to be preliminarily screened.

[0068] (4) Use the RANSAC method to calculate the projection matrix.

[0069] Specifically, it includes the following steps:

[0070] (4.1) Randomly draw multiple sample data from the dataset (the drawn samples cannot be collinear), calculate the transformation matrix H, denoted as model M.

[0071] (4.2) Calculate the projection error between all the data in the dataset and model M. If the error is less than the threshold, add it to the inlier set I.

[0072] (4.3) If the number of elements in the current inlier set I is greater than the optimal inlier set Best_I, then update Best_I = I, and at the same time update the iteration count k.

[0073] (4.4) If the iteration count is greater than k, then exit; otherwise increment the iteration count by 1 and repeat the above steps.

[0074] (5) Project the matrix to be stitched onto the reference image plane to complete image fusion.

[0075] Specifically, it includes the following steps:

[0076] (5.1) Use the set of matching points obtained through the above steps to find the transformation matrix.

[0077] (5.2) Perform image registration through the transformation matrix.

[0078] Copy the image to be stitched onto the registered image.

[0079] Figure 3 、 Figure 4 The flow shown above improves the RANSAC method on the basis of the original image stitching scheme. By modifying the method for judging the optimal solution, it converts whether the inliers are evenly distributed into the solution of the area of the polygon formed by the inliers, so that both the number of inliers and the problem of over-concentration of inliers are considered in the evaluation formula, making the result of image stitching more robust.

[0080] Specifically, the specific steps of the improved RANSAC method are as follows:

[0081] (1) Randomly draw multiple sample data from the dataset (the drawn samples cannot be collinear), calculate the transformation matrix H, denoted as model M.

[0082] (2) Calculate the projection error between all data in the dataset and model M. If the error is less than the threshold, add it to the inlier set I. And calculate the area R of the polygon formed by the inliers on the reference image.

[0083] (3) If the product of the number of elements in the current inlier set I and the area R of the polygon formed by the inliers on the reference image is greater than the optimal inlier set Best_I, then update Best_I = I * R, and at the same time update the iteration count k.

[0084] (4) If the iteration count is greater than k, then exit; otherwise increment the iteration count by 1 and repeat the above steps.

[0085] In summary, the present invention improves the binary weight accumulation method of the traditional RANSAC algorithm to multiply the original weight by the inlier distribution area. Through this method, the robustness problem of image stitching of unmanned aerial vehicles in areas without sufficient feature points such as grasslands and rivers can be efficiently improved.

[0086] It should be noted that the above description of the disclosed embodiments is intended to enable those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined in the present invention can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown above, but will conform to the widest scope consistent with the principles and novel features disclosed in the present invention. For those of ordinary skill in the art, there will be changes in the specific implementation manners and application scopes according to the idea of the present invention. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. An image stitching method, characterized in that, It includes the following steps: (1) Use the SURF method to extract the feature points of the reference image and the image to be stitched; (2) Use the K-nearest neighbor method to pair the feature points of the reference image with those of the image to be stitched; (3) Filter out the incorrect matches in the matching feature points according to the relative distances obtained by the K-nearest neighbor method to obtain a set of matching points; (4) Use the improved RANSAC method to calculate the projection matrix; Specifically, it includes the following steps: (4.1) Randomly extract multiple sample data from the set of matching points. The extracted samples are not collinear. Use the RANSAC method to calculate the transformation matrix H and denote the transformation matrix H as the model M; (4.2) Calculate the projection error of all data in the reference image feature point dataset with respect to the model M. Add the data with errors less than the threshold to the inlier set I; (4.3) If the product of the number of elements N in the current inlier set I and the area R of the polygon formed by the inliers on the reference image is greater than the optimal inlier set description value A, then update A = N * R and update the optimal inlier set Best_I to the inlier set I. The initial value of the optimal inlier set description value A is 0; (4.4) Repeat steps (4.1), (4.2), and (4.3) until the number of iterations reaches the preset value; (5) Copy the calculated projection matrix to the reference image plane to complete image fusion.

2. The image stitching method according to claim 1, characterized in that The specific steps of step (1) include the following: (1.1) Calculate a Hessian matrix for each pixel point. When the discriminant of the Hessian matrix obtains a local maximum value, it is determined that the current point is a point that is brighter or darker than other points in the surrounding neighborhood, thereby locating the position of the key point; (1.2) Compare each pixel point processed by the Hessian matrix with the points in the two-dimensional image space and scale space neighborhoods. Locate the key points from the positions of the key points, and then filter out the key points with energy lower than the threshold to screen out the final stable key points, that is, feature points.

3. The image stitching method according to claim 2, wherein The specific steps of step (2) include the following: (2.1) Calculate the distance from the points in the reference image feature point dataset to the midpoint of the feature points of the image to be stitched; (2.2) Record the pairing situation and distance between the feature points.

4. The image stitching method according to claim 3, characterized in that, The specific steps of step (3) include the following: For the one-to-many situation existing in the feature point matching calculated by the K-nearest neighbor method, compare the distance of each matching point calculated by the K-nearest neighbor method with the distance to the midpoint described in step (2.1), and screen out the distance with the smallest gap to obtain a set of matching points.

Images