Image splicing method based on regional hierarchical feature matching and improved SIFT

Abstract

The application discloses an image splicing method based on regional hierarchical feature matching and improved SIFT, and the method comprises the following steps: firstly, changing the regional hierarchical random K-D tree feature matching means for the slow SIFT algorithm; secondly, using the RANSAC algorithm for the mismatched points of the SIFT algorithm; finally, using the two means of pretreatment and selecting the main image for the distortion generated by the multi-image splicing of the SIFT algorithm. The image splicing method based on regional hierarchical feature matching and improved SIFT can better optimize the problems of slow SIFT algorithm, mismatched points and distortion generated by multi-image splicing, and has certain practical significance.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to an image splicing method based on regional hierarchical feature matching and improved SIFT. BACKGROUND

[0002] Image splicing refers to seamlessly combining a sequence of spatially continuous images together to generate a panoramic image with wide coverage of a scene. In the process of image acquisition, a single image often cannot display global information and cannot cover the entire area required. Therefore, the main goal of image splicing is to complete the acquisition of complete image information and achieve the purpose of fusing multiple images into one image. The effect of splicing is the basis for subsequent target detection and directly affects the quality and effect of target extraction. At present, there are many studies on image splicing technology, but due to the problems such as illumination variation, motion overlap, noise generated in the acquisition process, distortion, occlusion and the like of the acquired images, the current image splicing often cannot achieve the desired effect.

[0003] The previous image splicing generally uses the SIFT algorithm. The traditional SIFT algorithm has the characteristics of high precision and good stability, and can often achieve good results in the splicing of two images. However, with the development of the times and the increasing requirements of aerial image splicing, simple two-image splicing has long been unable to meet people's demand for digital image splicing. Emerging fields such as unmanned aerial vehicle aerial photography have put forward higher requirements for image splicing algorithms. Under such background, the SIFT algorithm gradually exposes shortcomings such as slow speed, existence of error matching points, and distortion in multi-image splicing. Some of these shortcomings of SIFT can be solved by using relatively simple means and methods. SUMMARY

[0004] The purpose of the present application is to provide an image splicing method based on regional hierarchical feature matching and improved SIFT, which solves the problem of insufficient matching speed and matching precision in current image splicing.

[0005] The technical solution adopted by the present application is as follows:

[0006] The image splicing method based on regional hierarchical feature matching and improved SIFT is implemented according to the following steps:

[0007] Step 1: using a Gaussian pyramid to construct a scale space of an input original image;

[0008] Step 2: performing extreme value detection on the scale space constructed in step 1, and locating feature points in the space according to the detection result;

[0009] Step 3: performing multi-image matching according to the feature points determined in step 2 to obtain a feature matching set;

[0010] Step 4, the unit matrix is solved by feature matching set calculation, and the multi-image image splicing is realized.

[0011] The application also has the characteristics of;

[0012] Step 1 is implemented according to the following steps:

[0013] Step 1.1, select the main image and convert it into a gray image for splicing;

[0014] Step 1.2, convert the input original image into a gray image;

[0015] Step 1.3, use Gaussian smoothing to process the image to reduce noise;

[0016] Step 1.4, the image at this time is sampled by a Gaussian convolution kernel of different scales to obtain a series of images of different sizes, and Gaussian filtering is added in the process of down-sampling.

[0017] Step 2 is implemented according to the following steps:

[0018] Step 2.1, difference between two layers of the Gaussian pyramid to obtain a DoG pyramid, the pixel value change on the image can be obtained through the Gaussian difference function, thereby obtaining potential feature points;

[0019] Step 2.2, traverse each pixel point in the image, compare it with 8 adjacent pixel points in the layer and 9+9 pixel points in the adjacent upper and lower layers, if it is a local extreme point, it is reserved as a candidate point of the feature point;

[0020] Step 2.3, the Taylor series expansion is carried out at the detected extreme point to fit the three-dimensional quadratic function, and the extreme value of the extreme point is calculated; when positioning the feature point, if the feature point is at the edge of the image, the principal curvatures of the points in the edge direction and the principal curvatures of the points in the direction perpendicular to the edge need to be detected, and the ratio of the two is used for screening, and the extreme points obtained through the above steps that meet the requirements are used as the feature points of the image.

[0021] Step 3 is as follows:

[0022] Step 3.1, for the feature points in the DoG pyramid, the gradient and direction features of the pixels in the circular neighborhood window of the pyramid image are calculated, and the modulus and direction are counted;

[0023] Step 3.2, determine the main direction of the feature point according to the statistical value, calculate the gradient information of 8 directions in the corresponding neighborhood according to the 4x4 template, establish a descriptor to describe the feature point, and a total of 8x4x4-dimensional feature descriptor is obtained, and the feature descriptor is normalized to obtain a feature vector;

[0024] Step 3.3, a K nearest neighbor search based on an improved random K-D tree, i.e. a fast nearest neighbor approximation search function library, is used to find the matching point with the shortest Euclidean distance in another image for each feature point in an image;

[0025] Step 3.4, a random sample consensus algorithm is used to remove the false matching points by dynamically adjusting the ratio of the nearest neighbor and the second nearest neighbor, so that a more perfect feature matching set is obtained.

[0026] Step 4 is specifically as follows:

[0027] Step 4.1, a homography matrix is calculated and solved through the feature matching set of the two images;

[0028] Step 4.2, all pixel points in one of the images are subjected to homographic transformation through the obtained homography matrix, and the other image is directly copied to the corresponding position to realize image stitching.

[0029] Step 4.3, after the stitching of the two images is realized, multi-image stitching is further realized, the main image is selected for stitching, and the image is subjected to cylindrical projection before stitching to pre-process and suppress distortion.

[0030] The beneficial effects of the present application are that, compared with the existing SIFT image stitching method, the present application, based on the image stitching method of regional hierarchical feature matching and improved SIFT, according to the similarity of the spatial position, aggregates the features through hierarchical classification, divides different regions, and uses the fast nearest neighbor approximation search function library of the random K-D tree to replace the brute force matching method to improve the matching speed, performs feature matching, then flexibly screens the features through dynamic threshold adjustment, uses the random sample consensus algorithm (RANSAC) to remove the false matching, reduces the matching number, and improves the matching accuracy. Finally, the image is pre-processed through cylindrical projection before stitching to achieve the purpose of suppressing distortion. The effectiveness of the present application is verified through the comparison between the classical SIFT algorithm and the improved SIFT algorithm and the multi-image stitching experiment. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 is a flowchart of the image stitching method based on regional hierarchical feature matching and improved SIFT of the present application;

[0032] Figure 2 is a schematic diagram of a Gaussian pyramid model for constructing an image scale space in the image stitching method based on regional hierarchical feature matching and improved SIFT of the present application;

[0033] Figure 3 is a schematic diagram of a Gaussian difference pyramid model obtained by pairwise difference between two layers of the Gaussian pyramid in the image stitching method based on regional hierarchical feature matching and improved SIFT of the present application;

[0034] Figure 4 is the relationship diagram of the relationship between the detected extreme points and the real extreme points in the image splicing method based on regional hierarchical feature matching and improved SIFT of the application;

[0035] Figure 5 is the schematic diagram of the feature point direction statistical histogram in the image splicing method based on regional hierarchical feature matching and improved SIFT of the application, and the direction of the peak value of the histogram is selected as the main direction;

[0036] Figure 6 is the feature descriptor gradient statistical histogram in the image splicing method based on regional hierarchical feature matching and improved SIFT of the application;

[0037] Figure 7 is the schematic diagram of the improved random sequence K-D tree construction process in the image splicing method based on regional hierarchical feature matching and improved SIFT of the application;

[0038] Figure 8 is the schematic diagram of the image taken in the campus and having deformation in the image splicing method based on regional hierarchical feature matching and improved SIFT of the application;

[0039] Figure 9 is the schematic diagram of the feature points in the two images taken due to some error matching points caused by occlusion and the like in the image splicing method based on regional hierarchical feature matching and improved SIFT of the application;

[0040] Figure 10 is the aerial photograph original image and the matching result image under different precisions in the image splicing method based on regional hierarchical feature matching and improved SIFT of the application;

[0041] Figure 11 is the comparison image processed by the RANSAC algorithm and not processed by the RANSAC algorithm in the image splicing method based on regional hierarchical feature matching and improved SIFT of the application;

[0042] Figure 12 is the experimental effect diagram of a group of campus shooting images in the image splicing method based on regional hierarchical feature matching and improved SIFT of the application;

[0043] Figure 13 is the image splicing method based on regional hierarchical feature matching and improved SIFT of the application Figure 12 a group of splicing effect images obtained by preprocessing the original image;

[0044] Figure 14This is a comparison image of the effect of processing the seam image generated by the traditional SIFT algorithm by image weighted fusion in the image stitching method based on regional hierarchical feature matching and improved SIFT of this invention;

[0045] Figure 15 This invention provides a panoramic image of mountain ranges stitched together using a region-level feature matching and improved SIFT image stitching method.

[0046] Figure 16 This invention provides a panoramic image stitched from a set of aerial images using a region-level feature matching and improved SIFT image stitching method.

[0047] Figure 17 This is a sharpening effect image after image stitching in the image stitching method based on region-level feature matching and improved SIFT of this invention;

[0048] Figure 18 The image stitching method based on region-level feature matching and improved SIFT in this invention is shown in the comparison chart of algorithm speed before and after the improvement. Detailed Implementation

[0049] The image stitching method based on regional hierarchical feature matching and improved SIFT (Scale-invariant feature transform) of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0050] This invention presents an image stitching method based on region-level feature matching and improved SIFT, the flowchart of which is shown below. Figure 1 As shown, please follow these steps:

[0051] Step 1: Use the Gaussian pyramid as follows Figure 2 As shown, the scale space of the input original image is constructed;

[0052] Step 1 is implemented in the following steps:

[0053] Step 1.1: Select the main image and convert it to grayscale for stitching;

[0054] Step 1.2: Convert the input raw image into a grayscale image;

[0055] Step 1.3: Use Gaussian smoothing to process the image to reduce noise;

[0056] Step 1.4: The image at this time is downsampled using Gaussian convolution kernels of different scales to obtain a series of images of different sizes, and Gaussian filtering is added during the downsampling process.

[0057] Step 2, extreme value detection is performed on the scale space constructed in step 1, and the feature points in the space are located according to the detection result;

[0058] Step 2 is specifically implemented according to the following steps:

[0059] Step 2.1, difference between two layers of the Gaussian pyramid is obtained to obtain a DoG pyramid as shown in Figure 3 The pixel value change on the image can be obtained through the Gaussian difference function, so that the points with the most drastic change, i.e. potential feature points, are obtained;

[0060] Step 2.2, each pixel point in the image is traversed, and its 8 adjacent pixel points in the layer and 9+9 pixel points in the adjacent upper and lower layers are compared, if it is a local extreme point, it is reserved as a candidate point of the feature point, and the possible relationship between the local extreme point and the feature point is as shown in Figure 4

[0061] Step 2.3, Taylor series expansion is performed on the detected extreme point to fit the three-dimensional quadratic function, and the extreme value at the extreme point is calculated. When locating the feature point, if the feature point is at the edge of the image, the principal curvatures of the points in the edge direction and the principal curvatures of the points in the direction perpendicular to the edge need to be detected, and the ratio of the two is used for screening, and the extreme points obtained through the above steps that meet the requirements are used as the feature points of the image.

[0062] Step 3, multi-image matching is performed according to the feature points determined in step 2 to obtain a feature matching set;

[0063] Step 3 is specifically implemented according to the following steps:

[0064] Step 3.1, the gradient and direction features of the pixels in the circular neighborhood window of the feature points in the DoG pyramid are calculated as shown in Figure 5 , and the modulus and direction thereof are counted;

[0065] Step 3.2, as shown in Figure 6 , the main direction of the feature point is determined according to the statistical value, the gradient information of 8 directions in the corresponding neighborhood is calculated according to a 4x4 template, a descriptor is established to describe the feature point, and a total of 8x4x4-dimensional feature descriptors are obtained. The feature descriptor is normalized to obtain a feature vector.

[0066] Step 3.3, K nearest neighbor search based on improved random K-D tree is adopted, and the construction process is as shown in Figure 7 , i.e. the fast nearest neighbor approximation search function library (FLANN-matchcer) finds the matching point with the shortest Euclidean distance in another image for each feature point in an image;

[0067] ​Step 3.4. The RANSAC algorithm is used to remove the mismatched points by dynamically adjusting the ratio of the nearest neighbor and the second nearest neighbor, so as to obtain a more complete feature matching set.

[0068] In step 3.2, k is 4, and the dimension n is 128.

[0069] Step 4. The homography matrix is calculated by the feature matching set to realize the image stitching of multiple images.

[0070] Step 4 is implemented according to the following steps:

[0071] Step 4.1. The homography matrix is calculated by the feature matching set of the two images and solved;

[0072] Step 4.2. All pixel points of one of the images are transformed by the obtained homography matrix, and the other image is directly copied to the corresponding position to realize image stitching.

[0073] Step 4.3. After the stitching of the two images is realized, the multi-image stitching is further realized, the main image is selected for stitching, and the image is preprocessed by cylindrical projection before stitching to suppress distortion.

[0074] In order to verify the effectiveness of the present application for improving the speed, accuracy and distortion suppression of image stitching, the comparison results given by the experiment. The experimental platform uses the processor Intel(R) Core(TM) i7-10700F CPU, 16GB memory, NVIDIA RTX 2060 super GPU, respectively realizes the classical SIFT algorithm and the improved SIFT algorithm, and obtains the corresponding performance and result comparison, to verify the effectiveness of the improvement idea of SIFT algorithm in the present application.

[0075] (1) Due to the characteristics of aerial images, generally no deformation will be generated, so the images taken in the campus and with deformation are selected as a group of experimental materials, as shown in Figure 8 In the classical SIFT algorithm, each feature point in (a) needs to traverse each feature point in (b) to find the point with the smallest Euclidean distance as the match, and the matching time complexity is O(N 2). If the picture resolution is high, the pixel points are more, and as the number of extracted feature points increases by orders of magnitude, the time consumption increases by power function, which greatly increases the overall processing time. Therefore, the improved random K-D tree-based K nearest neighbor search, i.e. the fast nearest neighbor approximation search function library (FLANN-matchcer) is used to replace the brute force matching (BF-matcher). The random K-D tree is a structure for partitioning K-dimensional data space, which is essentially a balanced binary tree, and its construction process is as shown in Figure 7 .

[0076] The average time complexity of the nearest neighbor search can be controlled at O(log N ), which can greatly shorten the matching speed to achieve the purpose of reducing the total time. The total time consumption of the improved algorithm is 450.34s, and the total time consumption of the BF-Matcher in the classic SIFT algorithm is 704.72s. It can be seen that the algorithm running time has been greatly improved.

[0077] (2) In the feature matching of the traditional SIFT algorithm, there is often a phenomenon of false matching, i.e. the phenomenon that the feature points in two images cannot be correctly matched together due to occlusion and other reasons. Figure 9 There are some false matching points, and if these false matching points are not properly processed, it will inevitably affect the subsequent homography transformation, and even cause the stitching to fail. Therefore, the present application uses the random sample consensus algorithm (RANSAC) to achieve the purpose of eliminating false matching points by dynamically adjusting the ratio of the nearest neighbor and the second nearest neighbor. The traditional algorithm eliminates those matches that do not meet the accuracy requirement by giving the ratio of the nearest neighbor and the second nearest neighbor (Lowe's rate), and the threshold value determines the matching accuracy, and the number of matching points will also decrease. In the present application, the user dynamically inputs this threshold value to adjust the matching accuracy required, and eliminates those matches that do not meet the requirement.

[0078] The aerial images are selected as shown in Figure 10 (a) and Figure 10 (b). The feature matching result when Lowe's rate = 0.7 is shown in Figure 10 (c), and the feature matching result when Lowe's rate = 0.45 is shown in Figure 10(d) shows that as the threshold decreases, the number of matches screened out is less and more accurate, and the specific data is shown in Table 1. The RANSAC formula randomly selects 4 matches from the original matching set, fits a model, and then calculates the distance of the model from the remaining matches. If it is less than the set threshold, the matching is added to the inlier set of the model, and the cost function is calculated. After several iterations, the model with the most inliers, i.e. the smallest cost function, is taken as the final result, and those matches not in the inlier set of the model are discarded. As shown in Figure 11 Fig. 11(a) and 11(b) are feature matching results processed by RANSAC and without RANSAC algorithm, respectively. As shown in Table 2, the matching feature number is plotted into a table, and it can be seen that most of the false matches are removed by the RANSAC algorithm, and a better matching result is obtained.

[0079] (3) The traditional SIFT algorithm often cannot achieve satisfactory results in multi-image stitching. For multi-image stitching, taking three images as an example, as shown in Figure 12 , the general approach is to first stitch (a) and (b) to generate result (d), and then stitch (d) and (c) to achieve the panoramic stitching result (f). It can be seen that the stitching effect is very poor, and (c) as the rightmost image is greatly stretched, resulting in serious distortion and loss of much information. At the same time, the stitching trace at the joint is obvious, leading to poor visual effect.

[0080] This phenomenon is caused by vertical distortion due to the use of planar homography transformation without considering spatial consistency when stitching. To solve this problem, the main image is selected for stitching, and the image is preprocessed by cylindrical projection before stitching. By these two means, better results can be achieved. As described, the traditional stitching method is to repeatedly stitch a new image to the previous result to complete panoramic stitching. Taking Figure 12 as an example, the present application stitches (a) and (b) into (d), then stitches (b) and (c) into (e), and finally stitches (d) and (e) to achieve the purpose of panoramic stitching. In this way, the main image (b) is used for stitching, which reduces the distortion to some extent. As shown in Figure 13 , the preprocessed multi-image stitching original image and result are shown, where (a) is the original image, (b) is the image obtained after preprocessing, and (c) is the final stitching result. It can be seen that after preprocessing, the stitching can achieve good results.

[0081] For the joint produced by the traditional SIFT algorithm stitching, as shown in Figure 14 (a), the image weighted fusion method is used for processing, and the processing result is shown in Figure 14 (b).

[0082] (4) The experiment also spliced pictures as shown in Figure 15 , wherein (a) and (b) are two groups of different scene spliced pictures, and the feature matching results are shown in (c); the spliced aerial pictures are shown in Figure 16 , wherein (a) and (b) are two groups of different scene spliced pictures, and (c) is a panoramic aerial image; the spliced aerial pictures are shown in Figure 17 , wherein (a) is an image to be spliced, (b) is a polygon image, and (c) is the sharpening effect after image splicing.

[0083] As shown in Figure 16 , Figure 17 , when the picture size is small, the improved SIFT algorithm proposed in this paper does not have obvious advantages compared with the classic SIFT, and the time consumption of splicing is basically consistent, but with the increase of the size of the picture to be spliced, it can be seen that the time consumption of the classic SIFT increases significantly faster than the improved SIFT algorithm in this paper, and when the size of the picture to be spliced reaches 4032x3024 pixels, the time consumption of the improved SIFT algorithm is reduced by 36.1% compared with the classic SIFT algorithm. In summary, the improvement point proposed in this paper has a very significant improvement compared with the classic SIFT algorithm in large size picture splicing, and is basically equivalent to SURF, as shown in Figure 18 .

[0084] The quality of the image itself is often evaluated by a no-reference evaluation such as peak signal-to-noise ratio (PSNR) and mean square error (MSE), and for the image splicing algorithm studied in this paper, the more important measurement indicator should be the degree of preservation of the original image information after splicing, and PSNR, a simple evaluation based on difference profile, is not suitable for the human visual system, so a no-reference evaluation is obviously not suitable. Therefore, the evaluation standard used in this paper is a partial reference evaluation, that is, the structural similarity evaluation method (SSIM, Structural Similarity). This algorithm evaluates the image quality from three aspects, namely brightness, contrast and structural similarity, and the SSIM evaluation standard is closer to the standard of the human visual system than the PSNR. The results of the classic SIFT algorithm and the improved SIFT algorithm in this paper are evaluated by SSIM. The specific steps are to use a sliding window to block the spliced image and the corresponding original image, and then evaluate the SSIM score by SSIM, and finally take the average of the three SSIM scores to get the overall evaluation.

[0085] It can be seen from Table 3 that the improved SIFT algorithm is higher than the classic SIFT algorithm for the middle picture and the rightmost picture while keeping the evaluation of the leftmost camera basically unchanged, and the overall evaluation is obviously better than that of the traditional SIFT algorithm, which proves that the improvement of the SIFT algorithm is effective in visual effect from the objective evaluation.

[0086] In conclusion, the improved SIFT algorithm is proved to have greater improvement in the speed of splicing, the accuracy of splicing and multi-picture splicing compared with the classic SIFT algorithm, which proves the practicability of the application.

[0087] Table 1 screening results of matching results under different Lowe' rates

[0088]

[0089] Table 2 screening results of matching by RANSAC

[0090]

[0091] Table 3 SSIM for the quality evaluation of spliced pictures

[0092]

[0093] The image splicing method based on regional hierarchical feature matching and improved SIFT can better optimize the problems of slow SIFT algorithm speed, existence of mismatching points and distortion in multi-picture splicing by reasonable design, and has certain practical significance.

Claims

1. An image stitching method based on regional hierarchical feature matching and improved SIFT, characterized in that, The method is implemented according to the following steps: Step 1, construct a scale space of the input original image using a Gaussian pyramid; Step 2, perform extreme value detection on the scale space constructed in step 1, and locate feature points in the space according to the detection results; Step 2.1, obtain a DoG pyramid by differentiating each two layers of the Gaussian pyramid, and obtain potential feature points by a Gaussian difference function; Step 2.2, traverse each pixel point in the image, compare it with 8 adjacent pixel points in the layer and 9+9 pixel points in the adjacent upper and lower layers, and if it is a local extreme point, it is reserved as a candidate point of the feature point; Step 2.3, perform Taylor series expansion on the detected extreme point to fit a three-dimensional quadratic function, and calculate the extreme value of the extreme point; when locating the feature point, if the feature point is at the edge of the image, the principal curvatures of the points in the edge direction and the principal curvatures perpendicular to the edge direction need to be detected, and the ratio of the two is used for screening, and the extreme points obtained through the above steps that meet the requirements are used as the feature points of the image; Step 3, perform multi-image matching according to the feature points determined in step 2 to obtain a feature matching set; Step 3.1, calculate the gradient and direction features of the pixels in the circular neighborhood window of the feature points in the DoG pyramid, and then count the modulus and direction of the pixels; Step 3.2, determine the main direction of the feature points according to the statistical values, calculate the gradient information of 8 directions in the corresponding neighborhood according to a 4x4 template, establish a descriptor to describe the feature points, and obtain a feature vector after normalizing the feature descriptor, which has a total of 8x4x4 dimensions; Step 3.3, use the K nearest neighbor search based on the improved random K-D tree, i.e. the fast nearest neighbor approximation search function library, to find the matching point with the shortest Euclidean distance in another image for each feature point in an image; Step 3.4, use the random sample consensus algorithm to remove the mismatched points by dynamically adjusting the ratio of the nearest neighbor and the second nearest neighbor, thereby obtaining a more complete feature matching set; Step 4, calculate the homography matrix through the feature matching set to realize the image stitching of multiple images.

2. The image stitching method based on region hierarchical feature matching and improved SIFT according to claim 1, characterized in that, Step 1 is implemented according to the following steps: Step 1.1, select a main image and convert it into a grayscale image for stitching; Step 1.2, convert the input original image into a grayscale image; Step 1.3, use Gaussian smoothing to process the image to reduce noise; Step 1.4, perform down-sampling on the image using different scale Gaussian convolution kernels to obtain a series of images with different sizes, and add Gaussian filtering during the down-sampling process.

3. The image stitching method based on region hierarchical feature matching and improved SIFT according to claim 1, characterized in that, Step 4 is implemented as follows: Step 4.1, calculate the homography matrix and solve it through the feature matching set of the two images; Step 4.2, perform homography transformation on all pixel points of one of the images through the obtained homography matrix, and then directly copy the other image to the corresponding position to realize image stitching; Step 4.3, after realizing the stitching of the two images, further realize the stitching of multiple images, select a main image for stitching, and perform cylindrical projection on the image before stitching to preprocess and suppress distortion.

Images