Image splicing method based on improved ORB and MLESAC

Abstract

The invention provides an image splicing method based on improved ORB and MLESAC, and belongs to the technical field of image splicing. According to the method, after a to-be-spliced image is obtained, firstly, feature point detection is performed on the image by using a rapid feature detection method, a descriptor is generated by using a binary robust independent basic feature method, and preliminary matching points are obtained after violent matching; secondly, re-extracting feature descriptors from the screened feature points by using a scale invariant feature transformation method, finding two optimal matches for each descriptor through a K-nearest neighbor algorithm, and adding a ratio test to filter out weak matches; and performing weighting processing on the data points by using a maximum likelihood estimation random consistency algorithm, and performing multiple iterations to obtain an optimal model. And finally, generating a spliced image through a weighted average image fusion technology. According to the method, ORB and SIFT are used in a cascading mode in the rough matching stage, a more efficient MLESAC algorithm is provided to complete fine matching, and the registration precision is remarkably improved while the image splicing real-time performance is ensured.

Description

Technical Field

[0001] The present invention relates to the field of image stitching, and in particular to an image stitching method based on improved ORB and MLESAC. Background Art

[0002] With the rapid development of digital imaging technology, image stitching technology, as one of the important research directions, is widely used in panoramic image generation, virtual reality, unmanned driving, medical imaging, remote sensing image processing and other fields. The goal of image stitching is to seamlessly synthesize a wide-angle panoramic image from a sequence of images with a certain overlapping area to provide a richer visual experience. To achieve this goal, it is necessary to solve multiple technical problems such as feature matching, image registration, and image fusion. Therefore, the research on image stitching algorithms has always been a hot issue in the field of computer vision and image processing.

[0003] At present, image stitching technology mainly includes several key steps such as image feature extraction, feature matching, image registration and image fusion. Although traditional image feature extraction methods can effectively extract key points in images, they are not efficient enough in real-time applications due to the high demand for computing resources. In recent years, ORB, as an efficient feature point extraction method, has gradually become the mainstream of image stitching methods. However, the algorithm is sensitive to complex scenes such as scale transformation and illumination change, and has poor robustness. Therefore, SIFT and ORB are added in cascade, which combines the advantages of both and improves matching accuracy while ensuring real-time matching. In the process of feature matching, due to the possible presence of noise, mismatched points and other interference in the image, feature point matching may result in mismatching, which in turn affects the quality of the stitching result. Therefore, in order to accurately eliminate mismatches from a large number of matches, the MLESAC algorithm is proposed to generate the optimal model by maximizing the likelihood function of the model, further improving the accuracy and robustness of the algorithm.

[0004] Therefore, an image stitching method based on improved ORB and MLESAC is proposed. The performance of efficient feature extraction and matching of the improved ORB and the optimization of the MLESAC algorithm are utilized to provide a new solution for the promotion of image stitching technology in practical applications. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides an image stitching method based on improved ORB and MLESAC.

[0006] The present invention provides an image stitching method based on improved ORB and MLESAC, comprising the following steps:

[0007] S1. Perform fast feature matching on the spliced ​​images through ORB, and select preliminary matching points through brute force matching;

[0008] S2, add SIFT and ORB cascade to re-extract feature descriptors for preliminary matching points;

[0009] S3, find the two best matches (nearest neighbor and second neighbor) for each descriptor through K-nearest neighbor algorithm, and add ratio test (Lowe's ratio test) to filter weak matches;

[0010] S4. In the feature precision matching stage, the more robust MLESAC algorithm is used to perform more accurate weighting of data points and generate the best transformation model after fewer iterations;

[0011] S5. The images are registered according to the estimated optimal geometric transformation model, and the image fusion is completed using the weighted average method to finally generate a natural and seamless stitched image.

[0012] Furthermore, in the feature detection stage in S1, the FAST method is used to detect feature points, and the BRIEF method is used to generate binary string descriptors at the locations where the feature points are detected. The difference between binary descriptors is measured by the Hamming distance, and for each descriptor, the descriptor with the smallest distance is selected as the matching point.

[0013] It should be noted that: in the rough feature matching stage in S2, the present invention adds SIFT and ORB cascade use, and on the basis of preliminary matching, SIFT is used to re-extract feature descriptors for the filtered feature points, and a 128-dimensional feature vector is generated according to the scale, rotation, illumination and local gradient information of the image.

[0014] Furthermore, the specific process of completing the feature rough matching by sequentially using the KNN algorithm and the ratio test in S3 is as follows:

[0015] S31, create a KNN matcher and store the SIFT (Scale Invariant Feature Transform) descriptors of all images in a database;

[0016] S32, for each test image, calculating the Euclidean distance between its SIFT descriptor and all descriptors in the database;

[0017] S33, setting the K value to 2, finding two best matches for each test descriptor, namely the nearest neighbor and the next nearest neighbor;

[0018] S34, introduce a ratio test (Lowe's ratio test) to calculate the Euclidean distance ratio between the nearest neighbor and the next nearest neighbor. If the ratio is less than a set threshold (determined according to the specific application scenario), the match is considered to be a valid match.

[0019] It should be noted that by combining the global search capability of KNN with the local optimization of the ratio test, the output matching result is the matching degree ranking of the target feature and the features in the candidate library. The matching degree score list is used for further image stitching applications.

[0020] Furthermore, in the feature precise matching stage in S4, the present invention optimizes the fitting of the model through the maximum likelihood function of the MLESAC algorithm. When optimizing the model, the algorithm not only considers the number of internal points, but also measures the contribution of each data point through the maximum likelihood criterion. The specific calculation function is:

[0021]

[0022] Among them, x i is the data point, y i is the measured value corresponding to each data point, P(y i |x i ,θ) is the measured value y i Probability of belonging to the model.

[0023] It should be noted that when dealing with more noise and complex scenarios, MLESAC can avoid overfitting or underfitting problems and converge to the optimal model in fewer iterations by combining random sampling methods and maximum likelihood estimation optimization strategies.

[0024] Furthermore, the specific process of generating the final stitched image through registration and fusion in S5 is:

[0025] S51, calculating and obtaining a transformation matrix according to the optimal model output by the MLESAC algorithm;

[0026] S52, using the transformation matrix to perform perspective transformation on each pixel in the source image, aligning the key points of the source image with the key points of the target image through a homography image transformation function, and realizing image registration;

[0027] S53, weighted average fusion is performed on the registered images, a smoothing algorithm is used to reduce mutations during boundary fusion, and finally the information of all input images is integrated and the fusion effect is adjusted according to the weight.

[0028] Compared with the related art, the image stitching method based on improved ORB and MLESAC provided by the present invention has the following beneficial effects:

[0029] 1. The improved ORB algorithm proposed in the present invention uses ORB and SIFT in cascade, combining the advantages of both. After ORB is used to quickly screen out preliminary matching points, the amount of calculation for subsequent SIFT fine matching can be greatly reduced. The matching accuracy is improved while ensuring the real-time performance of the matching.

[0030] 2. The maximum likelihood estimation random consensus algorithm MLESAC proposed in the present invention not only considers the number of inliers and outliers, but also pays attention to the weight of each data point, which can better eliminate false matches, thereby significantly improving image stitching in terms of robustness, accuracy and applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0032] Figure 1 A flowchart of an image stitching method based on improved ORB and MLESAC provided by the present invention;

[0033] Figure 2 is the solution diagram of the main direction of the feature point;

[0034] Figure 3 This is a schematic diagram of the maximum likelihood estimation random consistency algorithm flow proposed by the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0036] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] The specific implementation of the image stitching method based on improved ORB and MLESAC is described in detail below in conjunction with specific embodiments.

[0038] The present invention provides an image stitching method based on improved ORB and MLESAC, comprising the following steps:

[0039] S1. Perform fast feature matching on the spliced ​​images through ORB, and select preliminary matching points through brute force matching;

[0040] S2, add SIFT and ORB cascade to re-extract feature descriptors for preliminary matching points;

[0041] S3, find the two best matches (nearest neighbor and second neighbor) for each descriptor through K-nearest neighbor algorithm, and add ratio test (Lowe's ratio test) to filter weak matches;

[0042] S4. In the feature precision matching stage, the more robust MLESAC algorithm is used to perform more accurate weighting of data points and generate the best transformation model after fewer iterations;

[0043] S5. The images are registered according to the estimated optimal geometric transformation model, and the image fusion is completed using the weighted average method to finally generate a natural and seamless stitched image.

[0044] Furthermore, in the feature detection stage in S1, the FAST method is used to determine whether a pixel is a feature point to achieve fast detection. When the feature point is detected, the BRIEF method is used to generate a binary string descriptor. The descriptor of each feature point is compared with the descriptors of all other feature points through brute force matching, and the Hamming distance is used to measure the difference between the binary descriptors. The descriptors are sorted according to the Hamming distance after matching, and the descriptor with the smallest distance is used as the matching point.

[0045] Furthermore, in the rough feature matching stage in S2, the present invention adds SIFT and ORB cascade, and on the basis of preliminary matching, SIFT is used to re-extract feature descriptors from the selected feature points, specifically in the following manner:

[0046] For the input image I(x,y), first construct Gaussian blurred images of different scales L(x,y,σ)=G(x,y,σ)*I(x,y), where G(x,y,σ) is a Gaussian function: Then, the Gaussian blurred images of different scales are subtracted to construct the differential Gaussian image: D(x, y, σ) = L(x, y, kσ) - L(x, y, σ), where k is the scale factor. The key points are determined by detecting the extreme points in the differential Gaussian image. For each D(x, y, σ), determine whether it is a local extreme value: D(x, y, σ) > (x′, y′, σ) for all(x′, y′, σ) ∈ N(x, y, σ), where N is the set including the point and its neighborhood. For each detected key point, the gradient information in its neighborhood is calculated: Then calculate the direction of the gradient magnitude: Each feature point is assigned a main direction to ensure the rotation invariance of the descriptor. Then the gradient histogram in the neighborhood of the key point is calculated: H(θ) = ∑ i∈ N Magnitude i forθ∈[0, 360°], a 128-dimensional feature vector containing multiple directions and gradient information is generated. Finally, the descriptor is normalized: and

[0047] Furthermore, the specific process of completing the feature rough matching by sequentially using the KNN algorithm and the ratio test in S3 is as follows:

[0048] S31, create a KNN matcher and store the SIFT (Scale Invariant Feature Transform) descriptors of all images in a database;

[0049] S32, for each test image, calculating the Euclidean distance between its SIFT descriptor and all descriptors in the database;

[0050] S33, setting the K value to 2, finding two best matches for each test descriptor, namely the nearest neighbor and the next nearest neighbor;

[0051] S34, introduce a ratio test (Lowe's ratio test) to calculate the Euclidean distance ratio between the nearest neighbor and the next nearest neighbor. If the ratio is less than a set threshold (determined according to the specific application scenario), the match is considered to be a valid match.

[0052] It should be noted that by combining the global search capability of KNN with the local optimization of the ratio test, the output matching result is the matching degree ranking of the target feature and the features in the candidate library. The matching degree score list is used for further image stitching applications.

[0053] Furthermore, in the feature precise matching stage in S4, the present invention optimizes the fitting of the model through the maximum likelihood function of the MLESAC algorithm. When optimizing the model, the algorithm not only considers the number of internal points, but also measures the contribution of each data point through the maximum likelihood criterion. The specific calculation function is:

[0054]

[0055] Among them, x i is the data point, y i is the measured value corresponding to each data point, P(y i |x i , θ) is the measured value y i Probability of belonging to the model.

[0056] It should be noted that when dealing with more noise and complex scenarios, MLESAC can avoid overfitting or underfitting problems and converge to the optimal model in fewer iterations by combining random sampling methods and maximum likelihood estimation optimization strategies.

[0057] Furthermore, the specific process of generating the final stitched image through registration and fusion in S5 is:

[0058] S51, calculating and obtaining a transformation matrix according to the optimal model output by the MLESAC algorithm;

[0059] S52, using the transformation matrix to perform perspective transformation on each pixel in the source image, aligning the key points of the source image with the key points of the target image through a homography image transformation function, and realizing image registration;

[0060] S53, weighted average fusion is performed on the registered images, a smoothing algorithm is used to reduce mutations during boundary fusion, and finally the information of all input images is integrated and the fusion effect is adjusted according to the weight.

[0061] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An image stitching method based on improved ORB and MLESAC, characterized in that: The steps include: S1. Perform fast feature matching on the spliced ​​images through ORB, and select preliminary matching points through brute force matching; S2, add SIFT and ORB cascade to re-extract feature descriptors for preliminary matching points; S3. Use the K-nearest neighbor algorithm to find the two best matches for each descriptor, and add a ratio test to filter weak matches. The specific process is as follows: S31. Create a KNN matcher and store the SIFT descriptors of all images in a database. S32, for each test point, calculating the Euclidean distance between it and all data points in the database; S33, setting the K value to 2, that is, finding two best matches for each descriptor; S34, adding a ratio test to calculate the Euclidean distance ratio between the nearest neighbor and the next nearest neighbor. If the ratio is less than the set threshold, the match is considered valid. S4. In the feature precision matching stage, the more robust MLESAC algorithm is used to perform more accurate weighting processing on the data points, and the optimal transformation model is generated after fewer iterations. In the feature precision matching stage, the maximum likelihood function of the MLESAC algorithm is used to optimize the model fitting, while taking into account the number of internal points and the contribution of each data point to the model. The specific calculation function is: Among them, x i is the data point, y i is the measured value corresponding to each data point, P(y i |x i ,θ) is the measured value y i Probability of belonging to the model; S5. The images are registered according to the estimated optimal geometric transformation model, and the image fusion is completed using the weighted average method, and finally a natural and seamless spliced ​​image is generated. The specific process is as follows: S51, calculating the transformation matrix according to the best model output by MLESAC; S52, using the transformation matrix to perform perspective transformation on each pixel in the source image, that is, using the homography image transformation function to align the key points of the original image with the key points of the target image, and completing the registration; S53, performing weighted average fusion on the registered images, using a smoothing algorithm to reduce mutations during boundary fusion, and finally integrating the information of all input images to adjust the fusion effect according to the weights.

2. The image stitching method based on improved ORB and MLESAC according to claim 1, characterized in that: In the feature detection stage in S1, the FAST method is used to detect feature points. When the feature points are detected, the BRIEF method is used to generate binary string descriptors. The difference between binary descriptors is measured by the Hamming distance. For each descriptor, the descriptor with the smallest distance is selected as the matching point.

3. The image stitching method based on improved ORB and MLESAC according to claim 1, characterized in that: In the S2, SIFT and ORB are added in cascade at the feature rough matching stage. Based on the preliminary matching, SIFT is used to re-extract feature descriptors for the selected feature points, and a 128-dimensional feature vector is generated according to the scale, rotation, illumination and local gradient information of the image.

4. The image stitching method based on improved ORB and MLESAC according to claim 1, characterized in that: The MLESAC in S4 can be achieved by combining a random sampling method and an optimization strategy of maximum likelihood estimation.

Images