Image stitching method based on optimal stitching line self-selected region fade-in and fade-out algorithm

Abstract

The present invention relates to an image stitching method based on an optimal stitching line self-selected area fade-in and fade-out algorithm. The present invention uses a SIFT algorithm to extract feature points from a target image and a reference image respectively, establishes a feature description subset, obtains an initial matching result by comparing the nearest neighbor distance with the next nearest neighbor distance, uses a RANSAC algorithm to remove outliers and retain inliers from the initial matching result to complete image registration; uses an improved fade-in and fade-out fusion algorithm based on the optimal stitching line to perform image fusion, including using a dynamic programming method to find the optimal stitching line, limiting the fusion range according to the optimal stitching line, and finally uses a fade-in and fade-out fusion algorithm to fuse within the limited fusion range to complete image stitching. The present invention better solves the problem of unnatural image stitching results and the presence of stitching seams, enhances image results, and can be used in image stitching.

Description

Technical Field

[0001] The invention relates to the field of image processing, and in particular to an image splicing method based on an optimal stitching line self-selected region fade-in and fade-out fusion algorithm. Background Art

[0002] With the development and progress of the information age, shooting equipment is becoming more and more intelligent and convenient, and people can obtain massive images more conveniently. However, in daily life, the shooting angle of ordinary cameras is far from the human perspective. In order to cope with people's increasing demand for large-scene images, image stitching technology has been widely used. Image stitching refers to a series of overlapping small-angle, low-resolution images, through certain image registration and fusion technology, to generate a high-resolution, large-angle panoramic image, which contains all the information of all previous series of images. At present, image stitching technology has been widely used in civil and military fields, such as virtual reality, medical panoramic images, automotive driving assistance systems, remote sensing image processing and other specific applications.

[0003] The image stitching process can be summarized into three parts: image preprocessing, image registration and image fusion, each of which is related to the quality of the final stitching effect. Although image stitching technology has been studied for many years and the image registration technology in image stitching is relatively mature, image blur and the existence of stitching seams have always been difficult in image stitching. The overlapping area of ​​the stitched image may have stitching seams caused by uneven transition, and the image pixel value becomes smaller after perspective transformation, resulting in image blur. These problems directly affect the quality of the stitching result image and are not conducive to subsequent image interpretation, processing and classification. Therefore, choosing a suitable overlapping area fusion algorithm is helpful to generate seamless stitching images with clear structure and natural transition. Summary of the invention

[0004] In order to solve the technical problems existing in the prior art of splicing seams, unnatural effects and distortion after image transformation, the present invention provides a new splicing method based on improved image fusion.

[0005] Based on the existing image stitching technology, the present invention performs two optimizations:

[0006] (1) Use the mask matrix and the kernel to perform convolution operations on the image to enhance the image.

[0007] To solve the problem of image blur and distortion caused by the decrease of image pixel value after perspective transformation, the mask matrix [[0,-1,0],[-1,5,-1],[0,-1,0]] is selected, and the kernel is used to perform convolution operation on the image to enhance the image.

[0008] (2) Select the fusion area along the best stitching line and use the fade-in and fade-out algorithm to fuse the fusion area images on both sides of the stitching line.

[0009] People are more sensitive to the edge information of images, and need to fuse the areas with obvious color transition around the stitching line to achieve better results. By improving the fade-in and fade-out fusion algorithm, the present invention proposes a fade-in and fade-out fusion algorithm based on the optimal stitching line, limits the fusion area, and effectively eliminates the artifacts and stitching seams generated during the stitching process.

[0010] The specific steps of the method of the present invention are:

[0011] Step 1: Preprocess the acquired images A and B to be stitched and unify the image size. Use the sift algorithm to extract feature points from the target image and the reference image respectively and establish a feature descriptor set.

[0012] Step 2: Use the KNN matching algorithm to match feature points, use the Lowe's algorithm to filter, use the obtained matching pairs as the initial matching results, and use the RANSAC algorithm to remove outliers and retain inliers for the initial matching results.

[0013] Step 3: Calculate the affine transformation matrix H between the target image and the reference image, and sharpen the image after perspective transformation.

[0014] Step 4: Use the dynamic programming method to find the best stitching line, generate a strip-shaped fusion area along the best stitching line, and use the fade-in and fade-out method to fuse the selected fusion area images to complete the image stitching.

[0015] Beneficial effects of the present invention:

[0016] Effect 1: The kernel is used to implement convolution operations on the image, enhance the image, and effectively solve the problem of image distortion after perspective change;

[0017] Effect 2: Select the fusion area along the best stitching line, and use the fade-in and fade-out algorithm to fuse the fusion area images on both sides of the stitching line, effectively eliminating the stitching seam problem and artifacts generated during the image stitching process. Whether from the perspective of image fusion evaluation indicators or visual effects, the image quality is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a schematic diagram of the process of the present invention.

[0019] Figure 2 It is a test image in the embodiment.

[0020] Figure 3 is an image to be registered in the embodiment.

[0021] Figure 4 It is the rough registration result of the filtered images.

[0022] Figure 5 It is the registration result after eliminating false matches using the RANSAC algorithm.

[0023] Figure 6 This is a schematic diagram of the image after transmission transformation.

[0024] Figure 7 It is a schematic diagram of the optimal stitching seam result.

[0025] Figure 8 This is a schematic diagram of the splicing results of the direct fade-in and fade-out algorithm.

[0026] Fig. 9 This is a schematic diagram of the stitching results of the Laplace fusion algorithm.

[0027] Fig.10 It is a schematic diagram of the stitching result based on the improved optimal seam line fade-in and fade-out fusion algorithm in this embodiment. DETAILED DESCRIPTION

[0028] The present invention is further described in detail below in conjunction with examples. It should be understood that the specific implementation described herein is only used to explain the present invention and is not used to limit the present invention.

[0029] The pictures processed in this embodiment are as follows Figure 2 , Figure 3 As shown in FIG. 1 , a target image and a reference image are included. The two images are taken from two angles of the same large scene, and the two images have common parts.

[0030] This example provides an image stitching method to improve image fusion. Figure 1 , the method comprising:

[0031] Step 1: Preprocess the acquired images A and B to be stitched and unify the image size. Use the sift algorithm to extract feature points from the target image and the reference image respectively, and establish a feature descriptor set. The advantage of using the sift algorithm for feature point extraction is that the sift algorithm remains unchanged for image rotation, scaling, and brightness changes, and also maintains a certain degree of stability for perspective changes, affine transformations, and noise.

[0032] Step 2: Use the KNN matching algorithm to match feature points, use the Lowe's algorithm to filter, use the obtained matching pairs as the initial matching results, and use the RANSAC algorithm to remove outliers and retain inliers. The specific operation of filtering matching points is:

[0033] S1: Take a sift feature point in one image and find the two feature points with the closest Euclidean distance to the feature point in another image;

[0034] S2: Among these two feature points, if the ratio of the closest distance divided by the second closest distance is less than the threshold T-0.6, then this pair of matching points is accepted. The image registration results after screening are as follows: Figure 4 As shown;

[0035] S3: For the initial matching point results, use the RANSAC algorithm to remove outliers and retain inliers. Randomly select 4 pairs of feature points from the feature point pair set, use the least squares method to solve the transformation parameters, and use the 4 pairs of feature points selected above to solve the image geometric change matrix H. Figure 5 The registration result after eliminating wrong matches by RANSAC algorithm.

[0036] Step 3: Calculate the affine transformation matrix H between the target image and the reference image, and sharpen the image after perspective transformation. The processing methods include:

[0037] Use perspective transformation to project the image onto a new viewing plane. In a homogeneous coordinate system, the perspective transformation on a two-dimensional plane can be described in the form of a non-singular 3x3 matrix, namely:

[0038]

[0039] The parameter matrix H is the homography matrix, which describes the motion relationship between images in different coordinate systems and can also indirectly reflect the motion mode of the camera. Figure 6 shown.

[0040] In order to better highlight the clarity of the processed view image and enhance the blurred detail level, as a preferred embodiment, the present invention further sharpens the target view image and the image after perspective transformation, and according to the Laplacian operator, the template coefficient is deformed on the original basis of the operator, and the mask template matrix [[0,-1,0], [-1,5,-1], [0,-1,0]] is selected. The kernel is used to implement the convolution operation of the image. The matrix can add the negative of the sum of the values ​​of the four neighboring points above, below, left and right of the central pixel, plus five times the value of the central pixel, to highlight the position where the brightness value suddenly changes, so as to achieve the effect of making the edges, contours and details of the image clear, thereby enhancing the image.

[0041] Step 4: Use dynamic programming to find the best stitching line, generate a strip-shaped fusion area along the best stitching line, and use the fade-in and fade-out method to fuse the selected fusion area images to complete the image stitching. The processing methods include:

[0042] (A) Establish the optimal suture line solution criterion:

[0043]

[0044] Where E color (x, y) represents the image color difference value; E geometry (x, y) represents the intensity value of image structure difference; E geometry The solution criterion for (x,y) is

[0045] E geometry (x,y)=[S x (I1(x,y)-I2(x,y))] 2 +[S y (I1(x,y)-I2(x,y))] 2

[0046] Among them, S x and S y They are respectively represented as 3x3 Sobel operator templates in the x and y directions, specifically expressed as

[0047] (B) In the overlapping area, each pixel in the first row of the image is used as the starting point of a stitching line, and the criterion value is used as the intensity value.

[0048] (C) Continue searching the next row. The search directions are directly below the current point, 45 degrees to the lower left, and 45 degrees to the lower right of each stitching line. Using the best stitching line criterion, compare the criterion values ​​of these three points with the criterion values ​​of the two adjacent points of the current point. The pixel with the smallest criterion value is updated as the current point.

[0049] (D) If the points in the last row of the overlapping image are found, proceed to step E), otherwise return to step C) and continue the next search.

[0050] (E) Among all sutures, the suture with the smallest strength value is selected as the best suture.

[0051] The best suture line found in this step is Figure 7 shown.

[0052] This embodiment limits the fused area, and the specific steps for determining the fused area are as follows:

[0053] (a) Calculate the left boundary start and right boundary end of the minimum circumscribed matrix of the optimal stitching line, and determine the limited fusion area S according to the position of the optimal stitching line;

[0054] (b) The current point x of each row of stitching is compared with the midpoint mid of the overlapped area, and the shorter distance between the current point of the stitching and the two boundaries is selected as the current row fusion limit range d, and the limited boundary is [xd, x+d];

[0055] (c) Determine the fusion area range S according to d of each row. Generally speaking, it is an irregular area through traversal.

[0056] Furthermore, fusion is performed in the fusion area according to a fade-in and fade-out fusion algorithm.

[0057] In the overlapped area, within the determined fusion area S, the pixel values ​​of the images A and B to be fused are set to I1(x, y) and I2(x, y). The calculation formula for the pixel value I(x, y) of the image that needs transition processing at the fusion point is:

[0058] I(x,y)=ω1I1(x,y)+ω2I2(x,y)

[0059]

[0060] in,

[0061] formula In, x i is the horizontal coordinate of the current pixel, x l is the left boundary of the fusion area S, x r is the right boundary of region S.

[0062] In the non-fused area of ​​the overlapping area, determine whether it is on the left or right side of the stitching line. If it is on the left, the pixel point is the pixel point I1 (x, y) of the left image. If it is on the right, the pixel point is the pixel point I2 (x, y) of the left image. The experimental results after fusion in this step are as follows: Fig.10 shown.

[0063] The objective evaluation standard of image fusion is to obtain objective quantitative indicators by calculating the statistical parameters of the image. In order to verify the effectiveness of the improved algorithm proposed in this embodiment, the quality evaluation standard of image fusion is added to compare the improved fusion algorithm with the traditional weighted fusion algorithm. This embodiment mainly evaluates the fusion quality from three aspects: spatial frequency, standard deviation (STD) and information entropy.

[0064] Table 1 is a schematic diagram showing the comparison of the results of the traditional weighted average fusion and the improved fusion method in this embodiment.

[0065]

[0066] The spatial frequency, standard deviation and information entropy of the algorithm in this embodiment are higher than those of the existing fusion algorithm, which shows that the image quality obtained by the improved fusion algorithm is better and clearer, and it is more in line with actual needs.

[0067] Figure 8 , Fig. 9 , Fig.10 The stitching results of the direct fade-in and fade-out algorithm, the Laplace fusion algorithm and the improved fade-in and fade-out algorithm based on the optimal seam line self-selected area in this embodiment are shown in FIG. It can be seen that the direct fade-in and fade-out algorithm generates artifacts due to the direct fusion of the overlapping areas. The fade-in and fade-out fusion algorithm based on the optimal seam line self-selected area proposed in the present invention has a good stitching effect, and the image is clear and smooth without artifacts.

Claims

1. An image stitching method based on an optimal stitching line self-selected region fade-in and fade-out algorithm, characterized in that The method comprises the following steps: Step 1: Preprocess the acquired images A and B to be stitched and unify the image size; use the sift algorithm to extract feature points from the target image and the reference image respectively, and establish a feature descriptor set; Step 2: Use the KNN matching algorithm to match feature points, use the Lowe's algorithm to filter, use the obtained matching pairs as the initial matching results, and use the RANSAC algorithm to remove outliers and retain inliers from the initial matching results; Step 3: Calculate the affine transformation matrix H between the target image and the reference image, and sharpen the image after perspective transformation; Step 4: Use the dynamic programming method to find the best stitching line, generate a strip-shaped fusion area along the best stitching line, use the fade-in and fade-out method to perform image fusion processing on the selected fusion area, and complete the image stitching; Finding the best suture includes: (A) Establish the optimal suture solution criterion; (B) In the overlapping area, each pixel in the first row of the image is used as the starting point of a stitching line, and the criterion value is used as the intensity value; (C) Search the next row, with the search directions being directly below the current point, 45 degrees to the lower left, and 45 degrees to the lower right of each stitching line. Use the best stitching line to solve the criterion, compare the criterion values ​​of these three points with the criterion values ​​of the two adjacent points of the current point, and update the pixel point with the smallest criterion value as the current point. (D) If the points in the last row of the overlapping image are found, proceed to step E), otherwise return to step C) and continue the next search; (E) Among all sutures, the suture with the smallest strength value is selected as the best suture.

2. The image stitching method based on the optimal stitching line self-selected area fade-in and fade-out algorithm according to claim 1, characterized in that: The specific process of sharpening in step 3 is as follows: Select the mask matrix [[0,-1,0],[-1,5,-1],[0,-1,0]] and use the kernel to perform convolution operation on the image to enhance the image.

3. The image stitching method based on the optimal stitching line self-selected area fade-in and fade-out algorithm according to claim 1, characterized in that: Determining the fusion range in step 4 includes: (a) Calculate the left boundary start and right boundary end of the minimum circumscribed matrix of the optimal stitching seam, and determine the limited fusion area S according to the position of the optimal stitching line; (b) The current point x of each row of stitching is compared with the midpoint mid of the overlapped area, and the shorter distance between the current point of the stitching and the two boundaries is selected as the current row fusion limit range d, and the limited boundary is [xd, x+d]; (c) By traversing, the fusion area range S is determined according to d of each row.

Images