A panorama image stitching method

Abstract

The present application relates to the technical field of image data processing, in particular to a panoramic image splicing method, comprising preprocessing a first panoramic image and a second panoramic image to obtain two reduced images; selecting a target gray image from the two reduced images to calculate a remapping table; remapping the two reduced images using the remapping table to obtain two enhanced images; extracting features of the two enhanced images using a feature extraction algorithm to obtain two feature point sets; comparing the feature points of the two feature point sets to obtain a matching point set, filtering the matching point set to obtain a filtered set, and limiting the origins of the feature points in the filtered set; fitting a homography matrix using the matching pairs in the filtered set and applying the homography matrix to the second panoramic image to obtain pixel positions; and fusing the pixel positions into the first panoramic image to obtain a spliced image, thereby solving the problem of long time consumption in image splicing of panoramic image splicing methods.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and in particular to a method for stitching panoramic images. Background Technology

[0002] When stitching panoramic images, feature point detection and feature point matching are generally used to obtain the homography transformation matrix of the image. However, most feature point detection methods are based on texture contours, such as the SIFT feature point detection method. When facing image scenes with sky, the sky texture is too small or the texture gradient is not large, so only a very small number of feature points can be detected, resulting in the two images being unable to match.

[0003] Currently, existing technologies disclose a method for stitching panoramic images. By using a neural network algorithm to detect feature points, it is possible to detect sky feature points. However, the calculation speed is slow, resulting in a long time required for image stitching. Summary of the Invention

[0004] The purpose of this invention is to provide a method for stitching panoramic images, aiming to solve the problem that the image stitching process in panoramic image stitching methods is time-consuming.

[0005] To achieve the above objectives, the present invention provides a method for stitching panoramic images, comprising the following steps:

[0006] S1 preprocesses the first panoramic image and the second panoramic image respectively to obtain the first reduced image and the second reduced image;

[0007] S2 determines the pixel variance of the first and second reduced images respectively. If both pixel variances are greater than a preset threshold, then step S3 is executed. If either of the two pixel variances is less than a preset threshold, then a feature extraction algorithm is used to extract features from the first and second reduced images respectively to obtain a first feature point set and a second feature point set.

[0008] S3 selects a target grayscale image from the first and second reduced images;

[0009] S4 uses the target grayscale image to calculate the remapping table;

[0010] S5 The first and second reduced images are simultaneously remapped using the remapping table to obtain the first enhanced image and the second enhanced image, respectively.

[0011] S6 uses a feature extraction algorithm to extract features from the first enhanced image and the second enhanced image respectively, to obtain a first set of feature points and a second set of feature points;

[0012] S7 compares the feature points of the first feature point set and the second feature point set;

[0013] S8 filters out incorrect matching pairs in the matching point set to obtain a filter set, and defines the origin of each feature point in the filter set;

[0014] S9 uses the matching pairs in the filter set to fit the homography matrix;

[0015] S10 applies the homography matrix to the second panoramic image to obtain pixel positions, and then fuses the pixel positions into the first panoramic image to obtain a stitched image.

[0016] The preprocessing of the first panoramic image and the second panoramic image to obtain the first reduced image and the second reduced image includes:

[0017] S11 converts each pixel of the first panoramic image and the second panoramic image into grayscale, respectively, to obtain the first grayscale image and the second grayscale image;

[0018] S12 reduces the size of the first grayscale image and the second grayscale image respectively to obtain a first reduced image and a second reduced image.

[0019] The step of reducing the size of the first grayscale image and the second grayscale image to obtain a first reduced image and a second reduced image includes:

[0020] Obtain the maximum width and height values ​​of the first grayscale image and the second grayscale image respectively;

[0021] If the maximum length and width exceed the preset value, the scaling ratio of the first grayscale image and the second grayscale image is calculated, and the first grayscale image and the second grayscale image are scaled using the bilinear interpolation method according to the scaling ratio to obtain the first reduced image and the second reduced image, and step S2 is executed;

[0022] If the maximum length and width do not exceed the preset value, then the first grayscale image and the second grayscale image are used as the first and second scaled-down images, and step S2 is executed.

[0023] The step of selecting the target grayscale image from the first reduced image and the second reduced image includes:

[0024] S31 calculates the brightness of the first reduced image and the second reduced image respectively to obtain the first brightness value and the second brightness value;

[0025] S32 selects the maximum brightness value from the first brightness value and the second brightness value;

[0026] S33 uses the scaled-down image corresponding to the maximum brightness value as the target grayscale image.

[0027] The step of calculating the remapping table using the target grayscale image includes:

[0028] S41 counts the number of times each pixel from 0 to 255 appears in the target grayscale image;

[0029] S42 performs a cumulative integration of the occurrence probabilities of each pixel from 0 to 255 based on each occurrence count to obtain the probability density of each pixel from 0 to 255.

[0030] S43 multiplies the probability density of each pixel from 0 to 255 by 255 and rounds it to the nearest integer to obtain the remapping table.

[0031] The feature points used to compare the first set of feature points and the second set of feature points include:

[0032] S71 calculates the Euclidean distance between the feature vector of each point in the first feature point set and the feature vector of each point in the second feature point set, and takes the two points with the smallest distance as matching points to obtain the first matching point set;

[0033] S72 calculates the Euclidean distance between the feature vector of each point in the second feature point set and the feature vector of each point in the first feature point set, and takes the two points with the smallest distance as matching points to obtain the second matching point set;

[0034] S73 removes duplicate matching points from the first matching point set and the second matching point set, and then merges them to obtain the matching point set.

[0035] This invention discloses a panoramic image stitching method, which involves preprocessing a first panoramic image and a second panoramic image to obtain a first reduced image and a second reduced image; judging the pixel variance of the first reduced image and the second reduced image respectively; if both pixel variances are greater than a preset threshold, proceeding to the next step; if either pixel variance is less than the preset threshold, using a feature extraction algorithm to extract features from the first reduced image and the second reduced image respectively, obtaining a first feature point set and a second feature point set; selecting a target grayscale image from the first reduced image and the second reduced image; calculating a remapping table using the target grayscale image; and simultaneously remapping the first reduced image and the second reduced image using the remapping table to obtain a first augmented image. The first enhanced image and the second enhanced image are used; feature extraction algorithms are used to extract features from the first enhanced image and the second enhanced image respectively to obtain a first feature point set and a second feature point set; the feature points of the first feature point set and the second feature point set are compared; erroneous matching pairs in the matching point set are filtered out to obtain a filter set, and the origin of each feature point in the filter set is defined; a homography matrix is ​​fitted using the matching pairs in the filter set; the homography matrix is ​​applied to the second panoramic image to obtain the pixel position, and the pixel position is fused to the first panoramic image to obtain a stitched image. The above steps have a fast computation speed, which solves the problem of long image stitching time in panoramic image stitching methods. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a flowchart of a panoramic image stitching method provided by the present invention.

[0038] Figure 2 These are example images of the first and second panoramic views.

[0039] Figure 3 This is the result of extracting feature points from the first and second panoramic images directly using the SUFT method.

[0040] Figure 4 Is using Figure 3 The matching result image is the result image of the matching process.

[0041] Figure 5This is a result image of feature point extraction from a first panoramic image and a second panoramic image, provided by the panoramic image stitching method of the present invention.

[0042] Figure 6 Is using Figure 5 The matching result image is the result image of the matching process.

[0043] Figure 7 Is using Figure 6 The result image is obtained by stitching together the matching result images. Detailed Implementation

[0044] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0045] Please see Figures 1 to 7 This invention provides a method for stitching panoramic images, comprising the following steps:

[0046] S1 preprocesses the first panoramic image and the second panoramic image respectively to obtain the first reduced image and the second reduced image;

[0047] The specific method is as follows:

[0048] S11 converts each pixel of the first panoramic image I1 and the second panoramic image I2 into grayscale, respectively, to obtain the first grayscale image and the second grayscale image;

[0049] Specifically, given two panoramic images I1 and I2 with overlapping areas, each pixel is converted to grayscale using the following formula to obtain the corresponding grayscale image I. gray1 and I gray2 .

[0050] Gray=0.229*R+0.587*G+0.114*B

[0051] Where Gray is the grayscale value of a pixel, and R, G, and B correspond to the pixel values ​​of red, green, and blue pixels, respectively.

[0052] S12 reduces the size of the first grayscale image and the second grayscale image respectively to obtain a first reduced image and a second reduced image.

[0053] Specifically, the maximum width and height (MaxV) of the first grayscale image and the second grayscale image are obtained respectively. wh ;

[0054] If the maximum length and width MaxV whIf the value exceeds a preset value, then the scaling ratio (Scale) of the first grayscale image and the second grayscale image is calculated. wh and according to the scaling ratio Scale wh Bilinear interpolation was used to analyze the first panoramic image I1, the second panoramic image I2, and the first grayscale image I1. gray1 and the second grayscale image I gray2 Scaling is performed to obtain a first and a second scaled-down image, and step S2 is executed; the preset value is 960.

[0055] If the maximum length and width do not exceed the preset value, then the first grayscale image and the second grayscale image are used as the first and second scaled-down images, and step S2 is executed.

[0056]

[0057] S2 determines the pixel variance of the first and second reduced images respectively. If both pixel variances are greater than a preset threshold, then step S3 is executed. If either of the two pixel variances is less than a preset threshold, then a feature extraction algorithm is used to extract features from the first and second reduced images respectively to obtain a first feature point set and a second feature point set.

[0058] Specifically, since this method is largely ineffective for images whose content is close to a solid color, the following judgment is made first: Calculate the grayscale image I. gray1 and I gray2 pixel variance σ 2 If there is a grayscale image with pixel variance σ... 2 Images below a preset threshold are considered solid color images and are not applicable to this method; if the image is greater than or equal to the preset threshold, proceed to the next step; the preset threshold is 70.

[0059] Variance σ 2 The formula is calculated as follows:

[0060]

[0061]

[0062] Where Pix is ​​the pixel value of the grayscale image, and W and H are the width and height of the grayscale image, respectively.

[0063] S3 selects a target grayscale image from the first and second reduced images;

[0064] The specific method is as follows:

[0065] S31 calculates the brightness W of the first and second reduced images respectively. light Thus, the first brightness value and the second brightness value are obtained;

[0066] W light =|Pix mean -128|

[0067] S32 selects the maximum brightness value from the first brightness value and the second brightness value;

[0068] S33 uses the reduced image corresponding to the maximum brightness value as the target grayscale image I. target .

[0069] S4 uses the target grayscale image to calculate the remapping table;

[0070] Specifically, the remapping table has 256 values ​​(PixRemap). i ,i∈[0,1,…,255], the remapping value corresponding to pixel values ​​from 0 to 255.

[0071] The specific method is as follows:

[0072] S41 counts the number of times each pixel from 0 to 255 appears in the target grayscale image;

[0073] Specifically, the values ​​of 0-255 in the target grayscale image I are statistically analyzed. target The number of times each pixel value i appears in the data is denoted as rate. i .

[0074]

[0075]

[0076] Where Pix(x,y) is located at I target The pixel value at (x,y) is given by W and H, which are I and H, respectively. target Width and height.

[0077] S42 performs a cumulative integration of the occurrence probabilities of each pixel from 0 to 255 based on each occurrence count to obtain the probability density of each pixel from 0 to 255.

[0078] Specifically, the probability of occurrence of each pixel from 0 to 255 is accumulated and integrated to obtain the probability density PDensity of each pixel from 0 to 255. i ,i∈[0,1,…,255].

[0079]

[0080] S43 multiplies the probability density of each pixel from 0 to 255 by 255 and rounds it to the nearest integer to obtain the remapping table.

[0081] Specifically, the probability density of pixels 0-255 is multiplied by 255 and rounded to obtain the remapping table.

[0082]

[0083] Where W and H are respectively I target Width and height, Pix Darkness isI target The darkest pixel.

[0084] S5 The first and second reduced images are simultaneously remapped using the remapping table to obtain the first enhanced image and the second enhanced image, respectively.

[0085] Specifically, two grayscale images I gray1 and I gray2 Simultaneously use PixRemap i Remapping is performed on i∈[0,1,…,255] to obtain the enhanced image I. remap1 and I remap2 .

[0086] I remapi (x,y)=round(PixRemapI grayi (x,y))i∈[1,2]

[0087] Where I(x,y) is the pixel value at (x,y) in image I, and round(...) means rounding the floating-point number to the nearest integer.

[0088] S6 uses a feature extraction algorithm to extract features from the first enhanced image and the second enhanced image respectively, to obtain a first set of feature points and a second set of feature points;

[0089] Specifically, feature point extraction algorithms are used to extract feature points and their corresponding feature vectors. The most representative methods include SIFT, SUFT, and ORB. This yields I... remap1 and I remap2 The feature point sets FP1 and FP2.

[0090] S7 compares the feature points of the first feature point set and the second feature point set;

[0091] The specific method is as follows:

[0092] S71 calculates the Euclidean distance between the feature vector of each point in the first feature point set FP1 and the feature vector of each point in the second feature point set FP2, and takes the two points with the smallest distance as matching points to obtain the first matching point set MatchPair1;

[0093] S72 calculates the Euclidean distance between the feature vector of each point in the second feature point set FP2 and the feature vector of each point in the first feature point set FP1, and takes the two points with the smallest distance as matching points to obtain the second matching point set MatchPair2;

[0094] S73 removes duplicate matching points from the first matching point set and the second matching point set, and then merges them to obtain the matching point set.

[0095] Specifically, in the final matching point sets MatchPair1 and MatchPair2, duplicate matching results are removed and merged into a single matching point set, resulting in MatchPair. merge .

[0096] The method for calculating Euclidean distance is as follows:

[0097]

[0098] Where n equals the dimension of the feature vector, for example, the dimension of a SIFT feature vector is 128. i and y i This represents the value of the dimension corresponding to the feature vector of two feature points.

[0099] S8 filters out incorrect matching pairs in the matching point set to obtain a filter set, and defines the origin of each feature point in the filter set;

[0100] Specifically, the origin of each feature point in the filter set is defined, including the convention of MatchPair. merge From I remap1 The feature point is MatchPoint1, which comes from I remap2 The feature point is MatchPoint2, and the specific process is as follows:

[0101] 1) MatchPair will be added the first time the process is entered. merge All matching pairs are placed into MatchPair final In, from the MatchPair final Four sets of matching points are randomly selected: P1 (x1, y1), P2 (x2, y2), P3 (x3, y3), P4 (x4, y4), P5 (x5, y5), P6 (x6, y6), P7 (x7, y7), and P8 (x8, y8). Calculate their homography matrix. Where P1 and P2 are a pair of matching points, P3 and P4 are a pair of matching points, P5 and P6 are a pair of matching points, and P7 and P8 are a pair of matching points. P1, P3, P5, and P7 come from MatchPoint1, and the rest come from MatchPoint2. The solution formula is as follows:

[0102]

[0103] 2) Apply the homography matrix H to the points in MatchPoint2 to obtain a new point MatchPoint2'. Then calculate the distance between MatchPoint2' and the corresponding matching point in MatchPoint1. If the distance between two points is less than a certain threshold, it is recorded as an interior point and stored in MatchPair. correct Otherwise, it is recorded as an outside point.

[0104] Application methods of homography matrix:

[0105]

[0106] 3) If the number of outliers in the set is less than a certain threshold, exit the loop. Otherwise, treat the marked outliers as MatchPairs. final Continue the loop starting from 1).

[0107] S9 uses the matching pairs in the filter set to fit the homography matrix;

[0108] Specifically, use MatchPair correct The matching pairs in the dataset are used to calculate the homography matrix of the fitted images. If MatchPair correct If the number of matching pairs in the array is greater than 4, then the matching pairs will be matched. correct The data is randomly divided into G groups. If the number of groups is less than 4, the matching is considered a failure, and the program exits.

[0109]

[0110] Where N mp For MatchPair correct The number of matching pairs in the table, floor(...) means to discard the decimal places of the floating-point number and take the integer part directly.

[0111] The homography matrix H is calculated using the four pairs of matching points in each group. i i∈[1,2,…,G]. Finally, the least squares method is used to evaluate H. i By fitting the data, we can obtain H. final .

[0112] S10 applies the homography matrix to the second panoramic image to obtain pixel positions, and then fuses the pixel positions into the first panoramic image to obtain a stitched image.

[0113] Specifically, the homography matrix H is applied to the source image I2. final The position of each pixel in the source image I1 is obtained, and then it is merged into the source image I1 to complete the stitching.

[0114] This invention provides a panoramic image stitching method that uses histogram equalization to enable sky images to display more texture details and higher contrast. This allows most image feature point detection methods to detect more feature points with very fast computation speed. Furthermore, both images share a remapping list to increase image matching accuracy. This technique significantly improves the quantity and quality of feature point matching when dealing with sky scenes, thereby increasing the success rate of panoramic stitching, with very low additional computational time cost. This technique can also be used for panoramic stitching in other similar scenarios (where texture is not obvious or the texture gradient is small).

[0115] Please refer to the example. Figures 2-7 :

[0116] Figure 3 The hollow circle is the detected feature point.

[0117] Figure 4 and Figure 6 Matching is performed using steps S7 and S8.

[0118] from Figure 4 The matching results show that only a few feature points can be matched.

[0119] Figure 5 The result of feature point extraction using steps S1-S6 of the present invention.

[0120] from Figure 6 As can be seen, after image enhancement, the number of matching feature points between the two sky images increases significantly. In panoramic stitching tasks, feature point matching is highly dependent on estimating the spatial location information between two images; more and more widely distributed feature point matching pairs will inevitably lead to better stitching results. At the same time, this method does not have a negative effect on ordinary images, although the effect is not as pronounced as in sky scenes.

[0121] Figure 7 It is the result of splicing after using steps S9-S10 and using distortion correction techniques.

[0122] The above-disclosed method for stitching panoramic images is merely a preferred embodiment of the present invention and should not be construed as limiting the scope of the invention. Those skilled in the art can understand that implementing all or part of the above-described embodiments and making equivalent changes in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A method for stitching panoramic images, characterized in that, Includes the following steps: S1 preprocesses the first panoramic image and the second panoramic image respectively to obtain the first reduced image and the second reduced image; S2 determines the pixel variance of the first and second reduced images respectively. If both pixel variances are greater than a preset threshold, then step S3 is executed. If either of the two pixel variances is less than a preset threshold, then a feature extraction algorithm is used to extract features from the first and second reduced images respectively to obtain a first feature point set and a second feature point set. S3 selects a target grayscale image from the first and second reduced images; S4 uses the target grayscale image to calculate the remapping table; S5 The first and second reduced images are simultaneously remapped using the remapping table to obtain the first enhanced image and the second enhanced image, respectively. S6 uses a feature extraction algorithm to extract features from the first enhanced image and the second enhanced image respectively, to obtain a first set of feature points and a second set of feature points; S7 compares the feature points of the first feature point set and the second feature point set to obtain a matching point set; S8 filters out incorrect matching pairs in the matching point set to obtain a filter set, and defines the origin of each feature point in the filter set; S9 uses the matching pairs in the filter set to fit the homography matrix; S10 applies the homography matrix to the second panoramic image to obtain pixel positions, and then fuses the pixel positions into the first panoramic image to obtain a stitched image.

2. The panoramic image stitching method as described in claim 1, characterized in that, The preprocessing of the first panoramic image and the second panoramic image to obtain the first reduced image and the second reduced image includes: S11 converts each pixel of the first panoramic image and the second panoramic image into grayscale, respectively, to obtain the first grayscale image and the second grayscale image; S12 reduces the size of the first grayscale image and the second grayscale image respectively to obtain a first reduced image and a second reduced image.

3. The panoramic image stitching method as described in claim 2, characterized in that, The step of reducing the size of the first grayscale image and the second grayscale image respectively to obtain a first reduced image and a second reduced image includes: Obtain the maximum width and height values ​​of the first grayscale image and the second grayscale image respectively; If the maximum length and width exceed the preset value, the scaling ratio of the first grayscale image and the second grayscale image is calculated, and the first grayscale image and the second grayscale image are scaled using the bilinear interpolation method according to the scaling ratio to obtain the first reduced image and the second reduced image, and step S2 is executed; If the maximum length and width do not exceed the preset value, then the first grayscale image and the second grayscale image are used as the first and second scaled-down images, and step S2 is executed.

4. The panoramic image stitching method as described in claim 3, characterized in that, Selecting the target grayscale image from the first and second reduced images includes: S31 calculates the brightness of the first reduced image and the second reduced image respectively to obtain the first brightness value and the second brightness value; S32 selects the maximum brightness value from the first brightness value and the second brightness value; S33 uses the scaled-down image corresponding to the maximum brightness value as the target grayscale image.

5. The panoramic image stitching method as described in claim 4, characterized in that, The step of calculating the remapping table using the target grayscale image includes: S41 counts the number of times each pixel from 0 to 255 appears in the target grayscale image; S42 performs a cumulative integration of the occurrence probabilities of each pixel from 0 to 255 based on each occurrence count to obtain the probability density of each pixel from 0 to 255. S43 multiplies the probability density of each pixel from 0 to 255 by 255 and rounds it to the nearest integer to obtain the remapping table.

6. The panoramic image stitching method as described in claim 5, characterized in that, The step of comparing the feature points of the first feature point set and the second feature point set to obtain a matching point set includes: S71 calculates the Euclidean distance between the feature vector of each point in the first feature point set and the feature vector of each point in the second feature point set, and takes the two points with the smallest distance as matching points to obtain the first matching point set; S72 calculates the Euclidean distance between the feature vector of each point in the second feature point set and the feature vector of each point in the first feature point set, and takes the two points with the smallest distance as matching points to obtain the second matching point set; S73 removes duplicate matching points from the first matching point set and the second matching point set, and then merges them to obtain the matching point set.

Images