Panoramic image stitching method and system based on multi-feature guidance, and related equipment

Abstract

The invention discloses a panoramic image splicing method and system based on multi-feature guidance and related equipment, and belongs to the technical field of computer vision and image processing, and the method comprises the steps: respectively carrying out the extraction of point features, linear structures and edge curve structures of a reference image and a target image; performing neighborhood division on the extracted straight line structures and the edge curve structures, selecting point features in neighborhoods to construct projection invariants, and calculating matching relations between the straight line structures and between the edge curve structures to obtain straight line structure matching pairs and intersection point matching pairs based on curve structure matching; and performing grid division on the reference image and the target image, generating a distorted image aligned with the reference image according to the point feature matching pair, the linear structure matching pair and the intersection point matching pair based on curve structure matching, and obtaining a panoramic image. Under a low-texture image scene, the defects of point and line features are overcome through irregular edge matching constraints, and the alignment quality and splicing precision of image splicing are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and image processing, and particularly relates to a panoramic image stitching method, system and related devices based on multi-feature guidance. Background Art

[0002] Image stitching technology is one of the important research directions in the field of computer vision and image processing. Its main goal is to stitch two or more images with overlapping regions into a complete and high-quality image through methods such as feature matching, image transformation and fusion. With the rapid development of intelligent vision devices, image stitching technology has been widely applied in remote sensing images, medical images, 3D modeling, virtual reality and other scenarios. However, in practical applications, especially in low-texture image scenarios, traditional feature matching methods have significant limitations and it is difficult to ensure the accuracy and robustness of stitching.

[0003] Traditional image stitching methods mainly rely on point features, such as SIFT (Scale-Invariant Feature Transform), SURF (Speeded-Up Robust Features), ORB (Oriented FAST and Rotated BRIEF; where FAST is the abbreviation of Features from Accelerated Segment Test, i.e., fast feature detection; BRIEF is the abbreviation of Binary Robust Independent Elementary Features, i.e., binary robust independent basic features), etc. By extracting feature points in the image, calculating descriptors and feature matching, the alignment and stitching of the image are completed. However, in low-texture regions (such as sky, wall, grassland, etc.), there are almost no obvious corner points or high-gradient regions, and it is difficult for traditional algorithms to extract enough feature points; fewer feature points will further lead to insufficient matching constraints, easily introduce false matches, and affect the registration accuracy of the image.

[0004] To make up for the deficiencies of point feature matching, in recent years, line feature matching has become an important supplement in the field of image stitching. Line features have strong geometric constraint capabilities and are often more stable than point features. However, when the straight-line structure is insufficient, there are also certain problems with line feature matching. Specifically, when the straight-line structure is scarce, the extraction of line features becomes particularly difficult. This is because the detection of line features depends on clear and continuous straight edges in the image. If the number of straight lines in the image is insufficient or the straight edges are blurred or broken, the extraction accuracy of line features will be greatly reduced, thus affecting the subsequent matching process. In addition, even if line features are successfully extracted, if there are not enough matching pairs, the accuracy and reliability of the matching results will also be seriously affected. Summary of the Invention

[0005] The object of the present invention is to address the above problems in the prior art, and provide a panoramic image stitching method, system and related devices based on multi-feature guidance, which makes full use of rich edge features in the image, especially irregular edges, combines with point features and line features, and in the low-texture image scenario, through irregular edge matching constraints, effectively makes up for the deficiencies of point and line features, significantly improves the alignment quality and stitching accuracy of image stitching, and enhances stability and robustness.

[0006] To achieve the above object, the present invention has the following technical solutions:

[0007] In the first aspect, a panoramic image stitching method based on multi-feature guidance is provided, including:

[0008] Extract point features, straight line structures and edge curve structures from the reference image and the target image respectively;

[0009] Perform neighborhood division on the extracted straight line structures and edge curve structures, select point features within the neighborhood to construct projection invariants, calculate the matching relationships between straight line structures and between edge curve structures, and obtain straight line structure matching pairs and intersection matching pairs based on curve structure matching;

[0010] Perform grid division on the reference image and the target image, and generate a distorted image aligned with the reference image according to the point feature matching pairs, straight line structure matching pairs and intersection matching pairs based on curve structure matching, to obtain a panoramic image.

[0011] As a preferred solution, the step of extracting point features from the reference image and the target image includes:

[0012] Use the Scale-Invariant Feature Transform (SIFT) algorithm to obtain the SIFT feature points and corresponding feature descriptors of the reference image and the target image, and use the k-Nearest Neighbor (KNN) algorithm for rough matching of point features;

[0013] For each feature descriptor in the target image by the k-Nearest Neighbor (KNN) algorithm, find the k closest neighbor points in the reference image; calculate the Euclidean distance between the feature descriptors of the target image and the reference image, calculate the distance relationship between the nearest neighbor and the second nearest neighbor for each feature descriptor, when the distance ratio between the nearest neighbor and the second nearest neighbor satisfies the following relational expression:

[0014]

[0015] Then the corresponding matching point pairs are determined as valid matches, otherwise they are excluded, to obtain a rough matching result;

[0016] Use the Random Sample Consensus (RANSAC) algorithm to exclude the mismatched point pairs in the rough matching result, and obtain the purified point feature matching pairs;

[0017] Use the LSD method of line segment detector to detect and extract the straight-line structures of the reference image and the target image.

[0018] As a preferred solution, the steps of extracting the edge curve structures of the reference image and the target image include:

[0019] Use the Canny operator to extract the image edges of the reference image and the target image;

[0020] Perform branch separation, inflection point disconnection, and short line reconnection on the extracted image edges to obtain significant edge curve structures.

[0021] As a preferred solution, the steps of using the Canny operator to extract the image edges of the reference image and the target image include:

[0022] Perform Gaussian smoothing on the reference image and the target image to remove noise interference;

[0023] Use the Sobel operator to calculate the gradients of the reference image and the target image in the horizontal and vertical directions to obtain the edge intensity and direction information of the reference image and the target image;

[0024] Suppress the pixels in non-edge directions and retain the local maximum gradient magnitude points;

[0025] Binarize the edges using high and low thresholds to divide the pixels into strong edge, weak edge, and non-edge regions;

[0026] Based on the connectivity of the edges, connect the weak edges with the strong edges and remove the isolated edge points to obtain the corresponding image edges of the reference image and the target image.

[0027] As a preferred solution, the steps of performing branch separation, inflection point disconnection, and short line reconnection on the extracted image edges to obtain significant edge curve structures include:

[0028] Judge whether a corresponding pixel point is a bifurcation point by counting the number of edge pixel points in the 8-neighborhood of each pixel point on the image edge; the judgment conditions are:

[0029] (1) When the number of edge pixel points in the 8-neighborhood of a certain pixel point is greater than 3, determine that the corresponding pixel point is a bifurcation point;

[0030] (2) When the number of edge pixel points in the "plus" 4-neighborhood of a certain pixel point is greater than 2, determine that the corresponding pixel point is a bifurcation point;

[0031] (3) When the number of edge pixel points in the "X" 4-neighborhood of a certain pixel point is greater than 2, determine that the corresponding pixel point is a bifurcation point;

[0032] After determining the bifurcation point, separate the edge curve structure into multiple edge curve branches starting from the bifurcation point;

[0033] Calculate the maximum or minimum values of the edge curve in the longitudinal and transverse directions to obtain local extreme points, and the local extreme points are significant inflection points;

[0034] After the branch separation and inflection point disconnection processing, the edge curve structure is split into short curve structures, and short line reconnecting is performed on the short curve structures; if the endpoints of two short curve structures are both within the 8-neighborhood of each other, connect the starting points and ending points of the two short curve structures respectively, calculate the inclination angles of the two short curve structures, and if the difference in the inclination angles between the two short curve structures is within 15°, merge the corresponding two short curve structures into one long curve, thereby obtaining a significant edge curve structure.

[0035] As a preferred solution, in the step of performing neighborhood division on the extracted straight line structure and edge curve structure:

[0036] For the straight line structure, select a rectangular area of αL×βL on both sides of the straight line structure as the neighborhood corresponding to the straight line structure, and the point features within the neighborhood are neighborhood points;

[0037] For the significant edge curve structure, select a rectangular area of αL×βL on both sides of the connection line at the beginning and end of the significant edge curve structure as the neighborhood corresponding to the significant edge curve structure, and the point features within the neighborhood are neighborhood points;

[0038] Among them, L is the length of the connection line at the beginning and end of the straight line structure or the significant edge curve structure, α = 1, β = 1;

[0039] In the step of selecting point features in the neighborhood to construct projective invariants:

[0040] Select feature points in the neighborhood of the significant edge curve structure to construct feature numbers; use the cross-ratio to describe the relative position relationship of points;

[0041] For four collinear points A, B, C, D on a straight line L, the cross-ratio is calculated according to the following expression:

[0042]

[0043] For the feature numbers, let K be a field, and P m be an m-dimensional projective space over K, and P1, P2,..., P r be r different points in P m (K), and these points form a closed loop (P r+1 = P1); there are n different points on the line segment P i P i+1 ; where \(i = 1, 2, \cdots, r\); each point is linearly represented by \(P\) i and \(P\) i+1 as:

[0044]

[0045] Let the set correspond to the value of the characteristic number as:

[0046]

[0047] Select 4 point features \(P_1\), \(P_2\), \(P_3\), \(P_4\) in the neighborhood of the significant edge curve structure to construct a projective invariant: Connect points \(P_1\) and \(P_2\), which intersects the curve structure at point \(E\); Connect points \(P_1\) and \(P_4\), which intersects the curve structure at point \(F\); The straight lines \(FP_2\) and \(EP_4\) intersect at point \(K\), connect point \(P_1\) and \(K\) and intersect the curve structure at point \(N\); Connect points \(P_1\) and \(P_3\), which intersects the curve structure at point \(M\); Connect points \(P_2\) and \(P_3\), which intersects the curve structure at point \(V\); Connect points \(P_4\) and \(P_3\) and intersect the curve structure at point \(W\);

[0048] Obtain a closed-loop point set and the point set on the side of the closed-loop triangle Calculate the projective invariant constructed by the point features \(P_1\), \(P_2\), \(P_3\), \(P_4\)

[0049] For the straight line structure, use the same method to construct the projective invariant.

[0050] As a preferred solution, the calculation of the matching relationship between straight line structures and between edge curve structures, and obtaining the straight line structure matching pairs and the intersection point matching pairs based on curve structure matching includes:

[0051] For two curve structures \(a\) i and \(b\) j , where \(a\) i is the edge curve in the target image, and \(b\) j is the edge curve in the reference image; if there is no matching relationship or the number of matching pairs is less than 4 for the point features in a certain neighborhood of \(a\) i and \(b\) j , then it is considered that \(a\) i and \(b\) j do not match; conversely, if \(a\) i and \(b\) jIf the number of matching point features in a certain neighborhood is greater than or equal to 4, then 4 pairs of matching point features are used to construct projection invariants within their respective neighborhoods; specifically, first, 3 point features are extracted as a base point set for constructing a closed triangle with the curve structure, and then 1 point is extracted from the remaining point features as a sampling point; new points are constructed on the edge of the closed triangle, and thus projection invariants constructed by different sampling points are calculated for each group of base point sets and where i = 1, 2,..., m r , m r is the number of projection invariants corresponding to the r-th group of base point sets; for the r-th group of base point sets, using the projection invariants constructed by the same point feature matching pairs, calculate the similarity between the base point sets:

[0052]

[0053] Take the median of sim(i) as the similarity of the r-th group of base point sets:

[0054] s(r) = median(sim(i))

[0055] Determine two curve structures a i and b j The maximum value of the similarities of all base point sets between them:

[0056] SIM(a i , b j ) = max(s(r)), r = 1, 2,..., M

[0057] where M is the number of base point sets;

[0058] According to the above steps, obtain the similarities between all curve structures in the reference image and the target image; traverse all similarities, and take the curve structure with the highest similarity as the matching pair; for the straight line structure, the matching process is the same as that of the curve structure;

[0059] Construct intersection point feature matching pairs based on curve structures: According to the obtained matching relationship between curve structures, use the same point feature matching pairs to construct intersection point feature pairs; assume that curve structures a i and b j are a pair of matching curves, then combined with the calculated condition of the similarity of the base point sets sim(i) > 0.9, the intersection points on the curve structures constructed by the same base point sets are used as new intersection point feature matching pairs.

[0060] As a preferred solution, the step of dividing the reference image and the target image into grids, and generating a distorted image aligned with the reference image based on the point feature matching pairs, the straight line structure matching pairs, and the intersection matching pairs based on curve structures to obtain a panoramic image includes:

[0061] Divide the reference image and the target image into grids, and use the four vertices v1, v2, v3, v4 of the grid to represent the coordinates of any point p inside the grid by bilinear interpolation. The expression is as follows:

[0062]

[0063] Construct a grid deformation constraint energy term based on point feature matching pairs, line structure matching pairs, and intersection matching pairs based on curve structure matching:

[0064] Use the point feature matching pairs {p i , p i ′}, i = 1, 2,..., N to construct a point feature alignment constraint energy term, where N is the number of point feature pairs:

[0065]

[0066] Use the line structure matching pairs {l i , l i ′}, i = 1, 2,..., K to construct a line feature alignment constraint energy term. By equally sampling at intervals on the line structure and according to the rule that the sampled points after distortion are located on the corresponding matching lines, construct a line feature alignment constraint energy term:

[0067]

[0068] In the formula, K is the number of line structure matching pairs, is the normal vector of the k-th line;

[0069] Use the intersection matching pairs {k i , k i ′}, i = 1, 2,..., M of the curve structure matching pairs {c j , c j ′}, j = 1, 2,..., N i to construct a curve feature alignment constraint energy term:

[0070]

[0071] In the formula, M is the number of curve structure matching pairs, and N i is the number of intersection feature matching pairs on the i-th pair of curves;

[0072] Construct a global energy function and assign weights to each energy term:

[0073]

[0074] The global energy function is solved by the non - linear least - squares optimization method to obtain the positions of the mesh vertices after deformation; the target image is interpolated and mapped according to the optimization result to generate a warped image aligned with the reference image, thus obtaining a panoramic image.

[0075] In a second aspect, a panoramic image stitching system based on multi - feature guidance is provided, including:

[0076] A feature extraction module, configured to extract point features, line structures, and edge curve structures from the reference image and the target image respectively;

[0077] A geometric structure matching module, configured to divide the neighborhoods of the extracted line structures and edge curve structures, select point features within the neighborhoods to construct projective invariants, calculate the matching relationships between the line structures and between the edge curve structures, and obtain line structure matching pairs and intersection matching pairs based on curve structure matching;

[0078] A mesh deformation module, configured to divide the reference image and the target image into meshes, and generate a warped image aligned with the reference image according to the point feature matching pairs, line structure matching pairs, and intersection matching pairs based on curve structure matching, thus obtaining a panoramic image.

[0079] In a third aspect, a computer - readable storage medium is provided, in which at least one instruction is stored, and the at least one instruction is executed by a processor in an electronic device to implement the panoramic image stitching method based on multi - feature guidance.

[0080] Compared with the prior art, the present invention has at least the following beneficial effects:

[0081] The present invention divides the neighborhood of the extracted straight-line structures and edge curve structures, selects point features within the neighborhood to construct projection invariants, uses the projection invariants to measure the similarity between straight-line structures and between edge curve structures, calculates the matching relationships between straight-line structures and between edge curve structures, obtains straight-line structure matching pairs and intersection matching pairs based on curve structure matching, then divides the reference image and the target image into grids, and generates a warped image aligned with the reference image according to the point feature matching pairs, straight-line structure matching pairs, and intersection matching pairs based on curve structure matching, thereby obtaining a panoramic image. Aiming at the problem of insufficient extraction of point features and line features in low-texture images, the present invention utilizes richer edge features in the image, especially irregular edges, for matching and alignment. Irregular edges can reflect geometric characteristics such as the contour and shape of an object, and are often more stable than point features and line features in low-texture regions. The panoramic image stitching method based on multi-feature guidance of the present invention combines points, straight lines, and edge curve structures, fully utilizes point features, straight-line features, and irregular edge curve features, so that in the low-texture image scenario, through the matching constraint of irregular edge curves, the problems of insufficient point features and straight-line features can be effectively compensated, and the alignment quality and stitching accuracy of image stitching can be significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present invention, and those of ordinary skill in the art can also obtain other related drawings without creative efforts based on these drawings.

[0083] Figure 1 Flowchart of the panoramic image stitching method based on multi-feature guidance in the embodiment of the present invention;

[0084] Figure 2 Flowchart of extracting and matching point features of the reference image and the target image in the embodiment of the present invention;

[0085] FIG. 3(a) Schematic diagram of branch separation when extracting significant edge curve structures of the reference image and the target image in the embodiment of the present invention;

[0086] FIG. 3(b) Schematic diagram of inflection point disconnection when extracting significant edge curve structures of the reference image and the target image in the embodiment of the present invention;

[0087] FIG. 3(c) Schematic diagram of short-line reconnecting when extracting significant edge curve structures of the reference image and the target image in the embodiment of the present invention;

[0088] FIG. 4(a) Schematic diagram of neighborhood division of straight-line structures in the embodiment of the present invention;

[0089] Figure 4(b) Schematic diagram of neighborhood division of the significant edge curve structure in the embodiment of the present invention;

[0090] Figure 5 Schematic diagram of constructing projection invariants on the significant edge curve structure in the embodiment of the present invention. Detailed implementation manners

[0091] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, those of ordinary skill in the art can also obtain other embodiments without creative efforts.

[0092] Please refer to Figure 1 , an embodiment of the present invention proposes a panoramic image stitching method based on multi-feature guidance, mainly including:

[0093] Extract point features, straight line structures, and edge curve structures from the reference image and the target image respectively;

[0094] Perform neighborhood division on the extracted straight line structures and edge curve structures, select point features in the neighborhood to construct projection invariants, calculate the matching relationships between the straight line structures and between the edge curve structures, and obtain straight line structure matching pairs and intersection matching pairs based on curve structure matching;

[0095] Perform grid division on the reference image and the target image, and generate a warped image aligned with the reference image according to the point feature matching pairs, straight line structure matching pairs, and intersection matching pairs based on curve structure matching, so as to obtain a panoramic image.

[0096] In a possible implementation, the panoramic image stitching method based on multi-feature guidance in the embodiments of the present invention uses the Scale-Invariant Feature Transform (SIFT) algorithm to extract point features, adopts the k-Nearest Neighbor (KNN) algorithm for rough matching of point features, and uses the Global Random Sample Consensus (RANSAC) algorithm to purify the point feature matching pairs. The present invention uses the Line Segment Detector (LSD) method to extract straight line structures in images on a large scale. The present invention uses the Canny operator to extract edges from images, and performs branch separation, inflection point disconnection, and short line reconnecting on the extracted edges to obtain significant edge curve structures. When matching geometric structures, the present invention uses point feature matching pairs to construct projection invariants in the neighborhood of geometric structures, calculates the similarity between invariants, and respectively obtains the matching relationships between straight line structures and between irregular edge curve structures, and further constructs intersection feature pairs based on curve structures. The reference image and the target image are divided into grids, and point features can be bilinearly interpolated and represented by the four vertices of the grid where they are located. On the basis of grid division, a grid distortion energy term is constructed, including a point feature alignment term, a line feature alignment term, and an edge curve alignment term. Finally, the grid deformation and warping model is obtained by minimizing the energy term to complete panoramic image stitching.

[0097] Please refer to Figure 2 , in a possible implementation, the steps of extracting point features from the reference image and the target image include:

[0098] Step101: To ensure the effective detection and description of significant points in the image, the Scale-Invariant Feature Transform (SIFT) algorithm is used to obtain the SIFT feature points and corresponding feature descriptors of the reference image and the target image. The Scale-Invariant Feature Transform (SIFT) algorithm has scale invariance, rotation invariance, and certain illumination invariance, and can stably detect key points in images with different scales, perspectives, or illumination changes, and generate highly discriminative feature descriptors.

[0099] Step102: After obtaining the SIFT feature points and corresponding descriptors of the reference image and the target image, the KNN (k-Nearest Neighbor) algorithm is used for rough matching of point features. The KNN algorithm is to find the k closest neighbor points in the reference image for each feature descriptor in the target image (k = 2 is set in the embodiments of the present invention). Specifically, first calculate the Euclidean distance between the feature descriptors of the target image and the reference image, and calculate the distance relationship between the nearest neighbor and the second nearest neighbor for each descriptor. When the distance ratio between the nearest neighbor and the second nearest neighbor satisfies the following condition:

[0100]

[0101] Then the corresponding matching point pairs are determined as valid matches, otherwise they are excluded to obtain the rough matching result.

[0102] Step103: To further eliminate the mis-matched point pairs in the rough matching results and improve the global consistency of the matching, the RANSAC (Random Sample Consensus) algorithm is used to purify the point feature matching pairs. The introduction of the RANSAC algorithm effectively eliminates the incorrect matching point pairs, ensuring the global accuracy and robustness of the point feature matching. By applying the RANSAC algorithm to the results of the rough matching of the KNN algorithm, the purity of the matching pairs can be significantly improved, and a reliable set of point feature matching pairs can be obtained.

[0103] In a possible implementation, the steps of extracting the edge curve structure of the reference image and the target image include:

[0104] Step301: Use the Canny operator to extract the image edges of the reference image and the target image. The Canny operator is a classic edge detection method with strong noise suppression ability and edge positioning accuracy. The specific process is as follows:

[0105] First, perform Gaussian smoothing on the reference image and the target image to remove noise interference and avoid the influence of noise on the edge detection results; then, use the Sobel operator to calculate the gradients of the image in the horizontal and vertical directions to obtain the edge strength and direction information of the image; further, suppress the pixels in the non-edge directions and retain the local maximum gradient amplitude points to make the edges clearer; use high and low thresholds to binarize the edges, dividing the pixels into strong edges, weak edges, and non-edge regions; finally, based on the connectivity of the edges, connect the weak edges with the strong edges and remove the isolated edge points to obtain the edge images corresponding to the reference image and the target image.

[0106] Step302: There are many bifurcations and pseudo-edges in the obtained edge curve structure, and the initial edges need to be further optimized. The optimization of the initial edge curve structure includes three steps: branch separation, inflection point disconnection, and short line reconnection.

[0107] As shown in Figure 3(a), it is judged whether a point is a bifurcation point by counting the number of edge pixel points in the 8-neighborhood of each pixel point on the edge. The judgment conditions are: (1) When the number of edge pixel points in the 8-neighborhood of a certain edge point is greater than 3, it is determined that this point is a bifurcation point; (2) When the number of edge pixel points in the "cross" 4-neighborhood of a certain edge point is greater than 2, it is determined that this point is a bifurcation point; (3) When the number of edge pixel points in the "X" 4-neighborhood of a certain edge point is greater than 2, it is determined that this point is a bifurcation point.

[0108] After determining the bifurcation points, separate the edge curve structure into multiple edge curve branches from the bifurcation points.

[0109] As shown in Fig. 3(b), in the edge curve structure after branch separation, there is a part of the curve with a large degree of bending, which will affect the accuracy of subsequent curve structure matching. Sample multi-scale templates to calculate the maximum or minimum values in the longitudinal and transverse directions of the edge curve, and obtain stable local extreme points. These extreme points are the significant inflection points. Further disconnect these curves with a high degree of folding. After branch separation and inflection point disconnection processing, the edge curve structure will be split into short curve structures. Further reconnect the short curves to ensure the continuity of the significant edge curves.

[0110] As shown in Fig. 3(c), if the endpoints of two edge curves are both within the 8-neighborhood of each other, connect the starting points and ending points of the two edge curves respectively, and calculate the approximate inclination angles of the two curves. If the difference in the inclination angles between the two curves is within 15°, it is determined that these two curve structures can be merged into one long curve.

[0111] Through the above steps, the significant edge curve structures of the reference image and the target image are obtained.

[0112] In a possible implementation manner, in the step of performing neighborhood division on the extracted straight line structure and edge curve structure:

[0113] As shown in Fig. 4(a), for the straight line structure, select a rectangular area of αL×βL on both sides of the straight line structure as the neighborhood corresponding to the straight line structure, and the point features in the neighborhood are neighborhood points;

[0114] As shown in Fig. 4(b), for the significant edge curve structure, select a rectangular area of αL×βL on both sides of the connection line at the beginning and end of the significant edge curve structure as the neighborhood corresponding to the significant edge curve structure, and the point features in the neighborhood are neighborhood points;

[0115] Wherein, L is the length of the connection line at the beginning and end of the straight line structure or the significant edge curve structure, α = 1, β = 1;

[0116] In the step of selecting point features in the neighborhood to construct projective invariants:

[0117] Select the feature points in the neighborhood of the significant edge curve structure to construct feature numbers. Feature numbers are projective invariants newly developed on the basis of "cross-ratio". Cross-ratio is an important invariant in projective geometry. Its definition is based on the projective transformation characteristics of points, that is, under projective transformation, the cross-ratio value remains unchanged. This characteristic makes the cross-ratio a very useful measure for describing the relative position relationship of points. For four collinear points A, B, C, D on a straight line L, the cross-ratio value is calculated according to the following expression:

[0118]

[0119] For feature numbers, let K be a field, Pm is an m-dimensional projection space on K, P1,P2,…,P r YesP m (K) has r different points, which form a closed loop (P r+1 =P1); on line segment P i P i+1 There are n different points on Where i = 1, 2, ..., r; each point By P i and P i+1 The linear representation is:

[0120]

[0121] Let Collection The corresponding characteristic number value is:

[0122]

[0123] See also Figure 5 , select four point features P1, P2, P3, and P4 in the neighborhood of the significant edge curve structure to construct the projection invariant: connect points P1 and P2, intersecting with the curve structure at point E; connect points P1 and P4, intersecting with the curve structure at point F; straight lines FP2 and EP4 intersect at point K, connect points P1 and K and intersect with the curve structure at point N; connect points P1 and P3, intersecting with the curve structure at point M; connect points P2 and P3, intersecting with the curve structure at point V; connect points P4 and P3 and intersect with the curve structure at point W;

[0124] Get a closed loop point set And the points on the edge of the closed triangle Calculate the projection invariant constructed by point features P1, P2, P3, P4

[0125] For the straight line structure, the same method is used to construct the projection invariant.

[0126] In a possible implementation, calculating the matching relationship between straight line structures and edge curve structures to obtain straight line structure matching pairs and intersection matching pairs based on curve structure matching includes:

[0127] For two curve structures a i and b j , where a i is the edge curve in the target image, b j is the edge curve in the reference image; if a i and b j If there is no matching relationship between the point features in a certain neighborhood of a or the number of matching pairs is less than 4, then it is considered that ai and b j do not match; conversely, if a i and b j The number of matching point features in a certain neighborhood of is greater than or equal to 4, then use 4 pairs of matching point features to construct projection invariants within their respective neighborhoods; specifically, first extract 3 point features as the base point set for constructing a closed triangle with the curve structure, and then extract 1 point from the remaining point features as the sampling point; construct new points on the edge of the closed triangle, and thus calculate the projection invariants constructed by different sampling points for each group of base point sets and where i = 1, 2, …, m r , m r is the number of projection invariants corresponding to the r-th group of base point sets; for the r-th group of base point sets, calculate the similarity between the base point sets using the projection invariants constructed by the same point feature matching pairs:

[0128]

[0129] To reduce the influence of accidental factors, take the median of sim(i) as the similarity of the r-th group of base point sets:

[0130] s(r) = median(sim(i))

[0131] Determine the maximum value of the similarities of all base point sets between two curve structures a i and b j :

[0132] SIM(a i , b j ) = max(s(r)), r = 1, 2, …, M

[0133] where M is the number of base point sets;

[0134] Obtain the similarities between all curve structures in the reference image and the target image according to the above steps; traverse all similarities and take the curve structure with the highest similarity as the matching pair; for the straight line structure, the matching process is the same as that of the curve structure;

[0135] Construct the intersection point feature matching pairs based on the curve structure:

[0136] According to the obtained matching relationship between the curve structures, construct the intersection point feature pairs using the same point feature matching pairs; assume that the curve structures a i and b j are a pair of matching curves, then combined with the calculated base point set similarity condition sim(i) > 0.9, take the intersection points on the curve structures constructed by the same base point set (such as Figure 5 the E and F points in) as the new intersection point feature matching pairs.

[0137] In a possible implementation, the steps of dividing the reference image and the target image into grids, generating a warped image aligned with the reference image based on point feature matching pairs, line structure matching pairs, and intersection matching pairs based on curve structure matching, and obtaining a panoramic image include:

[0138] Divide the reference image and the target image into grids, and use the four vertices v1, v2, v3, v4 of the grid to represent the coordinates of any point p within the grid by bilinear interpolation. The expression is as follows:

[0139]

[0140] Construct a grid deformation constraint energy term based on point feature matching pairs, line structure matching pairs, and intersection matching pairs based on curve structure matching:

[0141] Use the obtained point feature matching pairs {p i , p i ′}, i = 1, 2,..., N to construct a point feature alignment constraint energy term to ensure that after the grid is deformed, the point features in the reference image are precisely aligned with the corresponding point features in the target image, where N is the number of point feature pairs:

[0142]

[0143] Use the line structure matching pairs {l i , l i ′}, i = 1, 2,..., K to construct a line feature alignment constraint energy term. By equally spacing sampling on the line structure and according to the rule that the sampled points after warping are located on the corresponding matching lines, construct a line feature alignment constraint energy term:

[0144]

[0145] In the formula, K is the number of line structure matching pairs, is the normal vector of the k-th line;

[0146] Use the intersection matching pairs {k i , k i ′}, j = 1, 2,..., N of the curve structure matching pairs {c j , c j ′}, i = 1, 2,..., M i to construct a curve feature alignment constraint energy term:

[0147]

[0148] In the formula, M is the number of curve structure matching pairs, N i ​is the number of intersection feature matching pairs on the i-th pair of curves;

[0149] Taking into account the above constraints and the smoothness of the mesh deformation, a global energy function is constructed and weights are assigned to each energy term:

[0150]

[0151] The global energy function is solved by the non-linear least squares optimization method to obtain the positions of the deformed mesh vertices; according to the optimization results, interpolation mapping is performed on the target image to generate a distorted image aligned with the reference image, and a panoramic image is obtained.

[0152] Another embodiment of the present invention also proposes a panoramic image stitching system based on multi-feature guidance, including:

[0153] A feature extraction module for respectively extracting point features, straight line structures, and edge curve structures from the reference image and the target image;

[0154] A geometric structure matching module for partitioning the neighborhoods of the extracted straight line structures and edge curve structures, selecting point features within the neighborhoods to construct projective invariants, calculating the matching relationships between the straight line structures and between the edge curve structures, and obtaining straight line structure matching pairs and intersection matching pairs based on curve structure matching;

[0155] A mesh deformation module for partitioning the reference image and the target image into meshes, and generating a distorted image aligned with the reference image according to the point feature matching pairs, straight line structure matching pairs, and intersection matching pairs based on curve structure matching, to obtain a panoramic image.

[0156] Another embodiment of the present invention also proposes a computer-readable storage medium, in which at least one instruction is stored, and the at least one instruction is executed by a processor in an electronic device to implement the multi-feature-guided panoramic image stitching method.

[0157] Exemplarily, the instructions stored in the memory can be divided into one or more modules / units, and the one or more modules / units are stored in the computer-readable storage medium and executed by the processor to complete the multi-feature-guided panoramic image stitching method of the present invention. The one or more modules / units can be a series of computer-readable instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the server.

[0158] The electronic device may be a computing device such as a smart phone, a notebook, a palmtop computer, and a cloud server. The electronic device may include, but is not limited to, a processor and a memory. Those skilled in the art can understand that the electronic device may further include more or fewer components, or combine certain components, or different components. For example, the electronic device may further include input / output devices, network access devices, a bus, etc.

[0159] The processor may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0160] The memory may be an internal storage unit of the server, such as the hard disk or memory of the server. The memory may also be an external storage device of the server, such as a plug-in hard disk equipped on the server, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory may also include both the internal storage unit and the external storage device of the server. The memory is used to store the computer-readable instructions and other programs and data required by the server. The memory may also be used to temporarily store data that has been output or is to be output.

[0161] It should be noted that for the information interaction, execution process, etc. between the above module units, since they are based on the same concept as the method embodiment, for their specific functions and the technical effects brought, reference may be specifically made to the method embodiment section, and details are not elaborated here.

[0162] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working processes of the units and modules in the above system can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0163] If the above integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the above method embodiments of this application, a computer program can be used to instruct the relevant hardware to complete. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the photographing device / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disc, etc.

[0164] In the above embodiments, the descriptions of the various embodiments have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0165] The above embodiments are only used to illustrate the technical solutions of this application, rather than to limit them; although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of this application, and should all be included in the protection scope of this application.

Claims

1. A panoramic image stitching method based on multi-feature guidance, characterized in that: include: Extract point features, straight line structures and edge curve structures from the reference image and the target image respectively; The extracted straight line structure and edge curve structure are divided into neighborhoods, point features are selected in the neighborhood to construct projection invariants, the matching relationships between straight line structures and edge curve structures are calculated, and straight line structure matching pairs and intersection matching pairs based on curve structure matching are obtained; The reference image and the target image are meshed, and a warped image aligned with the reference image is generated according to point feature matching pairs, straight line structure matching pairs, and intersection matching pairs based on curve structure matching to obtain a panoramic image.

2. The panoramic image stitching method based on multi-feature guidance according to claim 1, characterized in that: The steps of extracting point features from the reference image and the target image include: The SIFT algorithm is used to obtain the SIFT feature points and corresponding feature descriptors of the reference image and the target image, and the k-nearest neighbor KNN algorithm is used to perform rough matching of point features. The k-nearest neighbor KNN algorithm finds the k closest neighbor points in the reference image for each feature descriptor in the target image; calculates the Euclidean distance between the feature descriptors of the target image and the reference image, and calculates the distance relationship between the nearest neighbor and the next nearest neighbor for each feature descriptor. When the distance ratio between the nearest neighbor and the next nearest neighbor satisfies the following relationship: The corresponding matching point pairs are judged as valid matches, otherwise they are eliminated to obtain a rough matching result; The random sampling consistency RANSAC algorithm is used to eliminate the mismatched point pairs in the rough matching results to obtain the purified point feature matching pairs; The line segment detector (LSD) method is used to detect and extract straight line structures in the reference image and the target image.

3. The panoramic image stitching method based on multi-feature guidance according to claim 1, characterized in that: The steps of extracting edge curve structures from the reference image and the target image include: Use the Canny operator to extract image edges from the reference image and the target image; The extracted image edges are branched, inflection points are disconnected, and short lines are reconnected to obtain significant edge curve structures.

4. The panoramic image stitching method based on multi-feature guidance according to claim 3, characterized in that: The step of using the Canny operator to extract image edges from the reference image and the target image comprises: Perform Gaussian smoothing on the reference image and the target image to remove noise interference; The Sobel operator is used to calculate the gradients of the reference image and the target image in the horizontal and vertical directions to obtain the edge strength and direction information of the reference image and the target image; Suppress pixels in non-edge directions and retain local maximum gradient amplitude points; Use high threshold and low threshold to binarize the edge and divide the pixels into strong edge, weak edge and non-edge area; Based on the connectivity of the edge, the weak edge is connected with the strong edge, and the isolated edge points are removed to obtain the image edge corresponding to the reference image and the target image.

5. The panoramic image stitching method based on multi-feature guidance according to claim 3, characterized in that: The step of performing branch separation, inflection point disconnection and short line reconnection on the extracted image edge to obtain a significant edge curve structure comprises: By counting the number of edge pixels in the 8-neighborhood of each pixel on the edge of the image, it is determined whether the corresponding pixel is a bifurcation point; the judgment condition is: (1) When the number of edge pixels in the 8-neighborhood of a certain pixel is greater than 3, the corresponding pixel is determined to be a bifurcation point; (2) When the number of edge pixels in the "cross" 4 neighborhood of a certain pixel is greater than 2, the corresponding pixel is determined to be a bifurcation point; (3) When the number of edge pixels in the "X"-shaped 4-neighborhood of a certain pixel is greater than 2, the corresponding pixel is determined to be a bifurcation point; After the bifurcation point is determined, the edge curve structure is separated into a plurality of edge curve branches from the bifurcation point; Calculate the maximum or minimum value of the edge curve in the longitudinal and transverse directions to obtain a local extreme point, which is a significant inflection point; After branch separation and inflection point disconnection, the edge curve structure is split into short curve structures, and the short curve structures are reconnected by short lines; if the endpoints of the two short curve structures are within the 8-neighborhood of each other, the starting points and end points of the two short curve structures are connected respectively, and the inclination angles of the two short curve structures are calculated. If the difference in inclination angle between the two short curve structures is within 15°, the corresponding two short curve structures are merged into a long curve, thereby obtaining a significant edge curve structure.

6. The panoramic image stitching method based on multi-feature guidance according to claim 3, characterized in that: In the step of dividing the extracted straight line structure and edge curve structure into neighborhoods: For the straight line structure, the rectangular area of ​​αL×βL on both sides of the straight line structure is selected as the neighborhood of the corresponding straight line structure, and the point features in the neighborhood are neighborhood points; For the significant edge curve structure, the rectangular area of ​​αL×βL on both sides of the end connection line of the significant edge curve structure is selected as the neighborhood of the corresponding significant edge curve structure, and the point features in the neighborhood are neighborhood points; Wherein, L is the length of the connecting line from the beginning to the end of the straight line structure or the significant edge curve structure, α=1, β=1; In the step of selecting point features in the neighborhood to construct a projection invariant: Select feature points in the neighborhood of significant edge curve structure to construct feature numbers; use cross ratio to describe the relative position relationship of points; For four collinear points A, B, C, and D on a straight line L, the intersection ratio is calculated according to the following expression: For the number of features, let K be a field, P m is an m-dimensional projection space on K, P1,P2,…,P r YesP m (K) has r different points, which form a closed loop (P r+1 =P1); on line segment P i P i+1 There are n different points on Where i = 1, 2, ..., r; each point By P i and P i+1 The linear representation is: Let Collection The corresponding characteristic number value is: Four point features P1, P2, P3, and P4 are selected in the neighborhood of the significant edge curve structure to construct the projection invariant: connecting points P1 and P2, intersecting with the curve structure at point E; connecting points P1 and P4, intersecting with the curve structure at point F; straight lines FP2 and EP4 intersecting at point K, connecting points P1 and K intersecting with the curve structure at point N; connecting points P1 and P3, intersecting with the curve structure at point M; connecting points P2 and P3, intersecting with the curve structure at point V; connecting points P4 and P3 intersecting with the curve structure at point W; Get a closed loop point set And the points on the edge of the closed triangle Compute the projection invariant constructed by point features P1, P2, P3, P4 For the straight line structure, the same method is used to construct the projection invariant.

7. The panoramic image stitching method based on multi-feature guidance according to claim 6, characterized in that: The calculating the matching relationship between the straight line structures and the edge curve structures to obtain the straight line structure matching pairs and the intersection matching pairs based on the curve structure matching includes: For two curve structures a i and b j , where a i is the edge curve in the target image, b j is the edge curve in the reference image; if a i and b j If there is no matching relationship between the point features in a certain neighborhood of a or the number of matching pairs is less than 4, then it is considered that a i and b j does not match; otherwise, if a i and b j If the number of matching point features in a neighborhood of is greater than or equal to 4, then use 4 pairs of matching point features to construct projection invariants in their respective neighborhoods; specifically, first extract 3 point features as the base point set to construct a closed triangle with the curve structure, and then extract 1 point from the remaining point features as a sampling point; construct a new point on the edge of the closed triangle, thereby calculating the projection invariant constructed by different sampling points for each set of base point sets and Where i = 1, 2, ..., m r ,m r is the number of projection invariants corresponding to the rth base point set; for the rth base point set, the projection invariants constructed by matching pairs of the same point features are used to calculate the similarity between the base point sets: Take the median of sim(i) as the similarity of the rth group of base point sets: s(r)=median(sim(i)) Determine the two curve structures a i and b j The maximum similarity of all base point sets between: SIM(a i ,b j )=max(s(r)),r=1,2,...,M Among them, M is the number of base point sets; According to the above steps, the similarities between all curve structures in the reference image and the target image are obtained; all similarities are traversed, and the curve structure with the highest similarity is taken as a matching pair; for the straight line structure, the matching process is the same as that of the curve structure; Construct intersection feature matching pairs based on curve structure: According to the matching relationship between the obtained curve structures, the intersection feature pairs are constructed using the same point feature matching pairs; assuming that the curve structure a i and b j If it is a pair of matching curves, then combined with the calculated base point set similarity condition sim(i)>0.9, the intersection points on the curve structure constructed by the same base point set are taken as new intersection feature matching pairs.

8. The panoramic image stitching method based on multi-feature guidance according to claim 7, characterized in that: The step of dividing the reference image and the target image into grids, generating a distorted image aligned with the reference image according to point feature matching pairs, straight line structure matching pairs, and intersection matching pairs based on curve structure matching, and obtaining a panoramic image comprises: The reference image and the target image are meshed, and the coordinates of any point p in the mesh are expressed as follows using bilinear interpolation of the four vertices v1, v2, v3, and v4 of the mesh: According to the point feature matching pairs, straight line structure matching pairs and intersection matching pairs based on curve structure matching, the mesh deformation constraint energy term is constructed: Use point feature matching to {p i ,p i ′}, i = 1, 2, ..., N constructs the point feature alignment constraint energy term, where N is the number of point feature pairs: Use straight line structure matching to {l i ,l i ′}, i = 1, 2, ..., K constructs the line feature alignment constraint energy term, by sampling at equal intervals on the straight line structure, according to the rule that the sampling points after distortion are located on the corresponding matching straight line, constructs the line feature alignment constraint energy term: In the formula, K is the number of straight line structure matching pairs, is the normal vector of the kth line; Using curve structure matching i ,c i ′}, i=1,2,...,M intersection matching pairs {k j ,k j ′},j=1,2,...,N i Construct the curve feature alignment constraint energy term: Where M is the number of curve structure matching pairs, N i is the number of matching pairs of intersection features on the i-th pair of curves; Construct a global energy function and assign weights to each energy term: The global energy function is solved by the nonlinear least squares optimization method to obtain the deformed mesh vertex positions; the target image is interpolated and mapped according to the optimization result to generate a distorted image aligned with the reference image to obtain a panoramic image.

9. A panoramic image stitching system based on multi-feature guidance, characterized in that: include: A feature extraction module is used to extract point features, straight line structures and edge curve structures of the reference image and the target image respectively; The geometric structure matching module is used to divide the extracted straight line structure and edge curve structure into neighborhoods, select point features in the neighborhood to construct projection invariants, calculate the matching relationship between straight line structures and edge curve structures, and obtain straight line structure matching pairs and intersection matching pairs based on curve structure matching; The grid deformation module is used to grid the reference image and the target image, and generate a distorted image aligned with the reference image according to point feature matching pairs, straight line structure matching pairs and intersection matching pairs based on curve structure matching to obtain a panoramic image.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, and the at least one instruction is executed by a processor in an electronic device to implement the panoramic image stitching method based on multi-feature guidance as described in any one of claims 1 to 8.

Images