An Alternative Iterative Line Matching and Image Stitching Method

Abstract

The present invention belongs to the field of computer vision and relates to an alternating iterative line matching and image stitching method. After using an existing algorithm for pre-alignment based on matching feature points, first, a global line-guided grid deformation is performed to obtain a preliminary stitching result; second, a new line matching method that does not depend on descriptors is used to obtain line matching pairs; then, the line matching pairs are used to obtain point matching pairs based on the invariant feature number, and the line matching pairs and the point matching pairs return to the previous steps again to jointly guide the global line-guided grid deformation. Global line-guided grid deformation and line-point matching are continuously and alternately iterated. Finally, the optimal line-point matching and the best stitching result are obtained, which can generate more accurate matching feature pairs, show clearer textures, and retain significant natural structures in the stitched image, with effects that surpass traditional image stitching methods and the latest deep learning-based methods.

Description

Technical Field

[0001] The invention belongs to the field of computer vision, and relates to line matching and image stitching, and in particular to an iterative image stitching method and a method for line matching that does not rely on descriptors, specifically an alternating iterative line matching and image stitching method. Background Art

[0002] Image stitching is an important research direction in computer vision. Through image stitching technology, multiple limited-angle or narrow-angle images collected can be synthesized into wide-angle stitching images, which not only ensures high resolution of the image, but also achieves low cost. It has been widely used in many fields such as photogrammetry and remote sensing, emergency rescue, security monitoring, virtual reality, scene understanding, medical imaging and military reconnaissance. Straight line features are an important type of image features, which are very rich in common life scenes, such as architectural images. Therefore, they are widely used in many fields such as target positioning, navigation, and 3D reconstruction.

[0003] Image stitching methods are divided into traditional methods and deep learning methods. Traditional image stitching methods usually estimate an optimal global transformation for each input image. These methods are only applicable to ideal coplanar scenes, and the generated images are often affected by local artifacts and projection distortion. For large parallax scenes, such methods are not flexible enough and it is difficult to effectively reduce distortion. Image stitching methods based on deep learning are also limited by the coplanarity assumption of the scene, but have advantages in feature extraction and can generate more robust and dense matching features than traditional methods; however, existing image stitching methods based on deep learning can only estimate the global transformation through the correlation of overlapping areas, and lack constraints on non-overlapping areas, which leads to misalignment and artifacts in the stitched images.

[0004] The more common line matching methods include those based on image texture information and those based on geometric information. Among traditional methods, the mean-standard deviation line descriptor MSLD (Y. Zhang, H. Yang, and X. Liu, "A linematching method based on local and global appearance," in 2011 4thInternational Congress on Image and Signal Processing, vol. 3, 2011, pp.1381–1385.) uses the mean and standard deviation near the line to solve the problem that the local areas of two corresponding lines are usually dissimilar due to changes in scale and viewpoint, and the line feature descriptors have large differences. However, it itself relies on descriptor implementation and has the disadvantages of limited local features and sensitivity to perspective and scale changes. Among deep learning-based methods, Pautrat et al. (R. Pautrat, J.-T. Lin, V. Larsson, MR Oswald, and M. Pollefeys, "Sold2:Self-supervised occlusion-aware line description and detection," in 2021IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 11363–11373.) uses a convolutional neural network (CNN) structure to explore line features and proposes a self-supervised occlusion-aware line description and detection method (SOLD 2 ), but it has the disadvantages of difficulty in capturing global geometric relationships and poor performance in low-texture areas; both traditional and deep learning methods have the disadvantages of relying on descriptors, low feature matching accuracy, and inconsistency caused by perspective changes. Summary of the invention

[0005] To solve the above problems, the present invention proposes an alternating iterative line matching and image stitching method. After using the existing algorithm for pre-alignment based on matching feature points, first, a global line-guided grid deformation is performed to obtain a preliminary stitching result; secondly, a new line matching method that does not depend on descriptors is used to obtain line matching pairs; then, the line matching pairs are used to obtain point matching pairs based on the invariant feature number, and the line matching pairs and point matching pairs return to the previous steps again to jointly guide the global line-guided grid deformation. Global line-guided grid deformation and line-point matching are continuously iterated alternately (line-point matching provides a robust anchor point for image stitching, and image stitching provides a more consistent image structure for line matching). Finally, the optimal line-point matching and the best stitching result are obtained. The result can generate more accurate matching feature pairs, show clearer textures, and retain the significant natural structure in the stitched image, and the effect exceeds the traditional image stitching method and the latest deep learning-based method.

[0006] The technical solution of the present invention:

[0007] An alternating iterative line matching and image stitching method comprises the following steps:

[0008] Step 1. Pre-registration based on matching feature points

[0009] The existing SURF (Speeded-Up Robust Features) algorithm is used to detect and match feature points of the two original images, and then the target image is calculated. and reference image The homography matrix between them makes them preliminarily registered and obtains the pre-registered image, as follows:

[0010] Detect and match feature points of two original images to obtain pairs of matching points ,in represents the number of matching point pairs, represents the number of paired matching points, and Represents the target image and reference image The homogeneous coordinates of the midpoint, 1 is the homogeneous term setting when the feature point coordinates are converted to homogeneous coordinates. Calculate the global homography of the point feature based on the coordinates of the matching point ;

[0011]

[0012] in, represents the initial homography, for The vector representation of Indicate point In the homography matrix The mapping result (i.e. the transformed position) under the action of for The vector representation of Represents the Euclidean norm of a vector. Minimized using singular value decomposition is 0, which means the point is Completely meet the constraints of the homography matrix. In the optimization process, the goal is to minimize , that is, minimize the geometric error of all point pairs, thereby estimating the best .

[0013] Step 2. Global line-guided mesh deformation

[0014] The pre-registered image and matching point pairs obtained in step 1 are used as input, and a total energy function based on global line guidance is used for image warping through a mesh deformation method to maintain the local and global linear structure of the stitched image. The total energy function contains line preservation terms, point-line alignment terms, and distortion terms. The details are as follows:

[0015] Step 2-1 Build a rectangular network for each image, original vertex coordinate vector , the distorted vertex coordinate vector is , Represents the mesh vertex index. Each feature point in the image Bilinear interpolation function of can pass through the four vertices of the closed mesh , , and express;

[0016]

[0017] Among them, the coefficient , , and is fixed, and the sum is 1. Also obtain the corresponding points after distortion Representation of;

[0018]

[0019] in, For point The bilinear interpolation function represents, , , , Indicates the coordinates of adjacent mesh vertices after deformation.

[0020] Step 2-2 Adaptive Threshold Divide lines into local line sets and the global line set , according to the line Length calculate ;

[0021]

[0022] in, Indicates the number of lines, Represents a local line set and the global line set A collection of lines, express The lines in.

[0023] For local line sets Each straight line in Uniform Sampling points, denoted as , calculate the energy representation of the global line set ;

[0024]

[0025] in, express The number of midlines, , is the straight line after deformation The adjacent sampling points on Represents lines The normal vector of .

[0026] For global line sets Each straight line in Uniform Sampling points, denoted as , calculate the energy representation of the global line set ;

[0027]

[0028] in, express The number of midlines, , is the straight line after deformation The adjacent sampling points on Represents lines The normal vector of .

[0029] Keep the line Defined as:

[0030]

[0031] in, , They represent the constraints of local lines and global collinear lines respectively, yes The weight of yes The weight of .

[0032] Step 2-3 The matched points and line pairs coincide with each other after deformation, so the point-line alignment item is obtained ;

[0033]

[0034] in, and denote point and line alignment items respectively, and Respectively and The weight of .

[0035] The point alignment term is the sum of the squared differences between the position of the interpolated point after transformation and the matching position in the reference image, and is given by:

[0036]

[0037] in, and are the matching point pairs in the target image and the reference image, Indicates that the point Mapping from the target image's coordinate system to the reference image's coordinate system, is the matrix containing the bilinear interpolation coefficients, is the position vector of the point in the reference image.

[0038] Line Alignment Item Ensure that the matched lines remain aligned and maintain their geometric relationship. The line alignment term is formulated based on the orthogonality of the line segments and their corresponding normal vectors, as follows:

[0039]

[0040] in, and are the line segments in the target image. The starting and ending points of and Respectively represent the line segments in the target image The starting and ending points of the transformed mapping positions, , , are the parameters of the line equation in the reference image, respectively.

[0041] Step 2-4 To control the distortion of the target image, a set of horizontal and vertical lines, called cross lines, are constructed. These lines are considered as the intrinsic linear structure of the target image. The distortion is adjusted by the angles of these lines and the spacing at their intersections. The characteristics of the horizontal and vertical lines depend on the parameters of the homography matrix.

[0042] Distortion term By the global distortion term and non-overlapping distortion terms composition;

[0043]

[0044] in, and Respectively and The weight of .

[0045] Global distortion term It focuses on maintaining the perspective relationship in the direction of the intersection line to ensure that image deformation does not introduce significant inconsistencies. The formula is as follows:

[0046]

[0047] in, Indicates that along the The first cross line is uniformly sampled Points, Indicates the number of crossing lines, Indicates The number of sampling points on the intersection lines, Represents the transformed The first The coordinates of the sampling points, Represents the transformed The first The coordinates of the sampling points.

[0048] Non-overlapping distortion terms Specially handles the distortion problem in non-overlapping areas of the image. The formula is as follows:

[0049]

[0050] in, represents the non-overlapping regions of the image, Indicates that along the The points are evenly sampled from the crossing lines. Represents the transformed The first The coordinates of the sampling points, Represents the transformed The first The coordinates of the sampling points.

[0051] Therefore, the total energy function It is expressed as:

[0052]

[0053] The better stitched image obtained by controlling the mesh deformation through the energy function has a better texture structure, which will be used to guide step 3 (line matching) and iterate with it.

[0054] Step 3. Line matching

[0055] Step 3-1: For the image distortion result obtained in step 2, search for candidate matches for each line detected in the image based on an adaptive threshold and geometric constraints. The threshold changes during the iteration according to the set relationship of the matching line pairs in the previous stage. The details are as follows:

[0056] The original line set after distortion in the target image is , the line set in the reference image is , The set of straight lines in exist Search for all reference lines that satisfy the geometric constraints ; The geometric constraints are as follows:

[0057] Constraint 1: Line The midpoint and line The distance between the midpoints of , ;in express A straight line, express A straight line, Represents a straight line Length, Represents a straight line Length.

[0058] Constraint 2: The distance between each line's endpoint and the other line satisfy , is the adaptive threshold; and Respectively represent the straight line after distortion The two endpoints to the reference line The distance and Respectively represent the reference line The two endpoints to the distorted straight line distance.

[0059] Step 3-2 For There may be multiple straight lines that satisfy the two geometric constraints in the reference image. Therefore, a scoring function composed of basic geometric metrics is used. , used to select the best line pair match: A specific value is calculated for each candidate line pair, where higher values ​​indicate a better match.

[0060] Scoring function The definition is as follows:

[0061]

[0062] in, Represents the angle between two lines in the candidate line pair; It is defined in constraint 2 to evaluate the distance between two lines; Represents the length of a straight line; function Taking the ratio of the lengths of two lines as input, equal lengths will yield the maximum value, the function The definition is as follows:

[0063]

[0064] in, yes The weight of have The straight line pair that satisfies the above constraints and obtains the highest score is considered a matching straight line pair and is added to the matching straight line pair set in the iteration loop. middle.

[0065] Threshold is set as an adaptive threshold, which is automatically adjusted according to the image size and the number of iterations: the threshold is set by the average distance of all approximately matching straight line pairs, and the distance changes from large to small during the iteration process. First, according to the scoring function, Each distorted line in the reference image is assigned a matching line with the highest score, so that the average distance of all line pairs can be calculated. At the same time, the size and linear structure of the image will affect the threshold These factors are affected by the diagonal length of the original image and the length of the longest straight line in the stitched image from step 2 Finally, the threshold The definition is as follows:

[0066]

[0067] in, for The number of straight lines in the middle; and Distorted straight line The distance between the endpoint of and the approximately matching line in the reference image.

[0068] When the line matching in the loop iteration is completed, The update formula is as follows:

[0069]

[0070] in, It is from the previous iteration. value, is the current iteration If the number of matched straight line pairs in the current iteration is less than 1 / 10 of that in the first iteration, the iteration process terminates.

[0071] This step and step 4 are a continuous process, and the straight line matching pairs obtained in this step will guide step 4 to generate point matching pairs.

[0072] Step 4. Generate point matching pairs from the matching sub-regions using the line matching pairs obtained in step 3 and CN (invariant Characteristic Number) based on existing methods. The details are as follows:

[0073] Invariant feature number The definition is as follows:

[0074] Assumptions For a space, For K dimensional projective space, yes The different points that constitute the closed loop, is the number of points. There are differences ,satisfy ,in Represents each straight line The number of different points on , is the weight, , ,

[0075]

[0076] reflects the intrinsic geometric properties of a given point. The projective invariant states a necessary and sufficient condition that if and are the matching point pairs in the target image and the reference image respectively, then they are in the formula The endpoints of the matched line segments and the original matching points are used to construct Point set in definition Therefore, if different views The values ​​are equal, then the corresponding points are correct matching points, and the point set The points in are considered as new matching points.

[0077] After this step is completed, the generated point matching pairs will be returned to step 2 together with the line matching pairs generated in step 3 to jointly optimize the global line-guided mesh deformation and start a new round of iteration again.

[0078] Beneficial effects of the present invention:

[0079] The present invention explores the global colinear structure through iterative matching, successfully maintains the local and global geometric structure in wide parallax image stitching, and reduces unnatural distortion; the designed adaptive threshold constraint is more effective than the descriptor-based straight line matching method, and the robust performance is independent of the image resolution and iteration stage. The coplanar sub-regions are explored using the invariant feature number to generate consistent matching points and line pairs, effectively suppressing artifacts and eliminating artifacts and non-uniform distortions existing in traditional matching strategies. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 This is the basic process framework diagram of the present invention; the "global line-guided mesh deformation" module and the "line point consistent matching based on splicing results" module are the main parts and are also alternating parts in iteration. The global line-guided mesh deformation retains the generated results and provides a consistent structure for line point matching; the generated line matching pairs and point matching pairs represented by the dotted lines provide more anchor points for the global line-guided mesh deformation.

[0081] Figure 2 is the geometric metric computed by the geometric constraints and scoring functions in steps 3 and 4.

[0082] Figure 3 is the constant feature number , a situation where the loop forms a triangle.

[0083] Figure 4 The five feature points in the image are connected to each other to construct a five-point feature number scenario with projective invariance.

[0084] Figure 5 are the input target image and reference image.

[0085] Figure 6 is the result of step 1: pre-registered image.

[0086] Figure 7 It is the final line matching result after the iteration of steps 2, 3 and 4.

[0087] Figure 8 It is the result obtained in step 2 based on the final line and point matching pairs.

[0088] Fig. 9 is the final stitched image generated. DETAILED DESCRIPTION

[0089] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0090] The basic process of the alternating iterative line matching and image stitching method of the present invention is as follows: Figure 1 As shown, this embodiment specifically includes the following steps:

[0091] Input image pair: Figure 5 (a) and (b) of the image.

[0092] Step 1: Take the input image pair as the original input, detect and match feature points by using the existing algorithm SURF, and then preliminarily align them by calculating the homography matrix between the target image and the reference image to obtain the pre-registered image, such as Figure 6 shown.

[0093] Step 2: Take the pre-registered image and matching points obtained in step 1 as input, and obtain the initial stitched image by global line guided grid deformation (grid the pre-registered image): The stitched image obtained in step 2 guides step 3 to obtain line matching pairs based on the adaptive threshold and geometric constraints to search for candidate matches for each line detected in the image. Return to step 2, and use the line matching pairs and point matching pairs obtained in steps 3 and 4 as input to guide global line-guided mesh deformation; then use the spliced ​​image generated in step 2 to guide steps 3 and 4 to find more and more accurate line matching pairs and point matching pairs; then use the obtained line matching pairs and point matching pairs as input to guide global line-guided mesh deformation.

[0094] In the energy function of step 2, the line retention term Weight and Set to 50 and 100 respectively, point and line alignment items Weight and Set to 1 and 5 respectively, distortion item Weight and The number of iterations is set to 50 and 100 respectively, and the final line matching result is as follows: Figure 7 shown.

[0095] Step 3: Based on the adaptive threshold, the geometric constraints are used to search for candidate matches for each line detected in the image to obtain a line matching pair diagram as shown in the figure below: Figure 2 shown.

[0096] Step 4: Use the line matching result obtained in step 3 Get the point matching result. The process is illustrated as follows:

[0097] When r = 3, the loop forms a triangle: yes There are three vertices of the triangle, and there are two other points on each side of the triangle (the straight line), forming a point set So according to the formula Calculate the number of features, such as Figure 3 shown.

[0098] By connecting the five feature points in the image to each other, a five-point feature number with projective invariance can be constructed, such as Figure 4 Shown: Given five coplanar points , , , , , satisfying that any three points are not collinear, and If they are not parallel, the five-point characteristic numbers can be constructed as shown in the figure. .Pick , , , , , , .here , represents the intersection of the straight line through points A and B and the straight line through points C and D. So we can get the point set , Then, the five-point characteristic number can be calculated using the calculation method of the triangle characteristic number:

[0099] Step 2 takes the final line and point matching results obtained in steps 3 and 4 as input and obtains the result, such as Figure 8 shown.

[0100] Finally, the final result of step 2 is used as input to obtain the final stitched image, as shown in Fig. 9 shown.

[0101] From the results, we can see that our method accurately aligns overlapping and non-overlapping regions on challenging test images. The local and global geometric structures in wide parallax image stitching are successfully preserved through an iterative image stitching method; this method effectively increases the number of anchor points, which helps to reduce distortion by exploring global colinear structures. The adaptive threshold-based line matching method effectively matches line pairs by geometrically constraining the results of consecutive image stitching. The invariant feature number construction provides consistent line-point constraints on coplanar overlapping regions, significantly suppressing artifacts.

Claims

1. An alternating iterative line matching and image stitching method, characterized in that: The steps include: Step 1. Pre-registration based on matching feature points The SURF algorithm is used to detect and match feature points of the two original images, and then the homography matrix between the target image I and the reference image I' is calculated to preliminarily align them to obtain the pre-registered image, as follows: Detect and match feature points of two original images to obtain paired matching point pairs {(p i ,p' i )} i＝1,2,…,N , where i represents the number of matching point pairs, N represents the total number of paired matching points, and p i =(x i ,y i ,1) and p' i =(x' i ,y i ',1) represents the homogeneous coordinates of the midpoints in the target image I and the reference image I', 1 is the homogeneous term setting when the feature point coordinates are converted to homogeneous coordinates; the global homography of the point feature is calculated based on the coordinates of the matching points Where H represents the initial homography, H * is the vector representation of H, Represents point p i The mapping result under the action of the homography matrix H, that is, the transformed position, For p' i The vector representation of , ‖‖ represents the Euclidean norm of the vector; the singular value decomposition is used to minimize is 0, which means that the point pair (p i ,p' i ) fully complies with the constraints of the homography matrix; in the optimization process, the goal is to minimize That is, minimize the geometric error of all point pairs to estimate the best H * ; Step 2. Global line-guided mesh deformation The pre-registered image and matching point pairs obtained in step 1 are used as input. The image is distorted by a mesh deformation method based on a global line-guided total energy function to maintain the local and global linear structure of the stitched image. The total energy function contains line preservation terms, point-line alignment terms, and distortion terms. The details are as follows: Step 2-1 constructs a rectangular network for each image, with the original vertex coordinate vector V = [x1, y1, x2, y2, ..., x n ,y n ] T , the distorted vertex coordinate vector is n represents the grid vertex index; the bilinear interpolation function τ(P) of each feature point P in the image is represented by the four vertices s1, s2, s3 and s4 of the closed grid; τ(P)=w1s1+w2s2+w3s3+w4s4 Among them, the coefficients w1, w2, w3 and w4 are fixed, and the sum is 1; the corresponding points after distortion are also obtained Representation of; in, For point The bilinear interpolation function represents, Represents the coordinates of adjacent mesh vertices after deformation; Step 2-2 Divide the lines into local line sets S by adaptive threshold μ lo and the global line set S gl , according to line l m The length len(l m ) Calculate μ; Among them, N l Indicates the number of lines, S t Represents the local line set S lo and the global line set S gl A collection of lines, l m Indicates S t The lines in For the local line set S lo Each straight line in {l a } a＝1,2,…,R Uniform sampling M a points, denoted as Compute the energy representation of the global line set Where R represents S lo The number of midlines, is the deformed straight line l a The adjacent sampling points on Indicates line l a The normal vector of For the global line set S gl Each straight line in {l b } b＝1,2,…,Q Uniform sampling M b points, denoted as Compute the energy representation of the global line set Where Q represents S gl The number of midlines, is the deformed straight line l b The adjacent sampling points on Indicates line l b The normal vector of Keep the line Defined as: in, They represent the constraints of local straight lines and global collinear lines, respectively, lo yes The weight, λ gl yes The weight of The points and lines matched in step 2-3 coincide with each other after deformation, so the point-line alignment item is obtained. in, and denote point and line alignment terms respectively, λ p and λ l Respectively and The weight of The point alignment term is the sum of the squared differences between the position of the interpolated point after transformation and the matching position in the reference image, and is given by: Among them, p i and p i ′ is the matching point pair in the target image and the reference image, Indicates that point p i Mapping from the target image’s coordinate system to the reference image’s coordinate system, W p is a matrix containing bilinear interpolation coefficients, and P is the position vector of the point in the reference image; Line Alignment Item Ensure that the matched lines remain aligned and maintain their geometric relationship; the line alignment term is formulated based on the orthogonality of the line segments and their corresponding normal vectors, as follows: Among them, p sj and p ej are the line segments l in the target image. j The starting and ending points of and Respectively represent the line segment l in the target image j The starting and ending points of the transformed mapping positions, a j , b j 、c j are the parameters of the line equation in the reference image, respectively; Step 2-4: To control the distortion of the target image, construct a set of horizontal and vertical lines, called cross lines; these lines are regarded as the intrinsic linear structure of the target image; the distortion is adjusted by the angle of these lines and the spacing at their intersections; Distortion term By the global distortion term and non-overlapping distortion terms composition; Among them, λ dg and λ dn Respectively and The weight of Global distortion term Focus on maintaining the perspective relationship in the direction of the intersection line to ensure that image deformation does not introduce significant inconsistencies; the formula is as follows: in, represents the kth point uniformly sampled along the i-th intersection line, S represents the number of intersection lines, L i represents the number of sampling points on the i-th intersection line, represents the coordinates of the k+1th sampling point on the i-th intersection line after transformation, Represents the coordinates of the kth sampling point on the i-th intersection line after transformation; Non-overlapping distortion terms Specially handles the distortion problem in non-overlapping areas of the image; the formula is as follows: Where Ω represents the non-overlapping area of ​​the image, p u,i represents the points uniformly sampled along the i-th intersection line, represents the coordinates of the k+2th sampling point on the i-th intersection line after transformation, Represents the coordinates of the k+1th sampling point on the i-th intersection line after transformation; Therefore, the total energy function It is expressed as: The better stitched image obtained by controlling the mesh deformation through the energy function has a better texture structure, which will be used to guide step 3 and iterate with it; Step 3. Line matching Step 3-1 searches for candidate matches for each line detected in the image based on an adaptive threshold and geometric constraints for the image distortion result obtained in step 2. The threshold changes during the iteration according to the set relationship of the matching line pairs in the previous stage. The details are as follows: The original line set after distortion in the target image is The line set in the reference image is Represents the original line set after distortion The set of straight lines in In S r Search for all reference lines l that satisfy the geometric constraints r ; The geometric constraints are as follows: Constraint 1: Line The midpoint and line The distance between the midpoints is denoted as d h , 2; among which express A straight line, Indicates l r A straight line, Represents a straight line Length, Represents a straight line Length; Constraint 2: The distance between each line’s endpoint and the other line and d v =d1+d2+d3+d4 satisfies d v / 2<γ, γ is the adaptive threshold; d1 and d2 represent the straight line after distortion. The two endpoints to the reference line The distances d3 and d4 are respectively from the reference line The two endpoints to the distorted straight line distance; Step 3-2 For The straight line in the image is scored using a scoring function composed of basic geometric metrics. Used to select the best pair match: Calculate a specific value for each candidate line pair, where a higher value indicates a better match; Scoring function The definition is as follows: Where θ represents the angle between the two lines in the candidate line pair; d v In constraint 2, it is defined to evaluate the distance between two straight lines; len(.) represents the length of the line; the function f(x) takes the ratio of the lengths of the two straight lines as input, and equal lengths will get the maximum value. The function f(x) is defined as follows: Among them, e 2 yes The weight of; function f(x) has The property of , so that the exchange of numerator and denominator remains unchanged; the straight line pair that satisfies the above constraints and obtains the highest score is considered to be a matching straight line pair, and it is added to the matching straight line pair set in the iteration loop middle; The threshold γ is set as an adaptive threshold, which is automatically adjusted according to the image size and the number of iterations: the threshold is set by the average distance of all matching straight line pairs, and the distance changes from large to small during the iteration process; first, according to the scoring function, Each distorted line in the reference image is assigned a matching line with the highest score, thereby calculating the average distance of all line pairs; at the same time, the size and linear structure of the image will affect the threshold γ, which is reflected by the diagonal length len(l dia ) and the length of the longest straight line in the spliced ​​image of step 2, len(l max ) is used to represent it; finally, the threshold γ is defined as follows: Among them, n t for The number of straight lines in the middle; and Distorted straight line The distance between the endpoint and the approximate matching line in the reference image; When the line matching in the loop iteration is completed, γ loop The update formula is as follows: Among them, γ loop-1 is the value of γ in the previous iteration, γ loop is the value of γ in the current iteration; if the number of matched straight line pairs in the current iteration is less than 1 / 10 of that in the first iteration, the iteration process terminates; This step and step 4 are continuous processes. The straight line matching pairs obtained in this step will guide step 4 to generate point matching pairs. Step 4. Generate point matching pairs from the matching sub-regions through the line matching pairs obtained in step 3 and CN; the details are as follows: The invariant feature number CN is defined as follows: Assumptions For a space, is an m-dimensional projection space on K, {P i } i＝1,2,…,H yes The different points that form the closed loop, H is the number of points; the straight line {P i P i+1 } i＝1,2,…,H There are differences satisfy Where Z represents each straight line P i P i+1 The number of different points on is the weight, let P = {P i } i＝1,2,…,R , CN reflects the intrinsic geometric properties of a given point; the projective invariant states a necessary and sufficient condition that if {P i } i＝1,2,…,H and {P i ′ } i＝1,2,…,H are the matching point pairs in the target image and the reference image respectively, then their values ​​in the formula CN(P,Q) are equal; the endpoints of the matching line segments and the original matching points are used to construct the point set P in the CN definition; therefore, if the CN values ​​in different views are equal, the corresponding points are correct matching points, and the points in the point set Q are regarded as new matching points; After this step is completed, the generated point matching pairs will be returned to step 2 together with the line matching pairs generated in step 3 to jointly optimize the global line-guided mesh deformation and start a new round of iteration again.

Images