A natural image stitching method based on extended point line features

Abstract

The application is a natural image splicing method based on extended point line features. The application relates to the technical field of image splicing, and the application performs geometric operation on straight lines in an image through a region line segment connection method, so that line features in the overlapping region of the image are preserved as much as possible. Then, point features are generated by using the matched line features, the matching points are supplemented, the image pre-alignment is optimized, and the geometric shape features of the image are preserved as much as possible. Finally, image pre-alignment, homography matrix estimation and grid deformation and a series of operations are performed through the matched point line features, so that image splicing is realized. Experiments show that the method of the application has obvious advantages in complex image splicing with obvious geometric shapes, and compared with the existing method D dis The maximum improvement is 28.8%, dir The maximum improvement is 63.2%.

Description

Technical Field

[0001] This invention relates to the field of image stitching technology, and is a natural image stitching method based on extended point and line features. Background Technology

[0002] Image stitching is the process of combining multiple images with overlapping and relatively narrow fields of view into an image with a wider field of view. After years of development, image stitching has made significant progress. However, with the advancement of smartphones, digital cameras, and video surveillance technologies, the requirements for acquiring panoramic images are becoming increasingly demanding. Combining images with characteristics such as wide baselines, weak textures, and large fields of view in complex scenes into a high-quality stitched image remains a very challenging task.

[0003] The overall naturalness of the stitched images is a crucial factor affecting the overall quality of image stitching. This is visually manifested in the presence of ghosting, misalignment of key features, and unnatural distortions in the stitched image. Image feature matching is key to aligning multiple images and eliminating ghosting and other influencing factors. Point features are the most widely used image feature matching method in traditional image stitching. Among them, SIFT and SURF point feature extraction methods are widely used in image alignment due to their good invariance to local rotation, scale transformation, and affine transformation. With in-depth research into image stitching methods, line features, due to their excellent representation of image features, are also increasingly widely used in image stitching, significantly improving the quality of image alignment during the stitching process.

[0004] In early research on image stitching, homography transformation was typically estimated using only a single feature. The AutoStitch method estimates global homography transformation using point features and employs multi-band blending for panoramic image stitching; however, this method cannot solve the problem of image stitching with multiple planar scenes. Image stitching based on dual homography feature distortion (DHW) abstracts the image scene into a distant background plane and a ground plane, representing a stitching method based on an ideal scene. As far as possible distortion (APAP) divides the image into grid regions and proposes a Moving Direct Linear Transform (Moving DLT) model to stitch image regions with inconsistent projection models, significantly eliminating ghosting during the stitching process. Image stitching based on robust elastic deformation (ELA) eliminates disparity errors generated in image feature point matching by constructing a constructor and distorts and deforms the input image based on a grid plane. These methods strive to minimize alignment errors in matching regions through a global transformation, but they are often inflexible and struggle to satisfy feature alignment across different planes. In addition, conformal projection (SPHP), adaptive distortion as natural as possible (AANAP), and global similarity prior (GSP) models attempt to address the distortion problem caused during the stitching process by applying different distortions to overlapping and non-overlapping regions.

[0005] With in-depth research on image stitching, Li et al. first proposed an image stitching method based on point and line dual features (DFW) to address the problem of limited features in image alignment and improve the naturalness of image distortion. Since multiple features make it easier to constrain image alignment, this method has also seen rapid development. Tianli Liao et al. proposed Single View Warp (SPW) based on point and line feature pre-matching and meshed warping, which to some extent solved the problem of projection distortion. Qi Jia et al. proposed the Point-Line Consistency (LPC) method, which preserves global feature changes during image stitching through the global collinearity of image features. Peng Du et al. introduced curve features into the Geometric Structure Preserving Global Similarity Transform (GES-GSP) to further constrain the geometry in the image, improving the naturalness of the stitched image.

[0006] Image stitching typically relies on local or global homography transformations to constrain image features, thereby improving stitching quality. Therefore, the matching feature points and lines generated during pre-alignment are crucial. Existing stitching methods can achieve good results when there are sufficient point and line features in the image. However, for images with weak texture regions, it is difficult to extract point and line features simultaneously in all regions, leading to feature loss in these areas during pre-alignment. Furthermore, for images with complex features, the feature lines detected by line detection (LSD) methods are often discontinuous, easily resulting in the loss of some feature lines during line feature matching, thus affecting the overall image stitching effect. Summary of the Invention

[0007] To overcome the shortcomings of existing technologies, this invention addresses the problem of feature line loss during line feature matching by extending line features. Simultaneously, it generates more matching points by leveraging the characteristic that the intersection of matched feature lines forms a new pair of matching feature points, and extends the matching features as much as possible to weakly textured regions, thereby solving the problems of line feature loss and insufficient features in weakly textured areas during alignment. Finally, an energy equation for global line-guided mesh deformation is introduced to achieve the final image stitching.

[0008] It should be noted that, in this invention, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0009] This invention provides a natural image stitching method based on extended dot and line features. The invention offers the following technical solutions:

[0010] A natural image stitching method based on extended dot and line features, the method comprising the following steps:

[0011] The method includes the following steps:

[0012] Step 1: Perform geometric operations on straight lines in the image by connecting region line segments to preserve the line features of overlapping areas in the image;

[0013] Step 2: Generate point features using the matched line features, supplement the matched points, optimize image pre-alignment, and preserve the geometric features of the image as much as possible;

[0014] Step 3: Perform image pre-alignment, homography matrix estimation, and meshing deformation operations using matched point and line features to achieve image stitching.

[0015] Preferably, step 1 specifically comprises:

[0016] Step 1.1: By using a method based on connecting line segments in the region, multiple line segments in similar regions are restored into continuous salient feature lines, thereby preserving the continuity of salient line features as much as possible and improving the pre-alignment quality;

[0017] By merging collinear line segments within the same region, they are restored to continuous salient feature lines as much as possible, thus preserving the features of the line structure as much as possible during the line feature matching process. During the connection of line segments, the slopes s(l1) and s(l2) of the two lines are evaluated. The slopes of the lines should be the same or very close.

[0018] Step 1.2: Calculate the endpoint of line segment l1 To the starting point of line segment l2 distance This distance should be small for straight lines to be connected and merged;

[0019] Step 1.3: Finally, analyze the merged line l a The slope s(l) a The slope of the line segment is very close to that of s(l1) and s(l2) to be retained. After multiple iterations, all line segments in the overlapping area are judged and connected to complete the connection of the line segments in the area and save the line features of the overlapping area of ​​the image.

[0020] Preferably, the 2 specifically refers to:

[0021] Can the matching features be expanded by generating more feature points through the extended matching line features, thereby expanding the matching features? For two pairs of coplanar matching lines l in a set of images to be stitched together... 13 and l 22 The intersection point p is a new set of matching feature points; by matching any two matching feature lines in the image, feature points located within the image region are selected to expand the matching feature point pairs.

[0022] RANSAC is used to remove mismatches from the extended matching point pairs, improving the reliability of the matching point pairs. Since lines have infinite extensibility and intersections exist at any position in the plane, the matching features are extended to weak texture regions in the image to a certain extent, which solves the problem of fewer features in weak texture regions and improves the quality of image pre-alignment and stitching.

[0023] Preferably, step 3 specifically comprises:

[0024] Step 3.1: Let p and p' be a pair of matching points in the overlapping regions of images I and I', where p = (x, y) and p' = (x', y'); then the correspondence between points p and p' is represented by the homography matrix H as follows:

[0025]

[0026]

[0027] in, Let p be the homogeneous coordinates, and H be the homography matrix; in the non-homogeneous coordinate system,

[0028]

[0029] Combining the homography matrix estimation methods in APAP and QH, the final homography distortion is defined as:

[0030]

[0031] Among them, H Q and H A These are the homography matrices obtained using methods from QH and APAP, respectively. H is derived from H. A Extrapolation to H Q The homography matrix;

[0032] Step 3.2: The image stitching method based on gridded deformation involves dividing the image to be stitched into multiple grid regions of the same size, and using the grid region as the smallest deformation object for homography matrix estimation. Its definition is as follows:

[0033]

[0034] in The problems of surface feature deformation and global scaling in the image stitching process are solved by constraining the distance and angle between features. The alignment problem of overlapping areas in images is solved by constraining matching points and matching lines. The distortion problem encountered in splicing is solved by constraining the grid lines. The problem of straightness is solved by constraining the straightness of the same feature line using local and global feature lines.

[0035] The preservation of surface features and distance features for maintaining the proportion of deformed images during image stitching is expressed by the following formula:

[0036]

[0037] in, and Constrain the length and angle of the planar boundary line, λ respectively. sd and λ sa These are the weights of the corresponding constraint terms;

[0038] The straight line in image I It is constructed from feature points and feature lines present in the image. i The length constraint is a distance constraint between different features, l i Angle constraints serve as positional constraints between different features, while length and angle constraints work together to achieve relative position and relative distance constraints between different objects in the image.

[0039] Constraining the features of the coplanar plane in space, let image I have its corresponding deformed image I′, and the straight line Considering that there will inevitably be lines of excessive length, the line l will be represented by M points during the calculation. i Divide into straight lines evenly L j The starting and ending points can be represented as The corresponding straight line in image I′ can be obtained. and points

[0040]

[0041] The function σ(·) is defined as the calculation of the corresponding coordinates of a point. For point Image pixel coordinates and points Perform corresponding calculations on the image pixel coordinates.

[0042] The relative directional changes between different features during deformation are controlled by constraining the angle of the constructed straight line.

[0043]

[0044] in,

[0045] Preferably, the association operation specifically involves: associating each individual in the group with a corresponding reference point: the line connecting the origin and the reference point is used as the reference line, and S is calculated. t The distance from an individual to each reference line is used to determine the relationship between that individual and the corresponding reference line.

[0046] Preferably, the method further includes evaluation indicators, specifically:

[0047] To quantitatively evaluate the performance of image stitching methods, evaluation metrics were designed for the performance of surface features in the image and the variation of distance characteristics between different features. For any two distinct features in the overlapping regions of a set of images to be stitched, taking feature point p and feature line l as an example, the distance between features passing through point p... j and line l i The two endpoints Construct indirect features respectively The deformed indirect features are denoted as follows: The degree of deformation is defined as follows:

[0048]

[0049]

[0050] Where, K = min(num(l) i ),num(p j )), num(l i ) and num(p j ) represent the number of detected feature points and feature lines, respectively, and len(l) represents the length of the line segment;

[0051] Establish a quantitative evaluation index table to evaluate the method.

[0052] A computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement a natural image stitching method based on extended dot-line features.

[0053] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement a natural image stitching method based on extended point and line features.

[0054] The present invention has the following beneficial effects:

[0055] This invention proposes a natural image stitching method based on extended point and line features. It performs geometric operations on straight lines in the image by connecting region line segments, thereby preserving the line features of overlapping areas as much as possible. Then, it uses the matched line features to generate point features, supplements the matching points, optimizes image pre-alignment, and retains the geometric shape features of the image as much as possible. Finally, it performs a series of operations such as image pre-alignment, homography matrix estimation, and mesh deformation using the matched point and line features to achieve image stitching. Experiments show that the method of this invention has significant advantages for stitching complex images with obvious geometric shapes, compared to existing methods. dis Maximum increase of 28.8%, D dir Maximum improvement of 63.2%.

[0056] This invention minimizes the loss of feature lines during the matching process by performing geometric operations on feature lines, thereby avoiding the loss of matching features to a certain extent, improving the quality of pre-alignment, and better preserving details in the image.

[0057] This invention proposes to extend the matching features as far as possible into weakly textured regions by matching the intersections of feature lines, thereby generating new feature points. By expanding the number of matching features, the quality of pre-alignment can be improved, while further eliminating ghosting and uneven distortion. Attached Figure Description

[0058] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0059] Figure 1 To match images;

[0060] Figure 2 Here is a flowchart of the algorithm for connecting line segments in a region;

[0061] Figure 3 This is a feature point map added after the method was extended;

[0062] Figure 4 Expand the graph for matching points;

[0063] Figure 5 The image contains any two non-collinear feature lines.

[0064] Figure 6 Indirect feature point map;

[0065] Figure 7 This is a stitched image of some of the images. Detailed Implementation

[0066] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0068] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances. Specific Implementation Example 1:

[0070] according to Figures 1 to 7 As shown, the specific optimization technical solution adopted by the present invention to solve the above-mentioned technical problems is: The present invention relates to a natural image stitching method based on extended point and line features.

[0071] A natural image stitching method based on extended dot and line features, characterized by the following steps:

[0072] Step 1: Perform geometric operations on straight lines in the image by connecting region line segments to preserve the line features of overlapping areas in the image;

[0073] Step 1 specifically involves:

[0074] Step 1.1: By using a method based on connecting line segments in the region, multiple line segments in similar regions are restored into continuous salient feature lines, thereby preserving the continuity of salient line features as much as possible and improving the pre-alignment quality;

[0075] By merging collinear line segments within the same region, they are restored to continuous salient feature lines as much as possible, thus preserving the features of the line structure as much as possible during the line feature matching process. During the connection of line segments, the slopes s(l1) and s(l2) of the two lines are evaluated. The slopes of the lines should be the same or very close.

[0076] Step 1.2: Calculate the endpoint of line segment l1 To the starting point of line segment l2 distance This distance should be relatively small; proceed with connection merging;

[0077] Step 1.3: Finally, analyze the merged line l a The slope s(l) a The slope of the line segment is very close to that of s(l1) and s(l2) to be retained. After multiple iterations, all line segments in the overlapping area are judged and connected to complete the connection of the line segments in the area and save the line features of the overlapping area of ​​the image.

[0078] Step 2: Generate point features using the matched line features, supplement the matched points, optimize image pre-alignment, and preserve the geometric features of the image as much as possible;

[0079] Step 2 specifically involves:

[0080] Can the matching features be expanded by generating more feature points through the extended matching line features, thereby expanding the matching features? For two pairs of coplanar matching lines l in a set of images to be stitched together... 13 and l 22 The intersection point p is a new set of matching feature points; by matching any two matching feature lines in the image, feature points located within the image region are selected to expand the matching feature point pairs.

[0081] RANSAC is used to remove mismatches from the extended matching point pairs, improving the reliability of the matching point pairs. Since lines have infinite extensibility and intersections exist at any position in the plane, the matching features are extended to weak texture regions in the image to a certain extent, which solves the problem of fewer features in weak texture regions and improves the quality of image pre-alignment and stitching.

[0082] Step 3: Perform image pre-alignment, homography matrix estimation, and meshing deformation operations using matched point and line features to achieve image stitching.

[0083] Step 3 specifically involves:

[0084] Step 3.1: Let p and p' be a pair of matching points in the overlapping regions of images I and I', where p = (x, y) and p' = (x', y'); then the correspondence between points p and p' is represented by the homography matrix H as follows:

[0085]

[0086]

[0087] in, Let p be the homogeneous coordinates, and H be the homography matrix; in the non-homogeneous coordinate system,

[0088]

[0089] Combining the homography matrix estimation methods in APAP and QH, the final homography distortion is defined as:

[0090]

[0091] Among them, H Q and H A These are the homography matrices obtained using methods from QH and APAP, respectively. H is derived from H. A Extrapolation to H Q The homography matrix;

[0092] Step 3.2: The image stitching method based on gridded deformation divides the image to be stitched into multiple grid regions of the same size, and uses the grid region as the smallest deformation object for homography matrix estimation. The optimized energy equation fully considers the correspondence between point features and line features, greatly improving the quality of image stitching. Its definition is as follows:

[0093]

[0094] in The problems of surface feature deformation and global scaling in the image stitching process are solved by constraining the distance and angle between features. The alignment problem of overlapping areas in images is solved by constraining matching points and matching lines. The distortion problem encountered in splicing is solved by constraining the grid lines. The problem of straightness is solved by constraining the straightness of the same feature line using local and global feature lines.

[0095] The preservation of surface features and distance features for maintaining the proportion of deformed images during image stitching is expressed by the following formula:

[0096]

[0097] in, and Constrain the length and angle of the planar boundary line, λ respectively. sd and λ sa These are the weights of the corresponding constraint terms;

[0098] The straight line in image I It is constructed from feature points and feature lines present in the image. i The length constraint is a distance constraint between different features, l i Angle constraints serve as positional constraints between different features, while length and angle constraints work together to achieve relative position and relative distance constraints between different objects in the image.

[0099] Constraining the features of the coplanar plane in space, let image I have its corresponding deformed image I′, and the straight line Considering that there will inevitably be lines of excessive length, the line l will be represented by M points during the calculation. i Divide into straight lines evenly L j The starting and ending points can be represented as The corresponding straight line in image I′ can be obtained. and points

[0100]

[0101] The function σ(·) is defined as the calculation of the corresponding coordinates of a point. For point Image pixel coordinates and points Perform corresponding calculations on the image pixel coordinates.

[0102] The relative directional changes between different features during deformation are controlled by constraining the angle of the constructed straight line.

[0103]

[0104] in,

[0105] The method also includes evaluation indicators, specifically:

[0106] To quantitatively evaluate the performance of image stitching methods, evaluation metrics were designed for the performance of surface features in the image and the variation of distance characteristics between different features. For any two distinct features in the overlapping regions of a set of images to be stitched, taking feature point p and feature line l as an example, the distance between features passing through point p... j and line l i The two endpoints Construct indirect features respectively The deformed indirect features are denoted as follows: The degree of deformation is defined as follows:

[0107]

[0108]

[0109] Where, K = min(num(l) i ),num(p j )), num(l i ) and num(p j ) represent the number of detected feature points and feature lines, respectively, and len(l) represents the length of the line segment;

[0110] Establish a quantitative evaluation index table to evaluate the method. Specific Implementation Example 2:

[0112] The only difference between Embodiment 2 and Embodiment 1 of this application is that:

[0113] homography distortion and mesh deformation

[0114] Homography warping is a single-view warping method for image stitching based on the assumption that the image is purely captured by rotating camera motion. However, it easily introduces problems such as image pre-alignment and distortion when incorporating camera motion translation or multi-plane scenes into the image. For general images, homography warping connects the same scene from different views through a "least squares" operation on the common principal plane, and solves the alignment, distortion, and naturalness issues of the common principal plane in different images through parameterized mesh warping, ultimately achieving high-quality image stitching.

[0115] Homography matrix estimation

[0116] The theoretical basis of single-view image stitching based on feature homography transformation and image meshing deformation is the homography matrix transformation between feature points. Assume p and p' are a pair of matching points in overlapping regions of images I and I', where p = (x, y) and p' = (x', y'). Then the correspondence between points p and p' can be represented by the homography matrix H as follows:

[0117]

[0118]

[0119] in Let p be the homogeneous coordinates, and H be the homography matrix. In a non-homogeneous coordinate system,

[0120]

[0121] In practical applications, besides feature alignment, naturalness and distortion are also key factors affecting image stitching quality. Therefore, combining the homography matrix estimation methods in APAP and QH, the final homography distortion is defined as:

[0122]

[0123] Among them, H Q and H A These are the homography matrices obtained using methods from QH and APAP, respectively. H is derived from H. A Extrapolation to H Q The homography matrix.

[0124] Mesh deformation

[0125] The image stitching method based on gridded deformation divides the image to be stitched into multiple grid regions of equal size and uses these grid regions as the objects of least deformation for homography matrix estimation and other operations, thereby minimizing deformation and distortion during image stitching. This method establishes an energy term based on grid-based distortion to address issues of image alignment, naturalness, distortion, and saliency. It then minimizes the total energy function to obtain the optimal single-view distortion, fully considering the feature information of different targets in the image, and has wide applications in image stitching. Its definition is as follows:

[0126]

[0127] in The problems of surface feature deformation and global scaling in the image stitching process are solved by constraining the distance and angle between features. The alignment problem of overlapping areas in images is solved by constraining matching points and matching lines. The distortion problem encountered in splicing is solved by constraining the grid lines. The problem of straightness is solved by constraining the straightness of the same feature line using local and global feature lines.

[0128] Preservation of straight line features based on region segment connections

[0129] Image stitching line feature extraction is typically accomplished through LSD line detection. Therefore, during line detection, it's inevitable that continuous, salient lines will be segmented into multiple line segments for display, such as... Figure 1 (1) shows that, in the image matching process, the overlapping areas of the two images need to have common line segments to achieve line matching. Generally, the two images to be stitched have certain differences in shooting angle, shooting distance, etc., so the detection of the same significant straight line in the overlapping area of ​​the images will be displayed as line segments that are not completely identical. Figure 1 (1)b, Figure 1 (1)d), which inevitably leads to the loss of continuity features of some salient lines during feature matching, affecting the pre-alignment quality of overlapping areas. Figure 1 (2) This invention restores multiple line segments in similar regions into continuous salient feature lines by using a method based on connecting regional line segments, thereby preserving the continuity of salient line features as much as possible and improving the pre-alignment quality.

[0130] Figure 1 The first row of images is a set of original matching images, the second row is the matching feature lines in LPC after line feature matching, and the third row is the matching feature lines after connecting the region line segments. a is a set of matching images, and b and d are magnified images of the corresponding regions in a and c.

[0131] like Figure 1 As shown in (1)b, the red line segments l1, l2, and l3 are feature lines in the overlapping region of the image. In practice, these three lines should be collectively represented by the prominent yellow line l4, and their corresponding features are... Figure 1 (1)b) The salient line is displayed as only one feature line. To preserve the structural characteristics of the salient line as much as possible, this invention proposes a linear feature preservation measurement method. This involves merging collinear line segments within the same region to restore them as continuously salient feature lines as possible, thereby preserving the characteristics of the line structure during line feature matching. Figure 1 (1)b Taking this as an example, in the process of connecting line segments, the present invention first evaluates the slopes s(l1) and s(l2) of the two straight lines. The slopes of the straight lines should be the same or very close. Then, the endpoint of line segment l1 is calculated. To the starting point of line segment l2 distance This distance should be relatively small. After both conditions are met, the line segments can be connected and merged. Finally, the merged line l... a The slope s(l) a The algorithm determines whether a line segment is retained if its slope is very close to s(l1) and s(l2). After multiple iterations, all line segments within the overlapping region are determined and connected to complete the connection of line segments in the region. The specific algorithm is as follows: Figure 2 As shown:

[0132] Matching feature extension based on matching lines

[0133] In image stitching, matching features are a prerequisite for image alignment and also a crucial factor affecting the stitching effect. For the same set of images, the more matching features obtained during feature detection, the better the stitching effect in overlapping areas. Existing image stitching methods typically use SIFT or SURF point feature matching to initially extract matching point pairs from partial image regions, and then use RANSAC to remove mismatches. Figure 3 (As shown). This method has proven its effectiveness in the development of image stitching. However, in images with significant shape features, feature point extraction often misses key feature points such as vertices of the shape features. Matching these feature point pairs plays an important role in preserving the shape features.

[0134] Figure 3 Figure a shows SIFT+RANSAC feature points, Figure b shows SURF+RANSAC feature points, and Figure c shows the feature points added after the extension of the method of this invention.

[0135] In this study, more feature points are generated by expanding the matching line features, thereby expanding the matching features. For example... Figure 4As shown, for two pairs of coplanar matching lines l in a set of images to be stitched together 13 and l 22 Their intersection point p( Figure 4 The blue dots represent a new set of matching feature points. For images with complex scenes, due to the characteristics of camera imaging, any two straight lines in the image can be considered as coplanar lines with the camera imaging plane as the same plane. Based on this, by matching any two matching feature lines in the image, feature points located within the image region are selected to expand the matching feature point pairs. Simultaneously, to avoid mismatches, RANSAC is used to remove mismatches from the expanded matching point pairs, further improving the reliability of the matching point pairs. The new matching feature point pairs obtained through this method enable the extraction of some key feature point pairs. Furthermore, since straight lines have infinite extensibility, intersection points can exist at any position in the plane, thus extending the matching features to weak texture regions in the image to some extent, addressing the problem of fewer features in weak texture regions and improving the quality of image pre-alignment and stitching.

[0136] Mesh deformation that integrates surface feature information

[0137] Existing image stitching methods based on mesh deformation typically design energy equations based on the correspondence between point and line features, using the optimal solution of the energy equation as the final homography matrix of the image deformation. This approach relies on the positional offset of the same feature before and after deformation, providing good constraints on feature positions. However, in practice, due to the sensitivity of human vision to changes in shape and distance, changes in positional information and planar properties between different features before and after image deformation have a significant impact on the quality of image stitching.

[0138] In space, a straight line can be determined by two non-coincident points, and a unique plane can be determined by three points not on the same straight line. In an image, all information is imaged on the same camera plane, but in reality, different image features may lie on different spatial planes. Furthermore, due to the unordered nature of feature point detection and line detection, determining coplanar points and lines in space is extremely difficult.

[0139] In this invention, instead of searching for coplanar features in space, a new feature plane is constructed through the relationship between arbitrary feature points and feature lines. The features of the actual feature plane before and after image distortion are maintained by preserving the shape of the newly constructed feature plane. For example... Figure 5As shown, in the same image, if there are any two non-collinear feature lines AB and CD, a point P outside of lines AB and CD will necessarily form a plane with line AB. The planar properties of plane PAB can be constrained by the changes in the length and slope of the three lines PA, PB, and AB. However, in practice, only line AB can be obtained as a direct feature through line detection; lines PA and PB are indirect features constructed through feature points. Similarly, the constructed planes PCD, PAD, and PBC can be constrained in the same way. Generally, when lines AB and CD are not coplanar, the four planes constructed can constrain the relative distances and positional relationships between different feature objects, ensuring the change in distance between objects in the image before and after deformation. When lines AB and CD are coplanar, the constructed planes can further constrain the shape changes of plane ABCD before and after image deformation, maintaining the planar properties of the image.

[0140] For the target image I and the reference image I′, the images are meshed according to a certain size, and a set of images to be stitched are meshed. A vector V = [x1 y1 x2 y2 ... x ... y2 ... x ... y2 ... ... x ... ... n y n ] T Describing the coordinates of the original mesh vertices, the corresponding deformed mesh vertices are: For any point in the image, it can be represented by its four nearest neighbors using bilinear interpolation, which is represented by the function σ(·) in this invention. The overall energy equation is expressed as:

[0141]

[0142] in The problems of feature deformation and global scaling in image deformation are solved by constraining the distance and angle between features. The alignment problem of overlapping areas in images is solved by constraining matching points and matching lines. The distortion problem encountered in splicing is solved by constraining the grid lines. The problem of straightness is solved by constraining the straightness of the same feature line using local and global feature lines.

[0143] Area feature preservation and distance feature preservation are crucial for maintaining the proportions of deformed images during image stitching. Their definitions are as follows:

[0144]

[0145] in and Constrain the length and angle of the planar boundary line, respectively. λ sd and λ sa These are the weights of the corresponding constraint terms.

[0146] The straight line in image I It is constructed from feature points and feature lines present in the image, as can be seen from the above. i The length constraint is a distance constraint between different features, l i The angular constraint serves as the positional constraint between different features. The length and angular constraints work together to constrain the relative positions and distances between different objects in the image. Furthermore, constraints can be applied to coplanar features in space. Assuming image I has a corresponding deformed image I′, the straight line... Considering that there will inevitably be lines with excessively long lengths, constraining the length and angle of a line solely by its start and end points can easily overlook its overall characteristics. Therefore, in the calculation, the line l is constrained by M points. i Divide into straight lines evenly L j The starting and ending points can be represented as The corresponding straight line in image I′ can be obtained. and points Right now

[0147]

[0148] The function σ(·) is defined as the calculation of the corresponding coordinates of a point, for example... For point Image pixel coordinates and points Perform corresponding calculations on the image pixel coordinates.

[0149] The relative directional changes between different features during deformation are controlled by constraining the angle of the constructed straight line.

[0150]

[0151] in

[0152] Establish evaluation indicators

[0153] To quantitatively evaluate the performance of the image stitching method proposed in this invention, evaluation metrics were designed for the performance of surface features in the image and the variation of distance characteristics between different features. For any two distinct features (taking feature point p and feature line l as an example) in the overlapping regions of a set of images to be stitched, the distance between features passing through point p... j and line l i The two endpoints Construct indirect features respectively ( Figure 6 As shown), the indirect features after deformation are respectively denoted as The degree of deformation is defined as follows:

[0154]

[0155]

[0156] Where, K = min(num(l) i ),num(p j )), num(l i ) and num(p j ) represent the number of detected feature points and feature lines, respectively, and len(l) represents the length of the line segment.

[0157] The stitching result of some images is as follows Figure 7 As shown in Table 1, the quantitative evaluation indicators are as follows. Since this invention is an improvement on the LPC method and belongs to a single-view stitching method, the quantitative evaluation is only compared with SPW and LPC, which are also single-view stitching methods. The results are as follows:

[0158] Table 1 Quantitative Evaluation Results of Image Stitching

[0159]

[0160]

[0161] Depend on Figure 7 It can be seen that the method proposed in this invention exhibits certain advantages in image stitching, mainly in its better preservation of detailed features and excellent alignment effect. The stitched image almost does not show ghosting, while LPC is poor at preserving image detail information. In the quantitative evaluation of image stitching quality, the method in this invention also shows significant advantages, with the greatest performance improvement in column stitching, where D... dis Increased by 28.8%, D dir The improvement of 63.2% indicates that the method of this invention is effective in complex fields. Specific Implementation Example 3:

[0163] The present invention provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement a natural image stitching method based on extended dot-line features. Specific Implementation Example 4:

[0165] The present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement a natural image stitching method based on extended point and line features.

[0166] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or N embodiments or examples. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of the present invention, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified. Any process or method described in the flowcharts or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more N executable instructions for implementing custom logical functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order according to the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain. The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection having one or N wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic device, and portable optical disc read-only memory (CDROM).Furthermore, the computer-readable medium can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory. It should be understood that various parts of the invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0167] The above description is merely a preferred embodiment of a natural image stitching method based on extended point and line features. The scope of protection for such a method is not limited to the above embodiments; all technical solutions falling within this conceptual framework are within the scope of protection of this invention. It should be noted that for those skilled in the art, any improvements and variations made without departing from the principles of this invention should also be considered within the scope of protection of this invention.

Claims

1. A natural image stitching method based on extended point and line features, characterized by: The method includes the following steps: Step 1: Perform geometric operations on straight lines in the image by connecting region line segments to preserve the line features of overlapping areas in the image; Step 2: Generate point features using the matched line features, supplement the matched points, optimize image pre-alignment, and preserve the geometric features of the image as much as possible; Step 2 specifically involves: By generating more feature points through the expanded matching line features, the matching features are expanded. This is achieved for two pairs of coplanar matching lines in a set of images to be stitched together. and their intersection This results in a new set of matching feature points; by matching any two matching feature lines in the image, feature points located within the image region are selected to expand the matching feature point pairs. RANSAC is used to remove mismatches from the extended matching point pairs, thereby improving the reliability of the matching point pairs. Since the straight line has infinite extension, the intersection point exists at any position in the plane, which to some extent extends the matching features to the weak texture region of the image, solves the problem of fewer features in the weak texture region to a certain extent, and improves the quality of image pre-alignment and stitching. Step 3: Perform image pre-alignment, homography matrix estimation, and meshing deformation operations based on the matched point and line features to achieve image stitching; Step 3 specifically involves: Step 3.1: Let and It is a pair of images and A pair of matching points in the overlapping region, where , Then point and The correspondence between them is established through the homography matrix. Represented as: （1） （2） in, These are the homogeneous coordinates of p. It is a homography matrix; in a non-homogeneous coordinate system, （3） Combining the homography matrix estimation methods in APAP and QH, the final homography distortion is defined as: （4） in, and These are the homography matrices obtained using methods from QH and APAP, respectively. From Extrapolation The homography matrix; Step 3.2: The image stitching method based on gridded deformation divides the image to be stitched into multiple grid regions of the same size, and uses the grid region as the smallest deformation object for homography matrix estimation. The optimized energy equation is defined as follows: (1) in The problems of surface feature deformation and global scaling in the image stitching process are solved by constraining the distance and angle between features. The alignment problem of overlapping areas in an image is solved by constraining matching points and matching lines. The distortion problem encountered in splicing is solved by constraining the grid lines. The problem of straightness is solved by constraining the straightness of the same feature line using local and global feature lines.

2. The method according to claim 1, characterized in that: Step 1 specifically involves: Step 1.1: By using a method based on connecting line segments in the region, multiple line segments in similar regions are restored into continuous salient feature lines, thereby preserving the continuity of salient line features as much as possible and improving the pre-alignment quality; By merging collinear line segments within the same region, they are restored to continuous salient feature lines as much as possible. This preserves the characteristics of the line structure as much as possible during line feature matching. Furthermore, the slope of the two lines is evaluated during the line segment connection process. and The slopes of the straight lines should be the same or very close. Step 1.2: Calculate line segments The End to line segment The starting point distance Perform a join and merge; Step 1.3: Finally, process the merged lines. slope To determine, its slope and and Only those that are very close can be preserved; After multiple iterations, all line segments within the overlapping area are identified and connected, completing the connection of line segments in the region and saving the line features of the overlapping area of ​​the image.

3. The method according to claim 2, characterized in that: the proportion of deformed images maintained by surface feature preservation and distance feature preservation during image stitching is expressed by the following formula: （6） in, and Constrain the length and angle of the planar boundary line respectively. and These are the weights of the corresponding constraint terms; image straight line in It is constructed from feature points and feature lines present in the image. The length constraint is a distance constraint between different features. Angle constraints serve as positional constraints between different features, while length and angle constraints work together to achieve relative position and relative distance constraints between different objects in the image. Constraining the features of coplanar surfaces in space, making the image Its corresponding deformed image is ,straight line Considering that there will inevitably be lines with excessively long lengths, the calculation is performed by... The points will form a straight line. Divide into straight lines evenly , The starting and ending points are represented as The corresponding image can be obtained. straight line in and points : （7） Among them, the function Defined as calculating the corresponding coordinates of a point. For point Image pixel coordinates and points Perform corresponding calculations on the image pixel coordinates. ; The relative directional changes between different features during deformation are controlled by constraining the angle of the constructed straight line. （8） in, .

4. The method according to claim 1, characterized in that: The method also includes evaluation indicators, specifically: To quantitatively evaluate the performance of image stitching methods, evaluation metrics were designed for the performance of surface features in images and the variation of distance characteristics between different features. For any two distinct features in the overlapping regions of a set of images to be stitched, the feature points are used as the evaluation metrics. and feature lines For example, after point and straight line The two endpoints , Construct indirect features respectively , The indirect features after deformation are denoted as follows: , The degree of deformation is defined as follows: （9） （10） in, , and These represent the number of detected feature points and feature lines, respectively. Represents the length of the line segment; Establish a quantitative evaluation index table to evaluate the method.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the method as described in any one of claims 1-4.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the method of any one of claims 1-4.

Images