A parallax image splicing method based on image depth and feature density guidance

Abstract

The application discloses a parallax image splicing method based on image depth and feature density guidance, and relates to the technical field of video image splicing. The application sequentially adopts eight processing steps of feature extraction, initial matching pair construction, grid error matching filtering, splicing line pre-planning, splicing line grid projection field construction, pixel-level dense projection field generation, to-be-registered image projection transformation and image fusion, constructs a pre-planned splicing line based on depth cost and feature density cost, accurately guide a local structure consistent feature set to construct a local projection field, suppresses distortion projection caused by local cross-structure features, improves the robustness of projection field modeling, and is suitable for parallax image splicing of an ultralow-altitude complex space structure scene.

Description

Technical Field

[0001] This invention relates to the field of video image stitching technology, specifically providing a parallax image stitching method based on image depth and feature density guidance, which can be used for parallax image stitching in ultra-low altitude complex spatial structure scenes. Background Technology

[0002] With the iterative upgrades and widespread adoption of small rotary-wing drone technology, ultra-low-altitude aerial photography has become an indispensable core data acquisition method in many fields such as urban monitoring, terrain exploration, film and television production, and military reconnaissance. However, ultra-low-altitude shooting scenarios have inherent characteristics, including the highlighting of the three-dimensional geometric properties of objects due to the low shooting height, the spatial structural heterogeneity caused by complex terrain, and complex non-single-plane problems such as local areas spanning multiple geometric structures. These pose a severe challenge to the fully automatic seamless stitching of monocular drone images.

[0003] Traditional image stitching algorithms (such as SIFT, SURF, and their various improved variants based on feature point matching) can achieve ideal stitching results when dealing with near-planar scenes. However, in ultra-low-altitude scenes with multiple geometric structures, the local area spanning multiple structures can cause the global projection model based on homography to fail, resulting in large-scale structural distortion and misalignment in the stitching result, which severely restricts the visual effect of the stitching.

[0004] Although current mainstream image stitching algorithms have made their own technological breakthroughs in parallax scene processing, they still have significant limitations when facing complex ultra-low-altitude aerial photography scenes, as follows:

[0005] (1) Local spatial transformation algorithms (such as APAP, AANAP, REW, etc.) improve image alignment accuracy by constructing local projection models. Although they are effective in normal scenes, when the scene geometry changes drastically, the spatial complexity of the projection field will exceed the fitting ability of the local model, resulting in problems such as perspective distortion and structural misalignment in the stitching results.

[0006] (2) Traditional splicing line driven algorithms (such as SEAGULL, AutoStitch, OBJ-GSP, and commercial software PTGUI) adjust the position of the splicing line through iterative optimization strategies to continuously improve the visual consistency of the splicing line neighborhood. However, in ultra-low altitude complex structure scenarios, due to the sparsity of extracted feature points and the underfitting of feature consistency estimation, the purified feature points cannot completely cover non-dominant geometric surfaces, resulting in large areas without feature coverage. When the splicing line cannot bypass the complex geometric region without feature coverage, the iterative planning model will fail, and in extreme cases, it may even terminate the iteration process directly. Summary of the Invention

[0007] In view of this, the present invention provides a disparity image stitching method guided by image depth and feature density. The present invention features high computational efficiency and strong robustness, and is particularly suitable for disparity image stitching in scenarios with multiple geometric structures.

[0008] The technical solution adopted in this invention is as follows:

[0009] A parallax image stitching method guided by image depth and feature density includes the following steps:

[0010] Step 1: Extract the SURF feature point sets of the image to be stitched (I) and the reference image (J), denoted as P. I and P J ;

[0011] Step 2, calculate P using the brute-force matching algorithm. I With P J The initial feature matching point pairs of the set form the initial feature matching point pair set P0;

[0012] Step 3: Divide P0 into multiple local feature matching subsets; independently estimate the optimal affine transformation parameters within each subset and construct a gridded affine projection field; filter out mismatched pairs in P0 based on the gridded affine projection field to obtain a purified feature matching point pair set P1.

[0013] Step 4: Use the Depth-Anything depth estimation model to extract the depth information of the images to be stitched together, and generate a depth map D. I Combined with depth map D I Using set P1, we construct depth cost function and feature density cost function respectively, and then establish a unified cost function for pre-planning the stitching line; we use dynamic programming algorithm to solve the unified cost function to obtain the optimal pre-planned stitching line L. S ;

[0014] Step 5, combine with depth map D I With pre-planned splicing line L S Construct a set of splicing support feature matching point pairs P S This leads to the construction of a splicing line grid projection field. ;

[0015] Step 6, Projecting the field based on the stitching line grid Pixel-level dense projection field H is generated through weighted interpolation. I ={h q (i, j) | (i, j)∈I}, where (i, j) represents the pixel coordinates in image I, h q (i, j) represents the affine transformation model of the pixel (i, j);

[0016] Step 7, based on the pixel-level dense projection field H I The image I to be stitched is projected pixel by pixel onto the reference image space to obtain the projected image I'.

[0017] Step 8, align I' and the reference image J along the pre-planned stitching line L S Linear weighted fusion is performed to obtain the stitched image I. P .

[0018] Furthermore, step 3 is performed as follows:

[0019] Step 301, construct a structure containing P in the image I to be stitched. I The smallest bounding rectangle of all feature points in the region is given, with width W and height H. The rectangle is then meshed, with side length s and the number of grid cells in the vertical direction being... The number of grid cells in the horizontal direction is , Let G(x, y) denote rounding up. The grid cell in the x-th column and y-th row is denoted as G(x, y), where... , ;

[0020] Step 302: Construct a feature space R(x,y) for each grid cell to represent the set of coordinates of all pixels within the grid cell G(x,y).

[0021]

[0022] Step 303, set the initial feature matching point pair set. Where K0 is the number of initial feature matching point pairs, p k For set P I Feature points in, p' k For set P J In and p k Matched feature points; the initial feature matching point pairs falling within the R(x,y) space are used to form a local feature set P(x,y):

[0023]

[0024] Step 304: Within the local feature set P(x, y), the affine transformation model h(x, y) of the grid G(x, y) is obtained using the random sampling consensus algorithm; the affine transformation models are then combined to form a gridded affine projection field H. G ;

[0025]

[0026] Step 305: Based on the local projection model h(x, y), filter out mismatched pairs in P(x,y) and remove feature matching point pairs that do not satisfy the following relationship:

[0027]

[0028] in, The threshold value ranges from 1 to 4.

[0029] After filtering out all mismatched pairs in P(x,y), the remaining feature matching point pairs are the purified feature matching point pair set. K1 represents the number of feature matching point pairs after purification.

[0030] Furthermore, step 4 is specifically implemented as follows:

[0031] Step 401: Calculate the feature density at pixel coordinates (i, j) in image I. :

[0032]

[0033] Where || represents the Euclidean distance, Represented as a smoothing scale parameter, The Gaussian kernel function is expressed as follows:

[0034]

[0035] Step 402, for Normalization is performed:

[0036]

[0037] Step 403, construct the feature density cost function :

[0038]

[0039] Step 404: Generate the depth map D corresponding to image I using the Depth-Anything depth estimation model. I D I element D I (i, j) represents the depth value at pixel coordinates (i, j) in image I; obtain D I histogram F D The statistical frequency of depth value x is F. D (x), using the depth value C with the highest frequency as the baseline, construct the depth cost function E for the splicing line. d (i, j):

[0040]

[0041] Step 405, for E d(i, j) are normalized:

[0042]

[0043] Step 406: Construct a unified cost function E(i, j) based on the normalized feature density cost function and depth cost function:

[0044]

[0045] Where α1 and α2 represent weighting coefficients, and α1 + α2 = 1;

[0046] Step 407: Solve using the dynamic programming algorithm, defining the cumulative cost function M(i, j) as follows:

[0047]

[0048] E(i, j) is a unified cost function. Neighborhood terms outside the image range are taken as infinity. The cumulative cost is calculated column by column according to this recursive relationship, and the backtracking path is recorded. The pixel with the smallest cumulative cost in the last column is taken as the endpoint, and backtracking is performed to obtain the pre-planned stitching line L. S .

[0049] Furthermore, step 5 is implemented as follows:

[0050] Step 501, add the center stitching line L in image I. S L is composed of points whose minimum Euclidean distance is less than β pixels. S neighborhood space R L For feature matching point pairs (p) s , p' s ), find L S above and p s The nearest point p lmin , determine (p s , p' s Does it meet the following conditions:

[0051] Condition 1: (p s , p' s In the set of feature matching point pairs P1;

[0052] Condition 2: p s The coordinates fall on R L within the area;

[0053] Condition 3: p s With p lmin The difference in depth values ​​at each location is less than the threshold δ, where δ ranges from 1 to 3.

[0054] The feature matching point pairs that simultaneously meet the above conditions will form a set P of splicing support feature matching point pairs. S ;

[0055] Step 502, construct based on P S The gridded affine projection field, denoted as Let the side length of the grid be 'a', and the number of grid cells in the vertical direction be 'a'. The number of grid cells in the horizontal direction is The grid in column x and row y is G. s (x, y), v s (x, y) is the grid G s The pixel coordinates of the top left vertex of (x, y), h s (x, y) is the grid G s The affine transformation model of (x, y) is then The expression is as follows:

[0056] .

[0057] Furthermore, step 6 is performed as follows:

[0058] For pixel coordinates q=(i, j) in the image I to be stitched, the affine transformation model h corresponding to point q is... q (i, j) is:

[0059]

[0060] Pixel-level dense projection field H I ={h q (i, j) | (i, j)∈I}.

[0061] Furthermore, the specific method of linear weighted fusion in step 8 is as follows:

[0062]

[0063] In the formula, (i', j') are the pixels on the projection image I', and (i', j') > L. s This indicates that point (i', j') is at the splicing line L. s Above, (i', j')=L s This indicates that point (i', j') is at the splicing line L. s Above, (i', j') <L s This indicates that point (i', j') is at the splicing line L. s Below.

[0064] Compared with the prior art, the present invention has the following advantages:

[0065] 1. Based on depth cost and feature density cost, this invention constructs a pre-planned splicing line, which accurately guides the local structural consistency feature set to construct the local projection field, suppresses the distortion projection introduced by local cross-structural features, and improves the robustness of projection field modeling.

[0066] 2. Based on depth cost and feature density cost, this invention constructs a pre-planned stitching line, which accurately guides the local structural consistency feature set to construct the local projection field, suppresses the distortion projection introduced by local cross-structural features, improves the robustness of projection field modeling, and is suitable for parallax image stitching in ultra-low altitude complex spatial structure scenes.

[0067] 3. This invention constructs a gridded mismatch filtering method, which retains all possible features on the structural surface and significantly improves robustness compared with the traditional coefficient feature matching method. Attached Figure Description

[0068] Figure 1 This is a flowchart of a parallax image stitching method guided by image depth and feature density.

[0069] Figure 2 This is a schematic diagram of the image feature extraction results in an embodiment of the present invention.

[0070] Figure 3 This is a schematic diagram of the initial feature matching point pair construction in an embodiment of the present invention.

[0071] Figure 4 This is a schematic diagram of grid division in an embodiment of the present invention.

[0072] Figure 5 This is a schematic diagram of the gridded mismatch filtering results in an embodiment of the present invention.

[0073] Figure 6 This is the feature density map of the image to be stitched in this embodiment of the invention.

[0074] Figure 7 This is a depth map of the image to be stitched in this embodiment of the invention.

[0075] Figure 8 This is a schematic diagram of the pre-planned splicing line result in an embodiment of the present invention.

[0076] Figure 9 This is a schematic diagram of the splicing line grid division in an embodiment of the present invention.

[0077] Figure 10 This is a schematic diagram of the image fusion result in an embodiment of the present invention.

[0078] Figure 11 This is a comparison chart of the effects of related splicing algorithms. Detailed Implementation

[0079] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments. Obviously, these descriptions are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the following embodiments without creative effort are within the protection scope of the present invention.

[0080] A parallax image stitching method guided by image depth and feature density includes the following steps:

[0081] (1) Feature extraction: Extract the SURF feature point sets of the image to be stitched I and the reference image J respectively, and denote them as P. I and P J ;

[0082] (2) Initial matching pair construction: P is calculated using the brute-force (BF) matching algorithm. I With P J The initial feature matching point pairs of the set are denoted as Where K0 represents the number of initial feature matching point pairs;

[0083] (3) Mesh-based mismatch filtering: First, P0 is divided to obtain local feature matching subsets; then, the optimal affine transformation parameters are independently estimated in each subset to construct a mesh-based affine projection field; finally, mismatch pairs in P0 are filtered based on the mesh-based affine projection field to obtain a purified set of feature matching point pairs, denoted as P1; Step (3) specifically includes the following steps:

[0084] (301) Construct a structure containing P in the image I to be stitched together. I The minimum bounding rectangle of all feature points is used to divide this rectangle into a grid with the following parameters: the minimum bounding rectangle has dimensions W×H, the grid side length is s, and the number of grid cells in the vertical direction is... The number of grid cells in the horizontal direction is Let G(x, y) be the grid cell in the x-th column and y-th row.

[0085] (302) Construct a feature space R(x,y) for each grid:

[0086]

[0087] (303) Form a local feature set P(x,y) by matching the initial feature points that fall in the R(x,y) space:

[0088]

[0089] (304) Within the local feature set P(x, y), the affine transformation model h(x, y) of the grid G(x, y) can be obtained by using the Random Sample Consensus (RANSAC) algorithm; the affine transformation models are then combined to form a gridded affine projection field H. G ;

[0090]

[0091] The affine transformation model h(x, y) is a matrix. For the specific process of solving the affine transformation model h(x, y), please refer to the literature "Fischler MA, Bolles R C. Random sample consensus: aparadigm for model fitting with applications to image analysis and automated cartography[J]. Communications of the ACM,1981,24(6):381-395." The Random Sample Consensus Algorithm (RANSAC) takes a set of matching points in an image as input and outputs a matrix h. Its principle is to generate the matrix h by fitting and filtering the matching points.

[0092] (305) Based on the local projection model h(x, y), perform mismatch filtering on P(x,y) and remove feature matching pairs that do not satisfy the following relation:

[0093]

[0094] here It is a constant, and a value of 3 is recommended.

[0095] After simultaneously performing mismatch filtering on all local feature sets, the remaining feature matching point pairs constitute the purified feature matching point pair set P1:

[0096]

[0097] Where K1 represents the number of purified matching point pairs.

[0098] (4) Pre-planning of stitching lines: First, the depth information of the image to be stitched is extracted using the Depth-Anything depth estimation model to generate a depth map D. I Then, combined with depth map D I For set P1, a depth cost function and a feature density cost function are constructed respectively, and a unified cost function for pre-planning the stitching line is established based on the above cost functions. Finally, a dynamic programming algorithm is used to solve the unified cost function to obtain the optimal pre-planned stitching line L. S ;

[0099] Step (4) specifically includes the following steps:

[0100] (401) Order Let represent the feature density at pixel coordinates (i, j) in I, and its expression is as follows:

[0101]

[0102] in, Represented as a smoothing scale parameter, The Gaussian kernel function is expressed as follows:

[0103]

[0104] (402) After normalization, we get:

[0105]

[0106] (403) Order The feature density cost function is expressed as follows:

[0107]

[0108] (404) Let D I Let D represent the geometric depth map corresponding to image I, generated by a depth estimation model, such as the Depth-Anything algorithm. I (i, j) represents the depth value at pixel coordinates (i, j). Let F D D represents I histogram, F D Let (x) represent the statistical frequency of depth value x. Let C represent the depth value with the highest statistical frequency.

[0109] This method takes the principle of the stitching line traversing areas with small depth fluctuations as its core principle. Therefore, the depth value with the highest statistical frequency is used as the benchmark value, and the depth cost function E of the stitching line is then calculated. d The expression for (i, j) is as follows:

[0110]

[0111] (405) For E d Normalizing (i, j) yields:

[0112]

[0113] (406) A unified cost function, denoted as E(i, j), is constructed from the feature density cost function and the depth cost function. Its mathematical expression is as follows:

[0114]

[0115] Here, α1 and α2 represent weighting coefficients, and it is recommended that both be 0.5.

[0116] (407) Based on E(i, j), the pre-planned splicing line is obtained by dynamic programming (DP) algorithm, denoted as L. S Its expression is as follows:

[0117]

[0118] Where, p l Indicates the splicing line L S The pixel coordinates are given by L1, where L1 represents the number of pixels on the splicing line.

[0119] L S For the specific solution process, please refer to the literature "Image Quilting for Texture Synthesis and Transfer" and the patent application number 202110300585 "A panoramic image stitching method based on the best suture line".

[0120] (5) Construction of splicing line grid projection field: First, combine the depth map D I With pre-planned splicing line L S Construct a set of splicing support feature matching point pairs P S Then, based on the splicing support feature matching point pair set P S Constructing the splicing line grid projection field ;

[0121] Step (5) specifically includes the following steps:

[0122] (501) Define L S The neighborhood space: the stitching line L in image I. S The region whose Euclidean distance is less than β pixels is denoted as R. L Its expression is as follows:

[0123]

[0124] Where β is a constant, and a value of 50 is recommended; F(p) R , L S ) represents point p R Distance L S The minimum Euclidean distance.

[0125] Let (p) s , p' sLet F(p) represent a set of feature matching point pairs, assuming that F(p) satisfies the following conditions on the splicing line. s , L S The point is p. lmin Then (p) s , p' s The conditions for a splicing support point are:

[0126] Condition 1: (p s , p' s In set P1;

[0127] Condition 2: p s The coordinates fall on R L within the area;

[0128] Condition 3: p s With p lmin The difference in depth values ​​at each location is less than δ. Here, δ is a constant, and a value of 2 is recommended.

[0129] Extract all splicing support points of P1 to form a set of splicing support feature matching point pairs P. S .

[0130] (502) Construct a system based on P according to steps 301 to 304. S The gridded affine projection field, denoted as Let 'a' be the grid side length, and the number of grid cells in the vertical direction be... The number of grid cells in the horizontal direction is The grid in column x and row y is G. s (x, y), v s (x, y) is the grid G s The pixel coordinates of the top left vertex of (x, y), h s (x, y) is the grid G s The affine transformation model of (x, y) is then The expression is as follows:

[0131]

[0132] (6) Generation of pixel-level dense projection field: Based on the above splicing line grid projection field Pixel-level dense projection field H is generated through weighted interpolation. I ={h q (i, j) | (i, j)∈I}. Where (i, j) represents the pixel coordinates in image I, h q (i, j) represents the affine transformation model for the corresponding pixel position;

[0133] Step (6) specifically includes the following steps:

[0134] Let q=(i, j) represent the coordinates of any pixel in the image to be stitched, and h q (i, j) represents the affine transformation model corresponding to point q, then:

[0135]

[0136] Affine transformation models are constructed synchronously for all pixels in image I to obtain a pixel-level dense projection field, denoted as H. I ={h q (i, j) | (i, j)∈I}.

[0137] (7) Projection transformation of the image to be registered: based on pixel-level dense projection field H I The image I to be stitched is projected pixel by pixel onto the reference image space to obtain the projected image I'.

[0138] (8) Image fusion: I' and the reference image J are fused along the pre-planned stitching line L S Perform linear weighted fusion to obtain the fused image I. P :

[0139]

[0140] In the formula, (i', j') are the pixels on the projection image I', and (i', j') > L. s This indicates that point (i', j') is at the splicing line L. s Above, (i', j')=L s This indicates that point (i', j') is at the splicing line L. s Above, (i', j') <L s This indicates that point (i', j') is at the splicing line L. s Below.

[0141] Since feature matching has been completed in step 1, this method stores features as pairs of matching points by default. For example, (p k , p' k ) represents a set of matching point pairs, where p k p' represents a feature point in image I. k This represents the corresponding feature point in image J. All feature point-based subset partitioning, feature filtering, and other operations in this method represent the subdivision of feature points {p} in image I. k The features are then processed. Since the features are stored as pairs of matching points, the corresponding feature points in image J are also processed synchronously using the same method.

[0142] Here is a more specific example:

[0143] A parallax image stitching method based on image depth and feature density guidance, the process of which is as follows: Figure 1 As shown, the main processing steps include: feature extraction, initial matching pair construction, gridded mismatch filtering, stitching line pre-planning, stitching line grid projection field construction, pixel-level dense projection field generation, projection transformation of the image to be registered, and image fusion. Details are as follows:

[0144] (1) Feature extraction: Extract the SURF feature point sets of the image to be stitched I and the reference image J respectively, and denote them as P. I and P J The result of image feature point extraction is as follows: Figure 2 As shown.

[0145] (2) Initial matching pair construction: P is calculated using the brute-force (BF) matching algorithm. I With P J The initial feature matching point pairs of the set are denoted as K0 represents the number of initial feature matching point pairs. Figure 3 This diagram illustrates the construction of some matching pairs; the red lines represent the P pairs obtained by the BF algorithm. I With P J The partial feature matching pairs of the set show that the SURF and BF algorithms produce a large number of incorrect matching pairs when stitching parallax images of complex spatial structures at ultra-low altitudes.

[0146] (3) Mesh-based mismatch filtering: First, P0 is divided to obtain local feature matching subsets; then, the optimal affine transformation parameters are independently estimated within each subset to construct a mesh-based affine projection field; finally, mismatch pairs in P0 are filtered based on the mesh-based affine projection field to obtain a purified set of feature matching point pairs P1. Specifically, the following steps are included:

[0147] First, regarding P I The distribution area (i.e., the minimum bounding rectangle) is meshed with the following parameters: region size is W×H, mesh side length is s, and the number of grid cells in the vertical direction is... The number of grid cells in the horizontal direction is The grid in column x and row y is denoted as G(x, y).

[0148] Construct a feature space R(x,y) for each grid cell:

[0149]

[0150] Figure 4 The mesh generation results are shown, P IThe distribution area is divided into a 17×10 grid space, with each grid having a side length of s, x ranging from 1 to 17, and y ranging from 1 to 10. The top-left grid is denoted as G(1, 1), and R(1,1) represents the set of coordinates of all pixels within grid G(1, 1); the bottom-right grid is denoted as G(17, 10), and R(17,10) represents the set of coordinates of all pixels within grid G(17, 10).

[0151] Let P(x,y) denote the local feature set consisting of the initial feature matching point pairs falling in the R(x,y) space, as expressed below:

[0152]

[0153] Within the local feature set P(x, y), the affine transformation model h(x, y) of the grid G(x, y) can be obtained by using the Random Sample Consensus (RANSAC) algorithm.

[0154] Each affine transformation model is combined into a gridded affine projection field H. G ;

[0155]

[0156] Based on the affine transformation model h(x, y), mismatch filtering is performed on P(x,y), eliminating feature matching pairs that do not satisfy the following relation:

[0157]

[0158] here It is a constant, and a value of 3 is recommended.

[0159] After filtering out all mismatched pairs in P(x,y), the remaining feature matching point pairs are the purified feature matching point pair set. K1 represents the number of feature matching point pairs after purification.

[0160] Figure 5 The results after gridded mismatch filtering are shown. Figure 3 After the feature points in step 3 are processed, most of the incorrect matching pairs are filtered out by the algorithm. After purification, the feature point matching pairs P1 are basically located in the overlapping area of ​​the image to be stitched I and the reference image J.

[0161] (4) Pre-planning of stitching lines: First, the depth information of the image to be stitched is extracted using the Depth-Anything depth estimation model to generate a depth map D. I Then, combined with depth map D IFor set P1, a depth cost function and a feature density cost function are constructed respectively, and a unified cost function for pre-planning the stitching line is established based on the above cost functions. Finally, a dynamic programming algorithm is used to solve the unified cost function to obtain the optimal pre-planned stitching line L. S Specifically, it includes the following steps:

[0162] make Let represent the feature density at pixel coordinates (i, j) in I, and its expression is as follows:

[0163]

[0164] in, Represented as a smoothing scale parameter, The Gaussian kernel function is expressed as follows:

[0165]

[0166] Figure 6 The results of the above calculations on the image I to be stitched are visualized, namely, the feature density map of the image to be stitched. Low-density areas are mapped to blue, and high-density areas are mapped to red. The color intensity is positively correlated with the local feature density. It can be seen that the feature point matching exhibits high-density features in the dense region P1.

[0167] right After normalization, we get:

[0168]

[0169] make The feature density cost function is expressed as follows:

[0170]

[0171] Let D I Let D represent the geometric depth map corresponding to image I, generated by a depth estimation model, such as the Depth-Anything algorithm. I (i, j) represents the depth value at pixel coordinates (i, j), F D D represents I histogram, F D Let (x) represent the statistical frequency of depth value x. Let C represent the depth value with the highest statistical frequency.

[0172] Background depth values ​​are typically distributed within the range [0, 2], and this region accounts for the largest proportion of the image. Figure 7 The geometric depth map of image I to be stitched is shown, with the black area representing the background area.

[0173] The core principle is to ensure that the stitching line passes through areas with minimal depth fluctuations, and to make it pass through areas with the highest statistical frequency whenever possible. Figure 7 The black background area is crossed. Taking the depth value with the highest statistical frequency as the benchmark, the depth cost E of the stitching line is then calculated. d The expression for (i, j) is as follows:

[0174]

[0175] For E d After normalizing (i, j), we get:

[0176]

[0177] A unified cost function, denoted as E(i, j), is constructed from the feature density cost function and the depth cost function, and its mathematical expression is as follows:

[0178]

[0179] Here, α1 and α2 represent weighting coefficients, and it is recommended that both be 0.5.

[0180] Based on the above formula, the pre-planned splicing line can be obtained by solving the dynamic programming (DP) algorithm, denoted as L. S Its expression is as follows:

[0181]

[0182] Where, p l Indicates the splicing line L s The pixel coordinates are given by L1, where L1 represents the number of pixels on the splicing line. Figure 8 The pre-planned stitching line result for the image I to be stitched tends to pass through the background area with dense feature points.

[0183] (5) Construction of the splicing line grid projection field: The splicing line L in image I is constructed. S The region R with a Euclidean distance less than β pixels L Defined as L S The neighborhood space of is expressed as follows:

[0184]

[0185] Where β is a constant, and a value of 50 is recommended; F(p) R , L S ) represents point p R Distance L S The minimum Euclidean distance;

[0186] Let (p) s, p' s Let F(p) represent a set of feature matching pairs, assuming that F(p) satisfies the following conditions on the splice line. s , L S The point is p. lmin Then (p) s , p' s The conditions for a splicing support point are:

[0187] Condition 1: (p s , p' s In the feature matching set P1;

[0188] Condition 2: p s The coordinates fall on R L within the area;

[0189] Condition 3: p s With p lmin The difference in depth values ​​at each location is less than δ.

[0190] Where δ is a constant, and a value of 2 is recommended.

[0191] Extract all splicing support points of P1 to form a set of splicing support feature matching point pairs, denoted as P. S .

[0192] Following steps 301-304, construct a P-based system. S The gridded affine projection field, denoted as Let 'a' be the grid side length, and the number of grid cells in the vertical direction be... The number of grid cells in the horizontal direction is The grid in column x and row y is G. s (x, y), v s (x, y) is the grid G s The pixel coordinates of the top left vertex of (x, y), h s (x, y) is the grid G s The affine transformation model of (x, y) is then The expression is as follows:

[0193]

[0194] Figure 9 A schematic diagram of grid division based on splicing lines is shown.

[0195] (6) Generation of pixel-level dense projection field: Let q=(i, j) represent the coordinates of any pixel in the image to be stitched, h q (i,j) represents the affine transformation model corresponding to point q, then:

[0196]

[0197] Affine transformation models are constructed synchronously for all pixels in image I to obtain a pixel-level dense projection field, denoted as H. I ={h q (i, j) | (i, j)∈I}.

[0198] (7) Projection transformation of the image to be registered: based on pixel-level dense projection field H I The image I to be stitched is projected pixel by pixel into the reference image space, and its projection is denoted as I'.

[0199] (8) Image fusion: I' and the reference image J are fused along the pre-planned stitching line L S Perform linear weighted fusion, and fuse the images I. P The expression is as follows:

[0200]

[0201] Figure 10 The results of this method after stitching and fusing the image to be stitched (I) and the reference image (J) are shown.

[0202] In contrast. Figure 11 The performance of image stitching algorithms such as PtGUI, SEAGULL, and AANAP in this scenario is demonstrated. It is evident that in ultra-low-altitude complex structure scenarios, this invention preserves all possible features on the structural surfaces and suppresses distortion projections introduced by local cross-structural features. Compared to traditional stitching methods, robustness is significantly improved.

[0203] The above embodiments are merely specific implementations of the present invention, but the scope of protection of the present invention is not limited thereto. Those skilled in the art should understand that any modifications or equivalent substitutions to the specific implementations of the present invention that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A parallax image stitching method guided by image depth and feature density, characterized in that, Includes the following steps: Step 1: Extract the SURF feature point sets of the image to be stitched (I) and the reference image (J), denoted as P. I and P J ; Step 2, calculate P using the brute-force matching algorithm. I With P J The initial feature matching point pairs of the set form the initial feature matching point pair set P0; Step 3: Divide P0 into multiple local feature matching subsets; The optimal affine transformation parameters are independently estimated within each subset to construct a gridded affine projection field. Based on the gridded affine projection field, mismatched pairs in P0 are filtered to obtain a purified set of feature matching point pairs P1. Step 4: Use the Depth-Anything depth estimation model to extract the depth information of the images to be stitched together, and generate a depth map D. I Combined with depth map D I Using set P1, we construct depth cost function and feature density cost function respectively, and then establish a unified cost function for pre-planning the stitching line; we use dynamic programming algorithm to solve the unified cost function to obtain the optimal pre-planned stitching line L. S ; Step 5, combine with depth map D I With pre-planned splicing line L S Construct a set of splicing support feature matching point pairs P S This leads to the construction of a splicing line grid projection field. ; Step 6, Projecting the field based on the stitching line grid Pixel-level dense projection field H is generated through weighted interpolation. I ={h q (i,j) | (i, j)∈I}, where (i, j) represents the pixel coordinates in image I, h q (i, j) represents the affine transformation model of the pixel (i, j); Step 7, based on pixel-level dense projection field H I The image I to be stitched is projected pixel by pixel onto the reference image space to obtain the projected image I'. Step 8, align I' and the reference image J along the pre-planned stitching line L S Linear weighted fusion is performed to obtain the stitched image I. P .

2. The parallax image stitching method based on image depth and feature density guided according to claim 1, characterized in that, The specific method for step 3 is as follows: Step 301, construct a structure containing P in the image I to be stitched. I The smallest bounding rectangle of all feature points in the region is given, with width W and height H. The rectangle is then meshed, with side length s and the number of grid cells in the vertical direction being... The number of grid cells in the horizontal direction is , Let G(x, y) denote rounding up. The grid cell in the x-th column and y-th row is denoted as G(x, y), where... , ; Step 302: Construct a feature space R(x,y) for each grid cell to represent the set of coordinates of all pixels within the grid cell G(x,y). Step 303, set the initial feature matching point pair set. Where K0 is the number of initial feature matching point pairs, p k For set P I Feature points in, p' k For set P J In and p k Matched feature points; the initial feature matching point pairs falling within the R(x,y) space are used to form a local feature set P(x,y): Step 304: Within the local feature set P(x, y), the affine transformation model h(x, y) of the grid G(x, y) is obtained using the random sampling consensus algorithm; the affine transformation models are then combined to form a gridded affine projection field H. G ; Step 305: Based on the affine transformation model h(x, y), filter out mismatched pairs in P(x,y) and remove feature matching point pairs that do not satisfy the following relationship: in, The threshold value ranges from 1 to 4. After filtering out all mismatched pairs in P(x,y), the remaining feature matching point pairs are the purified feature matching point pair set. K1 represents the number of feature matching point pairs after purification.

3. The parallax image stitching method based on image depth and feature density guided according to claim 1, characterized in that, The specific method for step 4 is as follows: Step 401: Calculate the feature density at pixel coordinates (i, j) in image I. : Where || represents the Euclidean distance, Represented as a smoothing scale parameter, The Gaussian kernel function is expressed as follows: Step 402, for Normalization is performed: Step 403, construct the feature density cost function : Step 404: Generate the depth map D corresponding to image I using the Depth-Anything depth estimation model. I D I element D I (i, j) represents the depth value at pixel coordinates (i, j) in image I; obtain D I histogram F D The statistical frequency of depth value x is F. D (x), using the depth value C with the highest frequency as the baseline, construct the depth cost function E for the splicing line. d (i, j): Step 405, for E d (i, j) are normalized: Step 406: Construct a unified cost function E(i,j) based on the normalized feature density cost function and depth cost function: Where α1 and α2 represent weighting coefficients, and α1 + α2 = 1; Step 407: Solve using the dynamic programming algorithm, defining the cumulative cost function M(i, j) as follows: E(i, j) is a unified cost function. Neighborhood terms outside the image range are taken as infinity. The cumulative cost is calculated column by column according to this recursive relationship, and the backtracking path is recorded. The pixel with the smallest cumulative cost in the last column is taken as the endpoint, and backtracking is performed to obtain the pre-planned stitching line L. S .

4. The parallax image stitching method based on image depth and feature density guided according to claim 1, characterized in that, The specific steps for step 5 are as follows: Step 501, add the center stitching line L in image I. S L is composed of points whose minimum Euclidean distance is less than β pixels. S neighborhood space R L For feature matching point pairs (p) s , p' s ), find L S above and p s The nearest point p lmin , determine (p s , p' s Does it meet the following conditions: Condition 1: (p s , p' s In the set of feature matching point pairs P1; Condition 2: p s The coordinates fall on R L within the area; Condition 3: p s With p lmin The difference in depth values ​​at each location is less than the threshold δ, where δ ranges from 1 to 3. The feature matching point pairs that simultaneously meet the above conditions will form a set P of splicing support feature matching point pairs. S ; Step 502, construct based on P S The gridded affine projection field, denoted as Let the side length of the grid be 'a', and the number of grid cells in the vertical direction be 'a'. The number of grid cells in the horizontal direction is The grid in column x and row y is G. s (x, y), v s (x, y) is the grid G s The top-left vertex pixel coordinates of (x, y), h s (x, y) is the grid G s The affine transformation model of (x, y) is then The expression is as follows: 。 5. The parallax image stitching method based on image depth and feature density guided according to claim 1, characterized in that, The specific method for step 6 is as follows: For pixel coordinates q=(i, j) in the image I to be stitched, the affine transformation model h corresponding to point q is... q (i, j) is: Pixel-level dense projection field H I ={h q (i, j) | (i, j)∈I}.

6. The parallax image stitching method based on image depth and feature density guided according to claim 1, characterized in that, The specific method of linear weighted fusion in step 8 is as follows: In the formula, (i', j') are the pixels on the projection image I', and (i', j') > L. s This indicates that point (i', j') is at the splicing line L. s Above, (i', j')=L s This indicates that point (i', j') is at the splicing line L. s Above, (i', j') <L s This indicates that point (i', j') is at the splicing line L. s Below.

Images