An image stitching system and method based on registration fusion and seam line assistance

Abstract

The present invention discloses an image stitching system and method based on registration fusion and seam line assistance, comprising: an image overlapping area registration module, an image non-overlapping area registration module and a registration fusion module; the image overlapping area registration module comprises: a feature point pairing module, a preliminary registration module, a line segment pairing module, a precise registration module and a seam line calculation module; the image non-overlapping area registration module comprises: a line segment slope calculation module and a distortion correction module; the present invention can simultaneously handle image stitching problems with large parallax and large shooting angle difference, and based on the registration fusion and seam line algorithm, eliminates image registration errors and ghosting caused by parallax, and avoids distortion problems caused by shooting angle difference.

Description

Technical Field

[0001] The present invention belongs to the field of digital image processing, and in particular relates to an image stitching system and method based on registration fusion and seam line assistance. Background Art

[0002] Image stitching combines two or more images taken from different perspectives to create a wide-field image. The key to image stitching is image registration, which involves calculating the spatial alignment of different images. The pixels in the overlapping areas of adjacent images must be as identical as possible to ensure that the different images consistently depict the same object.

[0003] Traditional image stitching methods include two steps: feature matching and image registration. In the feature matching stage, feature points are first detected for each image, and then feature point pairs are calculated for each pair of adjacent images. In the image registration stage, the perspective transformation relationship between images is calculated based on the position of the feature point pairs, so that the pixels in the overlapping areas of the images after transformation are as similar as possible.

[0004] Traditional image stitching methods have the following drawbacks: When capturing different images, the camera must rotate only around the lens' optical center to prevent parallax. If parallax exists between images, the relative positions of objects in the overlapping areas of the images will differ. A single perspective transformation cannot fully align all objects, resulting in ghosting in the image registration results of traditional algorithms. Furthermore, if the angles of adjacent images differ significantly, non-overlapping areas of the images will be severely distorted during registration. Summary of the Invention

[0005] In view of the above-mentioned deficiencies in the prior art, the object of the present invention is to provide an image stitching system and method based on registration fusion and seam line assistance to solve the problems of parallax and distortion that traditional image stitching methods cannot handle.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] The present invention provides an image stitching system based on registration fusion and seam line assistance, comprising: an image overlapping area registration module, an image non-overlapping area registration module and a registration fusion module; wherein,

[0008] Image overlapping area registration module, including: feature point pairing module, preliminary registration module, line segment pairing module, precise registration module and seam line calculation module;

[0009] The feature point pairing module is used to detect feature points of images and pair the feature points between images to obtain feature point pairs;

[0010] The preliminary registration module uses the feature point pairs obtained by the feature point pairing module to calculate a grid transformation R that can globally align the overlapping areas of the images. O , and based on the grid transformation R O Calculate and obtain the preliminary registration results;

[0011] The line segment pairing module is used to detect line segments in an image and match the line segments of two images;

[0012] The precise registration module uses the line segment pairing results to optimize the grid transformation R O , get the new grid transformation R A , based on the grid transformation R A The precise registration result is calculated and can accurately align the line segments in the overlapping area;

[0013] The seam line calculation module finds a seam line in the image overlap area based on the precise registration result, and segments the image overlap area to eliminate ghosting problems caused by parallax;

[0014] Image non-overlapping area registration module, including: line segment slope calculation module and distortion correction module;

[0015] The line segment slope calculation module calculates a perspective transformation that minimizes distortion in non-overlapping areas of the image using feature point pairs, and determines the slope of each line segment in the image based on the perspective transformation;

[0016] The distortion correction module uses the feature point pairs and the slope of the line segment to calculate a grid transformation R that can minimize the distortion of the non-overlapping area of the image and maintain the line segment structure. S , and based on the grid transformation R S Calculate the distortion-corrected registration result;

[0017] The registration fusion module fuses the precise registration result and the distortion-corrected registration result to obtain a registration result with accurate alignment and distortion correction, and eliminates the ghosting caused by parallax in the registration result according to the seam line obtained by the seam line calculation module to obtain the final stitching result.

[0018] Furthermore, the specific process of the feature point pairing module includes:

[0019] (11) Receive two input images, one as the target image and the other as the reference image;

[0020] (12) Detect feature points of the input image based on the Scale-Invariant Feature Transform (SIFT) operator;

[0021] (13) Based on the scale-invariant feature transformation feature points, the feature point pairs between the two images are calculated;

[0022] (14) Based on the Random Sample Consensus (RANSAC) algorithm, multiple rounds of screening are performed on the feature point pairs to obtain a set of screened feature point pairs.

[0023] Furthermore, the specific process of the preliminary registration module includes:

[0024] (21) Assigning an initial weight to each pair of feature points in the feature point pair (interior point) set;

[0025] (22) Using a random sampling consistency algorithm, a subset of feature point pairs with the same depth of field and the largest sum of weights is selected from the feature point pair set;

[0026] (23) Using the subset of feature point pairs selected in step (22), the moving direct linear transform (MDLT) algorithm is used to calculate the grid transformation of the target image and obtain the intermediate registration result;

[0027] (24) Calculating the registration error and distortion error corresponding to the intermediate registration result in step (23); wherein the registration error is the average difference of pixels in the registration overlap area, reflecting the global alignment accuracy of the images, and the distortion error is the deviation of feature point pairs between the images, reflecting the degree of distortion in the registration result;

[0028] (25) updating the minimum value of the registration error and the distortion error according to the current registration error and the distortion error;

[0029] (26) By comparing the current value of the registration error and the distortion error with the minimum value, it is determined whether the intermediate registration result is the local optimal registration result;

[0030] (27) If the local optimal registration result is not updated in three consecutive rounds, the loop ends and the current local optimal registration result is used as the final result of the preliminary registration; otherwise, go to step (28);

[0031] (28) Reduce the weight of the feature point pair selected in step (22) and execute step (22).

[0032] Furthermore, the specific process of the line segment pairing module includes:

[0033] (31) Perform line segment detection on the preliminary registration results using a line segment detector (LSD);

[0034] (32) Pair the line segments between the two graphs.

[0035] Furthermore, the specific process of the precise registration module includes:

[0036] (41) Mapping the endpoints of each successfully matched line segment in the target image to the straight line of the paired line segment in the reference image to obtain a new set of point pairs;

[0037] (42) The new point pair set is merged with the feature point pair set used in calculating the preliminary registration, and the moving direct linear transform (MDLT) algorithm is used to recalculate the grid transformation R of the target image. A , accurately align the line segments between the target image and the reference image to obtain accurate registration results.

[0038] Furthermore, the specific process of the seam line calculation module includes:

[0039] (51) Create a mask matrix of the same size as the precise registration result and assign all elements an initial value of 0;

[0040] (52) Record the positions of all pixels that are not successfully paired with line segments, and set the elements of the corresponding positions in the mask matrix to a high penalty value;

[0041] (53) An undirected graph is established based on the mask matrix and the precise registration result. The edge weight between two adjacent nodes is the sum of the color difference between the two points and the element value of the mask matrix at the two node positions.

[0042] (54) The maximum flow algorithm is used to calculate the minimum cut from the undirected graph, and the minimum cut is the desired seam line.

[0043] Furthermore, the specific process of the line segment slope calculation module includes:

[0044] (61) Calculate the slope of each line segment of the target image in the precise registration result;

[0045] (62) Assigning an initial weight to each pair of feature points in the feature point pair (inlier point) set;

[0046] (63) Using the random sampling consensus algorithm, find a subset of feature point pairs with the same depth of field that has the largest sum of weights from the feature point pair set;

[0047] (64) Using the selected feature point pair subset, the perspective transformation of the target image is calculated by the Direct Linear Transform (DLT) algorithm, and the intermediate registration result is obtained;

[0048] (65) Calculate the registration error and distortion error corresponding to the intermediate registration result in step (64); wherein the registration error is the sum of the deviations between the slopes of all line segments of the target image in the intermediate registration result and the slopes of all line segments in the precise registration, reflecting the consistency of the perspective relationship between the target image and the reference image; the distortion error is the change in the internal angle of the target image relative to the right angle in the intermediate registration result. Generally, the change in the size of the internal angle of the image in the non-overlapping area will be greater, and the greater the angle change, the more serious the distortion of the perspective transformation;

[0049] (66) updating the minimum value of the registration error and the distortion error according to the registration error and the distortion error in step (65), and determining whether the intermediate registration result is a local optimal registration result;

[0050] (67) If the local optimal registration result is not updated in three consecutive rounds, the loop ends, and the current optimal registration result is used as the final result of the slope calculation, and the slope of the line segment in the result is used as the final slope of the line segment; otherwise, step (68) is executed;

[0051] (68) Reduce the weight of the feature point pairs selected during registration and execute step (63).

[0052] Furthermore, the distortion correction module calculates a grid transformation of the target image by minimizing an energy function to reduce the distortion of the corresponding registration; the energy function includes: an alignment term, a regularization term, and a slope term;

[0053] Alignment item, used to ensure the alignment accuracy of overlapping areas;

[0054] Regularization term, used to reduce distortion in non-overlapping areas;

[0055] The slope term is used to avoid curvature of the line segment.

[0056] Furthermore, the specific process of the registration and fusion module includes:

[0057] (71) Based on the seam lines obtained in the precise registration, the reference image is segmented and the side close to the target image is deleted;

[0058] (72) calculating the registration result of the target image, wherein the registration result is a fusion of the precise registration result and the registration result corresponding to the distortion correction;

[0059] (73) Based on the seam line, the side of the target image close to the reference image is deleted;

[0060] (74) The reference image is stitched with the target image to obtain an image stitching result.

[0061] The present invention also provides an image stitching method based on registration fusion and seam line assistance, the steps of which are as follows:

[0062] 1) Detect the feature points of the image and pair the feature points between the images to obtain feature point pairs;

[0063] 2) Using the feature point pairs, a grid transformation R that can globally align the overlapping areas of the images is calculated. O , and based on the grid transformation R O Calculate and obtain the preliminary registration results;

[0064] 3) Based on the preliminary registration, perform line segment detection and matching;

[0065] 4) According to the results of line segment matching, transform the grid R O Optimize to get the grid transformation R A , so that it can accurately align the line segments in the overlapping area and transform R based on the grid A Obtain accurate registration results;

[0066] 5) Based on the results of line segment matching, the seam lines are calculated from the precise registration results to eliminate ghosting caused by parallax;

[0067] 6) using the feature point pairs described in step 1), determining the final slope of each line segment in the target image;

[0068] 7) Based on the final slope of the line segment, a grid transformation R is calculated that minimizes the distortion of the non-overlapping area of the image and maintains the line segment structure. S , and based on the grid transformation R S Calculate the distortion-corrected registration result;

[0069] 8) Fusing the precise registration result and the distortion-corrected registration result to obtain a registration result with precise alignment and distortion correction, and using the seam lines described in step 5) to eliminate ghosting in the registration to complete image stitching.

[0070] Beneficial effects of the present invention:

[0071] 1. The present invention uses a combination of feature point pairs and line segment pairs to register images, which can avoid the problem of feature loss caused by parallax and ensure the accuracy of registration in overlapping image areas;

[0072] 2. The present invention adopts a line segment-based seam line algorithm, which can eliminate the ghosting problem caused by parallax in image registration and avoid the problem of object contours being cut off on both sides of the seam line;

[0073] 3. The present invention adopts a distortion correction method based on line segment slope control, which can effectively reduce the distortion in non-overlapping areas and avoid the bending of line segments;

[0074] 4. The present invention calculates registration for overlapping areas and non-overlapping areas separately and fuses the registrations, which can improve the flexibility of image registration and enable the registration results to achieve parallax tolerance and distortion correction at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 This is the schematic diagram of the module design of the present invention.

[0076] Figure 2 Schematic diagram of the principle of feature point pairing.

[0077] Figure 3 Schematic diagram of the principle of preliminary alignment.

[0078] Figure 4 Schematic diagram of the principle of line segment pairing.

[0079] Figure 5 Schematic diagram of the principle of calculating the slope of a line segment. DETAILED DESCRIPTION

[0080] In order to facilitate understanding by those skilled in the art, the present invention will be further described below with reference to embodiments and drawings. The contents mentioned in the embodiments are not intended to limit the present invention.

[0081] The present invention reduces the registration error caused by parallax and the distortion of the registration result by calculating the local optimal solution for the overlapping area and the non-overlapping area of the image respectively.

[0082] Reference Figure 1 As shown, the image stitching system based on registration fusion and seam line assistance of the present invention includes: an image overlapping area registration module, an image non-overlapping area registration module and a registration fusion module; wherein,

[0083] Image overlapping area registration module, including: feature point pairing module, preliminary registration module, line segment pairing module, precise registration module and seam line calculation module;

[0084] The feature point pairing module is used to detect feature points of images and pair the feature points between images to obtain feature point pairs;

[0085] The preliminary registration module uses the feature point pairs obtained by the feature point pairing module to calculate a grid transformation R that can globally align the overlapping areas of the images. O , and based on the grid transformation R O Calculate and obtain the preliminary registration results;

[0086] The line segment pairing module is used to detect line segments in an image and match the line segments of two images;

[0087] The precise registration module uses the line segment pairing results to optimize the grid transformation R O , get the new grid transformation R A , based on the grid transformation R A The precise registration result is calculated and can accurately align the line segments in the overlapping area;

[0088] The seam line calculation module finds a seam line in the image overlap area based on the precise registration result, and segments the image overlap area to eliminate ghosting problems caused by parallax;

[0089] Image non-overlapping area registration module, including: line segment slope calculation module and distortion correction module;

[0090] The line segment slope calculation module calculates a perspective transformation that minimizes distortion in non-overlapping areas of the image using feature point pairs, and determines the slope of each line segment in the image based on the perspective transformation;

[0091] The distortion correction module uses the feature point pairs and the slope of the line segment to calculate a grid transformation R that can minimize the distortion of the non-overlapping area of the image and maintain the line segment structure. S , and based on the grid transformation R S Calculate the distortion-corrected registration result;

[0092] The registration fusion module fuses the precise registration result and the distortion-corrected registration result to obtain a registration result with accurate alignment and distortion correction, and eliminates the ghosting caused by parallax in the registration result according to the seam line obtained by the seam line calculation module to obtain the final stitching result.

[0093] Reference Figure 2 As shown, the specific process of the feature point pairing module includes:

[0094] (11) Receive two input images I1 and I2, use I1 as the target image and I2 as the reference image;

[0095] (12) Detect feature points of images I1 and I2 based on the Scale-Invariant Feature Transform (SIFT) operator;

[0096] (13) Based on the scale-invariant feature transformation feature points, the feature point pairs between the target image I1 and the reference image I2 are calculated;

[0097] (14) Based on the Random Sample Consensus (RANSAC) algorithm, multiple rounds of screening are performed on the feature point pairs. The specific process is as follows:

[0098] (141) Let F be the set of matched SIFT feature point pairs, and initialize the screening result F filter =φ;

[0099] (142) Using the random sampling consensus algorithm, find the inner point set F with the largest number of feature point pairs from the feature point pair set F. inlier ;

[0100] (143) If F inlier If the number of feature point pairs contained in is greater than or equal to the threshold φ=5, then let F=FF inlier , F filter =F filter ∪F inlier , and return to step (142); otherwise, end the screening, set F filter is the final screening result of feature point pairs.

[0101] Reference Figure 3 As shown, the preliminary registration module calculates the local optimal registration result based on a random algorithm, and the specific process includes:

[0102] (21) For the set of feature point pairs (interior points) F filter Each pair of feature points is given an initial weight of 1, and the minimum registration error P is initialized. min =∞, minimum distortion error D min =∞;

[0103] (22) Using the random sampling consensus algorithm, we can select the feature point pair F from the feature point pair set F. filter Find the subset of feature point pairs with the same depth of field that has the largest sum of weights W inlier ;

[0104] (23) Using the feature point subset W selected in step (22) inlier , calculate the grid transformation of the target image I1 by using the Moving Direct Linear Transform (MDLT) algorithm, and obtain the intermediate registration result T of the target image I1 and the reference image I2;

[0105] (24) Calculate the registration error P corresponding to the intermediate registration result T in step (23) O and distortion error D O ,in,

[0106] Registration error P O : is the average color distance of pixels in the overlapping area between the target image I1 and the reference image I2 in the intermediate registration result T, reflecting the global alignment accuracy of the image. The formula is expressed as:

[0107]

[0108] Where N is the number of pixels in the overlapping area, I1(p) and I2(p) are the RGB color values of the target image I1 and the reference image I2(p) at pixel p, respectively;

[0109] Distortion error D O : is the deviation between the feature point pairs of the target image I1 and the reference image I2 in the intermediate registration result T, reflecting the degree of distortion in the registration result. The specific calculation steps are:

[0110] (241) Let the set V be the offset of each pair of feature points between the target image I1 and the reference image I2, and the formula is expressed as:

[0111]

[0112] in, is a pair of feature points in the feature point pair set F, Located in I1, Located at I2, The position of the feature point p is represented by a two-dimensional vector;

[0113] (242) Using the random sampling consensus algorithm, select the angle difference less than 1 from the set V. The largest vector set V whose vector modulus difference is less than 1 pixel inlier ;

[0114] (243) Delete subset V from V inlier If the number of random sampling consistency screening is less than the threshold t=3, then return to step (242); otherwise, end the calculation, and the distortion error between the final target image I1 and the reference image I2 is equal to the number of vectors remaining in the set V;

[0115] (25) According to the current distortion error D O and the registration error P O , update the distortion error and the minimum value of the distortion error D min and P min ;

[0116] (26) If αD is satisfied O <D min or (D O <αD min )∧(P O <P min ), where α = 1.2, the intermediate registration result T is regarded as the local optimal registration result T O ;

[0117] (27) If the local optimal registration result is not updated for three consecutive rounds, the loop ends and the current local optimal registration result T Oas the final result of the preliminary registration; otherwise, go to step (28);

[0118] (28) The feature points selected in step (22) are inlier The corresponding weight is divided by Among them, |W inlier | for W inlier The number of feature point pairs, |F filter | is the number of feature point pairs screened in the feature point pairing module, and then returns to step (242).

[0119] Reference Figure 4 As shown, the specific process of the line segment pairing module includes:

[0120] (31) Let I′1 and I′2 be the target image I1 and the reference image I2 in the preliminary registration result T O After the deformation, the line segment detector (LSD) is used to detect the line segments of I′1 and I′2;

[0121] (32) Pair the line segments on the target image I1 with the line segments on the reference image I2. For each line segment on the target image I′1, use the following algorithm to find the line segment that pairs with it in the line segment set of the reference image I′2:

[0122] (321) Select a line segment l on the target image I′1 and let L be the set of line segments on the reference image I′2;

[0123] (322) Delete from L all the slopes whose difference from l is greater than the threshold If L is empty, there is no line segment paired with l;

[0124] (323) Delete from L all line segments whose distance to the midpoint of l is greater than a threshold d = 10 pixels. If L is empty, there is no line segment paired with l;

[0125] (324) For each line segment in L, calculate its distance to the line band descriptor (LBD) of line segment l, and select the line segment with the smallest distance to match with l.

[0126] The specific process of the precise registration module includes:

[0127] (41) The endpoints of each successfully matched line segment l1 in the target image I1 are mapped to the line of the paired line segment in the reference image I2. For each endpoint on l1, its original endpoint position and the mapped endpoint position form a new point pair. When the endpoints of all successfully matched line segments are mapped, a new point pair set F is obtained.ed , the specific steps are as follows:

[0128] (411) Record the endpoint position of each successfully matched line segment l1 in I1 to obtain the endpoint set E;

[0129] (412) In the preliminary registration result T O In the equation, for each successfully matched line segment l1 on I′1 and its matched line segment l2 on I′2, calculate the projection of the endpoint of line segment l1 on the line where line segment l2 is located, and record the position of the perpendicular foot on the line to obtain the perpendicular foot set P;

[0130] (413) Pair the endpoints in set E with the points in P to obtain the point pair set F ed ;

[0131] (42) Calculate the initial registration result T O The feature point pair set used is F o , the new point pair set F ed With F o Merge and recalculate the grid transformation R of I1 using the moving direct linear transformation algorithm A , get the accurate registration result T of target image I1 and reference image I2 A .

[0132] The specific process of the seam line calculation module includes:

[0133] (51) Create a registration result T A Mask Matrix M A , assign all elements an initial value of 0;

[0134] (52) Let I′1 and I′2 be the exact registration of I1 and I2 in T A After the deformation, for each unpaired line segment on I′1 and I′2, record the positions of all its pixels and convert M A The element at the corresponding position above is assigned a penalty value of ψ = 1000000;

[0135] (53) According to T A and M A Create an undirected graph G as follows:

[0136] (531) Add a source node s and a sink node t to the undirected graph G;

[0137] (532) For the registration result T A For each pixel p in , add node v(p) to the undirected graph G;

[0138] (533) For adjacent pixels p and q, add an edge e(p,q) to the undirected graph G. If p and q are located in the overlapping region of I′1 and I′2, the edge weight w(p,q) is calculated as follows:

[0139] w(p,q)=|I1(p)-I2(p)|+|I q (q)-I2(q)|+M A (p)+M A (q)

[0140] Among them, I1(p) and I2(p) represent the RGB color values of I1 and I2 at p, respectively, and M A (p) represents the mask matrix M A The element value at p; otherwise, the edge weight w(p,q)=1000000;

[0141] (534) For a pixel p in the non-overlapping region of I′1 or I′2, if p∈I′1, add an edge e(s, p) with an edge weight of w(s, p)=1000000 to the undirected graph G; if p∈I′2, add an edge e(p, t) with an edge weight of w(p, t)=1000000 to the undirected graph G;

[0142] (54) Use the maximum flow algorithm to calculate the minimum cut from the undirected graph G. The minimum cut is the required seam line.

[0143] Reference Figure 5 As shown, the line segment slope calculation module also calculates the local optimal solution based on a random algorithm, and its specific process includes:

[0144] (61) Calculate the target image I1 in the precise registration result T A , the slope of each line segment;

[0145] (62) For the set of feature point pairs (interior points) F filter Each pair of feature points is given an initial weight of 1, and the minimum registration error P is initialized. min =∞, minimum distortion error D min =∞;

[0146] (63) Using the random sampling consensus algorithm, we can select the feature point pair F from the feature point pair set F. filter Find the subset of feature point pairs with the same depth of field that has the largest sum of weights W inlier ;

[0147] (64) Using the selected feature point pair subset, the perspective transformation of the target image I1 is calculated by the direct linear transform (DLT) algorithm, and the registration result T of the target image I1 and the reference image I2 is obtained;

[0148] (65) Calculate the registration error P corresponding to the intermediate registration result T in step (64) H and distortion error D H ,in,

[0149] Registration error P H : is the slope of the target image I1 in the intermediate registration result T and the precise registration T A The sum of the slope deviations of all line segments in reflects the consistency of the perspective relationship between the target image I1 and the reference image I2. The formula is expressed as:

[0150]

[0151] in, is the vector corresponding to the line segment (p,q) in the intermediate registration result T, is the registration result T A In , the vector corresponding to the line segment (p,q);

[0152] Distortion error D H : is the change in the internal angle of the target image I1 relative to the right angle in the intermediate registration result T. Usually, the change in the internal angle of the image in the non-overlapping area is greater, and the greater the angle change, the more serious the distortion of the perspective transformation corresponding to T. The formula is expressed as:

[0153]

[0154] Among them, θ i is the inner angle of the target image I1 in the intermediate registration result T;

[0155] (66) According to the registration error P in step (65) H and distortion error D H , update the minimum registration error P min and minimum distortion error D min , if βP is satisfied H <P min Or (P H <βP min )∧(D H <D min ), where β = 1.2, then the intermediate registration result T is regarded as the local optimal registration result T H ;

[0156] (67) If the local optimal registration result is not updated for three consecutive rounds, the loop ends and the current optimal registration result T H as the final result of the slope calculation, and use the slope of the line segment in the result as the final slope of the line segment; otherwise, execute step (68);

[0157] (68)W inlier The weight of the corresponding feature point is divided by Where |W inlier | for W inlier The number of feature point pairs in |F filter | is the F obtained by screening in step (14) filter The number of feature point pairs in , and then return to step (63).

[0158] The specific process of the distortion correction module includes:

[0159] (71) Define the energy function Its expression is as follows:

[0160]

[0161] Among them, E a is the alignment item, which is used to ensure the alignment accuracy of the overlapping area; E r is a regularization term used to reduce the distortion of non-overlapping areas; E s is the slope term, used to avoid bending of the line segment;

[0162] (72) The definition of alignment term is:

[0163]

[0164] in, is a variable, is the grid point position of the target image I1 we need to find, V is the target image I1 in the registration result T H The location of the grid points in ;

[0165] (73)The regularization term is defined as:

[0166]

[0167] Among them, C and R are the number of columns and rows of the grid matrix of the target image I1, respectively. is a variable, and is the grid point position of the target image I1 we need to find. is the grid point position at the i-th column and j-th row;

[0168] (74) The slope term is defined as:

[0169]

[0170] Where L is the set of line segments of the target image I1, φ(p) linearly represents the position of p with the four vertices corresponding to the grid where p is located, and (p′, q′) is the line segment (p, q) in the registration result T H The position in (p′, q′) ⊥is a vector perpendicular to the vector corresponding to (p′, q′);

[0171] (75) By minimizing the energy function Get the target position of each grid point of the target image I1, and calculate the registration result T of the target image I1 and the reference image I2 based on the grid transformation M .

[0172] The specific process of the registration and fusion module includes:

[0173] (81) Based on the seam line S obtained in the precise registration, the reference image I2 is segmented and all pixels close to the target image I1 are deleted;

[0174] (82) Calculate the registration result of the target image I1, which is a fusion of the precise registration result and the registration result corresponding to the distortion correction: Let W A is the registration result T A The registration function of I1, W M is the registration result T M The registration function of I1 in the image is used to calculate the registration function W of the target image I1 based on the position of the seam line S. C , function W C The expression at position p is given by W at that position A and W M The fusion formula is expressed as:

[0175] W C (p) = (1-(1 / 2) d )W A (p)+(1 / 2) d W M (p)

[0176] Among them, W A (p) and W M (p) are W A and W M The value at pixel p, d is the Euclidean distance between pixel p and the seam line S;

[0177] (83) Based on the seam line S, all pixels close to the reference image I2 in the result of I1 registration fusion are deleted;

[0178] (84) Splice I′1 and I2 to obtain the image stitching result.

[0179] The present invention also provides an image stitching method based on registration fusion and seam line assistance, the steps of which are as follows:

[0180] 1) Detect the feature points of the image and pair the feature points between the images to obtain feature point pairs;

[0181] 2) Using the feature point pairs, a grid transformation R that can globally align the overlapping areas of the images is calculated. O , and based on the grid transformation R O Calculate and obtain the preliminary registration results;

[0182] 3) Based on the preliminary registration, perform line segment detection and matching;

[0183] 4) According to the results of line segment matching, transform the grid R O Optimize to get the grid transformation R A , so that it can accurately align the line segments in the overlapping area and transform R based on the grid A Obtain accurate registration results;

[0184] 5) Based on the results of line segment matching, the seam lines are calculated from the precise registration results to eliminate ghosting caused by parallax;

[0185] 6) using the feature point pairs described in step 1), determining the final slope of each line segment in the target image;

[0186] 7) Based on the final slope of the line segment, a grid transformation R is calculated that minimizes the distortion of the non-overlapping area of the image and maintains the line segment structure. S , and based on the grid transformation R S Calculate the distortion-corrected registration result;

[0187] 8) Fusing the precise registration result and the distortion-corrected registration result to obtain a registration result with precise alignment and distortion correction, and using the seam lines described in step 5) to eliminate ghosting in the registration to complete image stitching.

[0188] The present invention has many specific application paths. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements can be made without departing from the principles of the present invention. These improvements should also be considered as the scope of protection of the present invention.

Claims

1. An image stitching system based on registration fusion and seam line assistance, characterized in that: include: Image overlapping area registration module, image non-overlapping area registration module and registration fusion module; Image overlapping area registration module, including: feature point pairing module, preliminary registration module, line segment pairing module, precise registration module and seam line calculation module; The feature point pairing module is used to detect feature points of images and pair the feature points between images to obtain feature point pairs; The preliminary registration module uses the feature point pairs obtained by the feature point pairing module to calculate a grid transformation R that can globally align the overlapping areas of the images. O , and based on the grid transformation R O Calculate and obtain the preliminary registration results; The line segment pairing module is used to detect line segments in an image and match the line segments of two images; The precise registration module uses the line segment pairing results to optimize the grid transformation R O , get the new grid transformation R A , based on the grid transformation R A The precise registration result is calculated and can accurately align the line segments in the overlapping area; The seam line calculation module finds a seam line in the image overlap area based on the precise registration result, and segments the image overlap area to eliminate ghosting problems caused by parallax; Image non-overlapping area registration module, including: line segment slope calculation module and distortion correction module; The line segment slope calculation module calculates a perspective transformation that minimizes distortion in non-overlapping areas of the image using feature point pairs, and determines the slope of each line segment in the image based on the perspective transformation; The distortion correction module uses the feature point pairs and the slope of the line segment to calculate a grid transformation R that can minimize the distortion of the non-overlapping area of the image and maintain the line segment structure. S , and based on the grid transformation R S Calculate the distortion-corrected registration result; The registration fusion module fuses the precise registration result and the distortion-corrected registration result to obtain a registration result with accurate alignment and distortion correction, and eliminates the ghosting caused by parallax in the registration result according to the seam line obtained by the seam line calculation module to obtain the final stitching result.

2. The image stitching system based on registration fusion and seam line assistance according to claim 1, characterized in that: The specific process of the feature point pairing module includes: (11) Receive two input images, one as the target image and the other as the reference image; (12) Detecting feature points of the input image based on a scale-invariant feature transformation operator; (13) Based on the scale-invariant feature transformation feature points, the feature point pairs between the two images are calculated; (14) Based on the random sampling consistency algorithm, the feature point pairs are screened for multiple rounds to obtain a set of screened feature point pairs.

3. The image stitching system based on registration fusion and seam line assistance according to claim 2, characterized in that: The specific process of the preliminary registration module includes: (21) Assigning an initial weight to each pair of feature points in the feature point pair set; (22) Using a random sampling consistency algorithm, a subset of feature point pairs with the same depth of field and the largest sum of weights is selected from the feature point pair set; (23) using the subset of feature point pairs selected in step (22), a moving direct linear transformation algorithm is used to calculate the grid transformation of the target image and obtain an intermediate registration result; (24) Calculating the registration error and distortion error corresponding to the intermediate registration result in step (23); wherein the registration error is the average difference of pixels in the registration overlap area, reflecting the global alignment accuracy of the images, and the distortion error is the deviation of feature point pairs between the images, reflecting the degree of distortion in the registration result; (25) updating the minimum value of the registration error and the distortion error according to the current registration error and the distortion error; (26) By comparing the current value of the registration error and the distortion error with the minimum value, it is determined whether the intermediate registration result is the local optimal registration result; (27) If the local optimal registration result is not updated in three consecutive rounds, the loop ends and the current local optimal registration result is used as the final result of the preliminary registration; otherwise, go to step (28); (28) Reduce the weight of the feature point pair selected in step (22) and execute step (22).

4. The image stitching system based on registration fusion and seam line assistance according to claim 3, characterized in that: The specific process of the line segment pairing module includes: (31) Performing line segment detection on the preliminary registration results using a line segment detector; (32) Pair the line segments between the two graphs.

5. The image stitching system based on registration fusion and seam line assistance according to claim 4 is characterized in that: The specific process of the precise registration module includes: (41) Mapping the endpoints of each successfully matched line segment in the target image to the straight line of the paired line segment in the reference image to obtain a new set of point pairs; (42) The new point pair set is merged with the feature point pair set used in calculating the preliminary registration, and the moving direct linear transformation algorithm is used to recalculate the grid transformation R of the target image. A , accurately align the line segments between the target image and the reference image to obtain accurate registration results.

6. The image stitching system based on registration fusion and seam line assistance according to claim 5, characterized in that: The specific process of the seam line calculation module includes: (51) Create a mask matrix of the same size as the precise registration result and assign all elements an initial value of 0; (52) Record the positions of all pixels that are not successfully paired with line segments, and set the elements of the corresponding positions in the mask matrix to a high penalty value; (53) An undirected graph is established based on the mask matrix and the precise registration result. The edge weight between two adjacent nodes is the sum of the color difference between the two points and the element value of the mask matrix at the two node positions. (54) The maximum flow algorithm is used to calculate the minimum cut from the undirected graph, and the minimum cut is the desired seam line.

7. The image stitching system based on registration fusion and seam line assistance according to claim 6, characterized in that: The specific process of the line segment slope calculation module includes: (61) Calculate the slope of each line segment of the target image in the precise registration result; (62) Assigning an initial weight to each pair of feature points in the feature point pair set; (63) Using the random sampling consensus algorithm, find a subset of feature point pairs with the same depth of field that has the largest sum of weights from the feature point pair set; (64) Using the selected feature point pair subset, the perspective transformation of the target image is calculated by a direct linear transformation algorithm, and an intermediate registration result is obtained; (65) Calculating the registration error and distortion error corresponding to the intermediate registration result in step (64); wherein the registration error is the sum of the deviations between the slopes of all line segments of the target image in the intermediate registration result and the slopes of all line segments in the precise registration, reflecting the consistency of the perspective relationship between the target image and the reference image; the distortion error is the change in the internal angle of the target image relative to the right angle in the intermediate registration result; (66) updating the minimum value of the registration error and the distortion error according to the registration error and the distortion error in step (65), and determining whether the intermediate registration result is a local optimal registration result; (67) If the local optimal registration result is not updated in three consecutive rounds, the loop ends, and the current optimal registration result is used as the final result of the slope calculation, and the slope of the line segment in the result is used as the final slope of the line segment; otherwise, step (68) is executed; (68) Reduce the weight of the feature point pairs selected during registration and execute step (63).

8. The image stitching system based on registration fusion and seam line assistance according to claim 7, characterized in that: The distortion correction module calculates the grid transformation of the target image by minimizing an energy function to reduce the distortion of the corresponding registration; the energy function includes: an alignment term, a regularization term, and a slope term; Alignment item, used to ensure the alignment accuracy of overlapping areas; Regularization term, used to reduce distortion in non-overlapping areas; The slope term is used to avoid curvature of the line segment.

9. The image stitching system based on registration fusion and seam line assistance according to claim 8, characterized in that: The specific process of the registration and fusion module includes: (71) Based on the seam lines obtained in the precise registration, the reference image is segmented and the side close to the target image is deleted; (72) calculating the registration result of the target image, wherein the registration result is a fusion of the precise registration result and the registration result corresponding to the distortion correction; (73) Based on the seam line, the side of the target image close to the reference image is deleted; (74) The reference image is stitched with the target image to obtain an image stitching result.

10. An image stitching method based on registration fusion and seam line assistance, based on the system according to any one of claims 1 to 9, characterized in that: Here are the steps: 1) Detect the feature points of the image and pair the feature points between the images to obtain feature point pairs; 2) Using the feature point pairs, a grid transformation R that can globally align the overlapping areas of the images is calculated. O , and based on the grid transformation R O Calculate and obtain the preliminary registration results; 3) Based on the preliminary registration, perform line segment detection and matching; 4) According to the results of line segment matching, transform the grid R O Optimize to get the grid transformation R A , so that it can accurately align the line segments in the overlapping area and transform R based on the grid A Obtain accurate registration results; 5) Based on the results of line segment matching, the seam lines are calculated from the precise registration results to eliminate ghosting caused by parallax; 6) using the feature point pairs described in step 1), determining the final slope of each line segment in the target image; 7) Based on the final slope of the line segment, a grid transformation R is calculated that minimizes the distortion of the non-overlapping area of the image and maintains the line segment structure. S , and based on the grid transformation R S Calculate the distortion-corrected registration result; 8) Fusing the precise registration result and the distortion-corrected registration result to obtain a registration result with precise alignment and distortion correction, and using the seam lines described in step 5) to eliminate ghosting in the registration to complete image stitching.

Images