An image registration method for large parallax scenes

The calculation of image registration parameters through brute force matching and random sampling methods is solved, and the calculation complexity and distortion problems of existing grid-based local registration algorithms in large disparity scenarios are achieved, and low complexity, high efficiency and high accuracy are achieved. Image registration is suitable for GPU parallel processing, meeting the needs of real-time video stitching.

CN113792777BActive Publication Date: 2024-11-22THE 54TH RESEARCH INSTITUTE OF CHINA ELECTRONICS TECHNOLOGY GROUP CORPORATION
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111040814.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-09-07
Publication Date
2024-11-22
Estimated Expiration
2041-09-07

AI Technical Summary

Technical Problem

The existing grid-based local registration algorithm has high computational complexity, low real-time performance when facing large disparity scenarios, and is prone to local distortion, making it difficult to meet the needs of real-time video stitching.

Method used

A image registration method for large disparity scenes is adopted, and the key point matching pairing is calculated through the brute force matching algorithm, uniformly sparse the matching results, calculate the affine transformation parameters, and obtain multiple sets of syn-solved equations by using random sampling method to perform local weighting. and, reduce algorithm complexity and improve computational efficiency.

Benefits of technology

It realizes image registration with low complexity, high efficiency and high accuracy, which is suitable for parallel processing of GPUs. It can complete image registration with high real-time performance of 10FPS, and ensures the accuracy of affine transformation parameters and the continuity of parameter space. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113792777B_ABST
    Figure CN113792777B_ABST
Patent Text Reader

Abstract

The present invention discloses an image registration method for large parallax scenes, and relates to the field of panoramic stitching of video images. The present invention first derives the conditions for obtaining the exact solution of affine transformation parameters and the expression of parameter solution according to the definition of affine transformation; then, a random sampling method is used to obtain 20 groups of exact solutions of affine transformation parameters at the same key point, and the affine transformation parameters that meet both the exact solution conditions and the requirements of local optimization are obtained by weighted average calculation; finally, the random sampling method is used again to calculate the radiation transformation matrix at non-key points. The present invention can accurately calculate the affine transformation parameters at each key point, and at the same time, the affine transformation parameters at each point in the image can be continuous in the linear space. At the same time, the present invention can well meet the GPU parallel computing requirements and achieve a high real-time performance of 10FPS on the GPU.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the technical field of video image panoramic stitching, and in particular to an image registration method for large parallax scenes. Background Art

[0002] The traditional video and image stitching process includes: inputting two frames of images and extracting their key points respectively, matching key points, purifying key points through the RANSAC algorithm, calculating projection transformation parameters, image fusion, etc. The projection transformation operator calculated in the above process is a 3×3 matrix, which is the least squares solution of the global projection transformation. When there is a large parallax between the two frames of images, the global projection transformation operator often performs poorly and it is difficult to fuse the two images well.

[0003] In order to overcome the parallax problem and improve the stitching quality of video images, scholars in the field have proposed a grid-based local registration technology (As-Projective As-Possible, referred to as APAP) in recent years. It first divides the key point space into a 100×100 grid, and then uses the weighted singular value decomposition technology to obtain the projection transformation operator of each small grid. Grid-based local registration technology has made significant breakthroughs in solving the parallax problem. In recent years, a series of grid-based registration optimization algorithms such as AANAP and GSP have been derived on this basis. Grid-based local registration is a major advancement in the field of video image stitching, but the existing grid-based local registration algorithms have the problem of high computational complexity.

[0004] Grid local registration first divides the image to be registered into n×n grids, calculates the Euclidean distance from the key point to the center of each grid as the weight to estimate the weight of each key point involved in the calculation; then the least squares solution of the overdetermined equations is obtained through weighted singular value decomposition. The APAP-based grid projection method has the following problems:

[0005] (1) It is very complex and has low real-time performance, which makes it difficult to meet the needs of real-time video stitching;

[0006] (2) When the key points near the grid are too sparse, it will cause large local distortion. Summary of the invention

[0007] The purpose of the present invention is to avoid the shortcomings of the above-mentioned background methods and provide an image registration method for large parallax scenes. The present invention has the characteristics of low computational complexity, high precision and strong robustness, and is particularly suitable for parallel processing on a GPU.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is:

[0009] An image registration method for large parallax scenes comprises the following steps:

[0010] (1) Input two frames of images to be registered, denoted as I L and I R , extract their SURF key points respectively, and record the key point set as K L {p} and K R {p};

[0011] (2) Calculate K using the brute force matching algorithm L {p} and K R The key points in the {q} set are matched, and the matching result is recorded as DM{p,q,dl|p ∈ K L ,q ∈ K R}, where dl represents the feature distance between the key point pair {p,q};

[0012] (3) Uniformly thin out DM and filter out bad matching pairs by partition. The matching result after uniform thinning is recorded as DM P ;

[0013] (4) Calculate the affine transformation parameters of the key points. The affine transformation parameter matrix of the key points is {M(p)|p∈DM P};

[0014] (5) Calculate the affine transformation parameters of non-key points. The affine transformation parameter matrix of non-key points is:

[0015] (6) Image I L Projection to I R In the coordinate system, the registration result is recorded as I L' , then

[0016]

[0017] I L′ (x,y)=I L (i,j);

[0018] Complete image registration.

[0019] Furthermore, step (4) specifically includes the following steps:

[0020] (401) Traverse DM P , let the current key point be a(x1,y1), take the 100×100 neighborhood around it, extract all the points in the neighborhood that are different from a, and form a set П a {(x,y)};

[0021] (402) From П aRandomly select two points b(x2,y2) and c(x3,y3) from {(x,y)}, and record the matching points corresponding to points a, b, and c as aˊ(x1ˊ,y1ˊ), bˊ(x2ˊ,y2ˊ), and cˊ(x3ˊ,y3ˊ); determine whether the following conditions are met:

[0022] Condition 1: Satisfy

[0023] Condition 2: Satisfy l 12 ≠l 13 , k 12 ≠k 13 ;in,

[0024]

[0025] The value of dl is 15, and the values ​​of i and j are 1, 2, and 3;

[0026] If the two randomly selected points b and c satisfy both conditions 1 and 2, proceed to step (403), otherwise repeat step (402) to re-select;

[0027] (403) with I L A new coordinate system is established with the center point as the origin of the coordinate system, and a, b, c, aˊ, bˊ, cˊ are translated to the new coordinate system; the affine transformation matrix at points a, b, and c is:

[0028]

[0029] Where n is the serial number of the affine transformation matrix;

[0030] Calculated from the above three pairs of points:

[0031]

[0032] (404) Calculate the distance d from point b and point c to point a an :

[0033]

[0034] (405) Repeat steps (401) to (404) until 20 affine transformation matrices are calculated and recorded as M a1 、M a2 …M a20 , then the affine transformation matrix M(a) of point a is:

[0035] M(a)=c1M a1 +c2M a2 +…c 20 M a20

[0036]

[0037] (406) Repeat steps (401) to (405) until the affine transformation matrix at all points is calculated.

[0038] Furthermore, step (5) specifically includes the following steps:

[0039] (501) Traversal I L If e(x,y) is not a key point, then take the surrounding 200×200 neighborhood and extract all N key points in the neighborhood. The set of all key points is recorded as П e {(x i ,y j )}, the corresponding affine transformation matrices constitute the set Ψ{M(i,j)};

[0040] (502) Calculate the affine transformation matrix M(e) of point e:

[0041]

[0042]

[0043] Among them, d en is the point e and the key point (x i ,y j ) between the two.

[0044] Compared with the background technology, the present invention has the following advantages:

[0045] 1. Low complexity. The method of the present invention only involves simple algebraic calculations, and the algorithm complexity is low, which is particularly suitable for GPU parallel computing.

[0046] 2. High solution efficiency. The present invention obtains 20 groups of equations with the same solution by introducing a random sampling method, which greatly improves the calculation efficiency.

[0047] 3. Accurate solution. The affine transformation parameters at the key points calculated by the present invention are absolutely accurate. In step (404), the characteristics of the same solution equation are used to weight the sum of all M matrices related to point a. The result is still accurate for a, while the prior art seeks the least square solution, which is not an absolutely accurate solution.

[0048] 4. Good continuity in parameter space. The present invention adopts a method of local weighted summation of the same solution equation, and the M matrix of all points calculated is continuous in the parameter space. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1This is a schematic diagram of the principle of solving the affine transformation parameters of point a (indicated by a white circle). The points in the 100×100 neighborhood around point a are represented by gray circles, which constitute the set П{(x,y)}. Two points are randomly selected from П{(x,y)}, denoted as b and c. A set of radial transformation parameters can be obtained from the three points a, b, and c.

[0050] Figure 2 is the result of parallel calculation of all key points. The solution at each point not only meets the accuracy requirements in its local neighborhood, but also has a high degree of overlap between different points and ensures the continuity of the global parameter space.

[0051] Figure 3 It is the registration result of a set of large parallax scenes. Figure 3 The two images in the upper left and upper right of the middle image not only have a large left-right field of view, but also a large difference in resolution between near and far, which is a typical complex registration scene. The projection image is semi-transparent to facilitate viewing of internal registration details. The black ellipse marks the four areas where the projection effect is reflected.

[0052] Figure 4 yes Figure 3 The local enlarged images of the four corresponding marks in Fig. 1. It can be seen that the proposed algorithm achieves accurate registration in all four places. DETAILED DESCRIPTION

[0053] The present invention will be further described below in conjunction with the accompanying drawings.

[0054] Reference Figure 1 to Figure 4 , an image registration method for large parallax scenes, comprising the following steps:

[0055] (1) Input two frames of images to be registered, denoted as I L and I R , extract their SURF key points respectively, and record the key point set as K L {p} and K R {p};

[0056] (2) Calculate K using the brute force matching (BF) algorithm L {p} and K R The key points in the {q} set are matched, and the matching result is recorded as DM{p,q,dl|p ∈ K L ,q ∈ K R}, where dl represents the key point distance between the key point pair {p,q};

[0057] (3) Uniformly thin the DM and filter out bad matching pairs by partitioning;

[0058] 3.1) Will I LDivide into 10×10 grids, traverse all grids, and calculate the matching point pair closest to the grid center point in each grid. The grid center point is recorded as (x g ,y g ), matching point pair {p,q,dl} in p i The point coordinates are (x p ,y p ), the distance from the matching point pair to the grid center is recorded as d g , defined as follows:

[0059] d g =|x p -x g |+|y p -y g |

[0060] In each grid, only d g The key point with the smallest value, the set of key point matching pairs after uniform thinning is recorded as DM F ;

[0061] 3.2) DM F The matching point pairs in are rearranged in ascending order according to dl;

[0062] 3.3) Will I L With the center point C t (x t ,y t ) is divided into Ω1{x <x t ,y <y t},Ω2{x <x t ,y <y t},Ω3{x <x t ,y <y t},Ω4{x <x t ,y <y t} and other 4 areas;

[0063] 3.4) Use the RANSAC algorithm to analyze Ω i The key point matching pairs in (i=1,2,3,4) are sampled consistently to obtain the best matching pairs, and the best matching pairs in the four regions are merged and recorded as DM P , where each region only takes the first 500 matching pairs that fall within the region;

[0064] (4) Calculate the affine transformation parameters of the key points, such as Figure 1 shown.

[0065] 4.1) Traversing DM P, let the current point be a(x1,y1), take the 100×100 neighborhood around it, extract all the points in the neighborhood that are different from a, and form the set П{(x,y)};

[0066] 4.2) Randomly select two points b(x2,y2) and c(x3,y3) from П{(x,y)}. The matching points corresponding to points a, b, and c are recorded as aˊ(x1ˊ,y1ˊ), bˊ(x2ˊ,y2ˊ), and cˊ(x3ˊ,y3ˊ). Determine whether the following conditions are met:

[0067] Condition 1: Satisfy

[0068] Condition 2: Assumption

[0069] Satisfy 12 ≠l 13 , k 12 ≠k 13 ;

[0070] The recommended value of dl is 15. If the two randomly selected points b and c meet both conditions 1 and 2, proceed to the next step, otherwise re-select.

[0071] 4.3) Take I L The center point is the origin of the coordinate system, and a, b, c, aˊ, bˊ, cˊ are translated to the new coordinate system. The affine transformation matrix at points a, b, and c is:

[0072]

[0073] Among them, n is the number of the nth sampling. Through the above three pairs of points, we can calculate:

[0074]

[0075] Furthermore, the detailed process of 4.3) is as follows:

[0076] 4.3.1) Let the affine transformation be:

[0077]

[0078] Move a(x1,y1), b(x2,y2), c(x3,y3) and the corresponding projection coordinates a′(x′1,y′1), b′(x′2,y′2), c′(x′3,y′3) to the a coordinate system respectively, and then substitute them into the above formula to obtain the affine transformation parameter equation group

[0079]

[0080] 4.3.2) If y1≠y2 and y1≠y3 then

[0081]

[0082] Solved

[0083]

[0084] 4.3.3) If x1≠x2 and x1≠x3, then

[0085]

[0086] 4.3.4) Finally, we get

[0087]

[0088] 4.4) Calculate the distance from point b and point c to point a, using d an express.

[0089]

[0090] 4.5) Repeat 4.1) to 4.4) until 20 valid matching point pairs are obtained. 20 affine transformation matrices are calculated and recorded as M a1 、M a2 …M a20 , then the affine transformation matrix M of point a n The calculation method is as follows:

[0091] M n =c1M a1 +c2M a2 +…c 20 M a20

[0092]

[0093] Among them, c n Represents the Euclidean distance weights from points a, b, and c to point a.

[0094] 4.6) Repeat 4.1) to 4.4) until the affine transformation matrix of all key points is calculated. For each key point, 20 sets of valid matching point pairs need to be obtained, such as Figure 2 shown.

[0095] (5) Calculate the affine transformation parameters of non-key points;

[0096] 5.1) Traversal I L For all pixel points, if e(x,y) is not a key point, then take the 200×200 neighborhood around it and extract all the key points in the neighborhood. The set of all key points is recorded as Пe {(x,y)}, the corresponding affine transformation matrix constitutes the set Ψ{M(x,y)};

[0097] 5.2) Then the affine transformation matrix of e is:

[0098]

[0099]

[0100] (6) Image I L Projection to I R In the coordinate system, the registration result is recorded as I L' , then

[0101]

[0102] I L′ (x,y)=I L (i,j)

[0103] Finally, I L A set of radial transformation parameters is calculated at all matching points in the matrix. The characteristics of this matrix are: it not only meets the accuracy requirements, but also meets the local optimization conditions, and is continuous in the parameter space, such as Figure 3 , Figure 4 shown.

[0104] The present invention can accurately calculate the affine transformation parameters at each key point, and at the same time make the affine transformation parameters at each point in the image continuous in the linear space. At the same time, the present invention can well meet the GPU parallel computing requirements and achieve a high real-time performance of 10FPS on the GPU.

[0105] It should be noted that the above embodiments are merely examples of the spirit of the present invention. Those skilled in the art of the present invention may make various modifications or additions to the specific embodiments described, or replace them with similar methods, but they will not deviate from the spirit of the present invention or exceed the scope defined by the attached claims.

Claims

1. An image registration method for large parallax scenes, characterized in that: The following steps are involved: (1) Input two frames of images to be registered, denoted as I L and I R , extract their SURF key points respectively, and record the key point set as K L {p} and K R {p}; (2) Calculate K using the brute force matching algorithm L {p} and K R The key points in the {q} set are matched, and the matching result is recorded as DM{p,q,dl|p ∈ K L ,q ∈ K R }, where dl represents the feature distance between the key point pair {p,q}; (3) Uniformly thin out DM and filter out bad matching pairs by partition. The matching result after uniform thinning is recorded as DM P ; (4) Calculate the affine transformation parameters of the key points. The affine transformation parameter matrix of the key points is {M(p)|p∈DM P }; specifically includes the following steps: (401) Traverse DM P , let the current key point be a(x1,y1), take the 100×100 neighborhood around it, extract all the points in the neighborhood that are different from a, and form a set П a {(x,y)}; (402) From П a Randomly select two points b(x2,y2) and c(x3,y3) from {(x,y)}, and record the matching points corresponding to points a, b, and c as a′(x1′,y1′), b′(x2′,y2′), and c′(x3′,y3′); determine whether the following conditions are met: Condition 1: Satisfy Condition 2: Satisfy l 12 1 l 13 , k 12 1k 13 ;in, The value of dl is 15, and the values ​​of i and j are 1, 2, and 3; If the two randomly selected points b and c satisfy both conditions 1 and 2, proceed to step (403), otherwise repeat step (402) to re-select; (403) with I L A new coordinate system is established with the center point as the origin of the coordinate system, and a, b, c, a′, b′, c′ are translated to the new coordinate system; the affine transformation matrix at points a, b, and c is: Where n is the serial number of the affine transformation matrix; Calculated from the above three pairs of points: (404) Calculate the distance d from point b and point c to point a an : (405) Repeat steps (401) to (404) until 20 affine transformation matrices are calculated and recorded as M a1 、M a2 …M a20 , then the affine transformation matrix M(a) of point a is: M(a)=c1M a1 +c2M a2 +…c 20 M a20 (406) Repeat steps (401) to (405) until the affine transformation matrix at all points is calculated; (5) Calculate the affine transformation parameters of non-key points. The affine transformation parameter matrix of non-key points is: The specific steps include: (501) Traversal I L If e(x,y) is not a key point, then take the surrounding 200×200 neighborhood and extract all N key points in the neighborhood. The set of all key points is recorded as П e {(x i ,y j )}, the corresponding affine transformation matrices constitute the set Ψ{M(i,j)}; (502) Calculate the affine transformation matrix M(e) of point e: Among them, d en is the point e and the key point (x i ,y j ) between the two sides; (6) Image I L Projection to I R In the coordinate system, the registration result is recorded as I L' , then I L¢ (x,y)=I L (i,j); Complete image registration.