A method for quickly determining matching epipolar pairs and calculating camera intrinsic matrices from image sequences or videos

Abstract

The present invention discloses a method for quickly determining matching epipolar line pairs from an image sequence or video and calculating a camera intrinsic matrix, which is used to achieve fast and accurate camera calibration and belongs to the field of computer vision research. The method described in the present invention mainly includes: first, using a background difference algorithm to extract a mask sequence or mask video of a moving object from a set of content-related image sequences or videos; second, determining a set of candidate corresponding epipolar lines of viewpoint (camera) A and viewpoint (camera) B; determining corresponding epipolar line pairs of viewpoint (camera) A and viewpoint (camera) B; and determining viewpoint-dependent poles from corresponding epipolar lines. Finally, the camera intrinsic matrix is ​​calculated from the poles and epipolar lines. The present invention can achieve fast and accurate camera calibration with low computational time complexity.

Description

Technical Field

[0001] The present invention proposes a new method for quickly determining matching epipolar line pairs and calculating camera intrinsic matrices from a set of content-related image sequences or videos, which belongs to the field of computer vision research. Background Art

[0002] Recovering camera parameters from a set of content-related image sequences or videos is an important problem in vision. The traditional 4-corresponding point solution is susceptible to insufficient or over-concentrated corresponding point pairs when the viewpoint changes greatly, resulting in failure of camera parameter recovery. Therefore, it is of great significance to robustly recover the required corresponding point pairs from a set of content-related image sequences or videos to obtain camera parameters.

[0003] Epipolar geometry is based on the theory of recovering camera parameters from a set of content-related image sequences or videos. The present invention provides a method for quickly determining corresponding point pairs from a set of content-related image sequences or videos by calculating epipolar geometry. The camera intrinsic matrix is ​​quickly determined from a set of content-related image sequences or videos by using the following knowledge: for a pixel point p in camera (viewpoint) A, all the pixel points corresponding to p in camera (viewpoint) B are located on the same epipolar line; similarly, the straight line passing through the center of camera (viewpoint) A and p on camera (viewpoint) B is located on an epipolar line; therefore, when camera A and camera B are synchronized, the two objects are at t of camera A. 1 and t 2 The instantaneous points projected onto the same pixel p at each moment lie on an epipolar line of camera B.

[0004] At present, calculating the camera intrinsic matrix is ​​the primary problem to be solved in many computer vision applications. The following briefly introduces the related issues of the camera intrinsic matrix.

[0005] 1. The camera intrinsic matrix is ​​a 3×3 matrix with rank 2 and 8 unknowns. The traditional solution to the camera intrinsic matrix uses an eight-point (4 pairs of corresponding points) algorithm. However, when the shooting angles between cameras change dramatically, the viewpoint changes greatly, and the corresponding point pairs are difficult to detect or sometimes even do not exist, which may lead to an unsolvable camera intrinsic matrix. It is error-prone to find corresponding point pairs from a large viewpoint image sequence or video, and even dense sampling may not guarantee that enough point pairs can be obtained to calculate the intrinsic matrix. The present invention can quickly obtain enough point pairs from a set of content-related image sequences or videos to calculate the camera intrinsic matrix.

[0006] 2. Using epipolar pairs instead of point pairs to calculate the camera intrinsic matrix has the following advantages: The exponential in the execution time of the random sampling consensus algorithm RANSAC depends on the minimum sampling set size required for the calculation model (algorithm). The epipolar geometry calculation of the camera intrinsic matrix requires at least 3 pairs of corresponding epipolar lines, and the corresponding point pairs require at least 4 pairs to calculate the camera intrinsic matrix. In actual calculations, a set of ≥3 pairs of corresponding epipolar lines or ≥4 pairs of corresponding point pairs will be used to calculate the camera intrinsic matrix.

[0007] For finding epipolar lines, the present invention accelerates the camera intrinsic matrix calculation by using pixels that record multiple depths to significantly reduce the search space for epipolar line matching; in addition, by using the center of mass of the detected object as a proxy for the position of the moving object, the accuracy can be further improved when performing global optimization. Summary of the invention

[0008] The purpose of the present invention is to provide a method for quickly determining matching epipolar line pairs and calculating the camera intrinsic matrix from a set of content-related image sequences or videos, in order to solve the problem that the logarithmic instability of feature points in a large viewing angle image sequence or video leads to easy errors in calculating the camera intrinsic matrix.

[0009] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0010] In one aspect, the present invention provides a method for quickly determining matching epipolar line pairs from an image sequence or video and calculating a camera intrinsic matrix, comprising the following steps:

[0011] Step 1: Use background subtraction algorithm to extract the mask sequence or mask video of the moving object from a set of content-related image sequences or videos;

[0012] Step 2: Determine the candidate corresponding epipolar line set of viewpoint A and viewpoint B;

[0013] Step 3: Determine the corresponding epipolar line pairs of viewpoint A and viewpoint B;

[0014] Step 4: The two epipolar lines of viewpoint A intersect to determine the pole e A , the two epipolar lines of viewpoint B intersect to determine the pole e B ;

[0015] Step 5: Calculate the camera intrinsic matrix from the poles and epipolar lines.

[0016] Furthermore, the specific steps of step 2 are as follows:

[0017] Step 2.1: Take any image from the image sequence or video and Two frames, in and Search for any corresponding point p in the two frames A , p ASet as the pixel point in viewpoint A;

[0018] Step 2.2: All pixels in the frame form a pixel set All pixels in the frame form a pixel set From and Take one point each to form an epiline Λ B Considered as the epipolar line in point B;

[0019] Step 2.3: Get t k (t k ≠t i ,t k ≠t j ) time frame, search for all satisfying conditions get p A and Λ A Any point Get with l B The corresponding epipolar

[0020] Furthermore, the specific steps of step 3 are as follows:

[0021] Step 3.1: If there are two pairs of corresponding epipolar lines of viewpoint A and viewpoint B, go to step 4, otherwise proceed to the next step;

[0022] Step 3.2: Use the random sampling consensus algorithm RANSAC from Select 20 epilines l B , using the random sampling consensus algorithm from Select 20 epilines l A ; Calculate l A and l B Movement coding;

[0023] Step 3.2: Calculate l A and l B Normalized Correlation Select the l with the highest normalized correlation A and l B To match the epipolar lines;

[0024] Step 3.3: Go to step 3.1 and determine a set of corresponding matching epipolar lines for a pair of viewpoints A and B.

[0025] Furthermore, in step 5, the camera intrinsic matrix is ​​assumed to be F,l B =F[e A ] × l A With lA =F T [e B ] × l B Established; when the scene contains planar objects, l B =Hl A , where H is the homography matrix; H is calculated from multiple sets of epipolar lines, and then the camera intrinsic matrix F is calculated = [e A ] × H.

[0026] On the other hand, the present invention also provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the above steps are implemented when the program or instruction is executed by the processor.

[0027] On the other hand, the present invention further provides a readable storage medium, on which a program or instruction is stored, and the above steps are implemented when the program or instruction is executed by a processor.

[0028] The beneficial effects of the present invention are: the present invention can realize fast and accurate camera calibration; it can realize fast and accurate camera calibration with low calculation time complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flow chart of the present invention.

[0030] Figure 2 This is an example diagram of the motion barcode described in the present invention.

[0031] FIG. 3( a ) is an example diagram of the present invention in which corresponding pixels are located on the determined epipolar lines.

[0032] FIG. 3( b ) is an example diagram of searching for matching epipolar lines at a viewpoint for candidate epipolar lines of the present invention.

[0033] Figure 4 It is the determined pole diagram of the present invention. DETAILED DESCRIPTION

[0034] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0035] The present invention is a method for quickly determining matching epipolar line pairs and calculating camera intrinsic matrices from a set of content-related image sequences or videos. The relevant technologies, methods and optimization parts are introduced below.

[0036] 1. Related technologies

[0037] 1. Background difference method

[0038] Background difference method is a traditional method of moving target detection. This method regards the frame image without moving objects as a static image, and compares other frames with the static image. The area with changes is the moving object. Let the current image frame be F, and the background image B be represented by mathematical modeling. The grayscale of the pixels corresponding to the current frame and the background image are F(x,y) and B(x,y) respectively, and the difference image D(x,y) = |F(x,y)-B(x,y)|. Given a threshold T, according to (Where 255 represents moving pixels and 0 represents background pixels) Each pixel is normalized to obtain a binary image R, and connectivity analysis is performed on R to obtain a complete moving object image. Compared with the frame difference method, the background difference method is prone to produce holes when extracting moving objects.

[0039] 2. Sports barcode

[0040] Motion barcodes use whether a straight line in a frame image intersects with a moving object to represent a moving object in an image sequence or video. The moving object in the video is extracted using the background difference method. If a straight line in a frame intersects with a moving object, it is represented by 1, otherwise it is represented by 0. Motion barcode technology is used to represent moving objects in motion barcode videos.

[0041] Given a video, b l (i) = 1 means that the motion barcode of line segment l in the i-th frame image is {0, 1} N The binary vector b in l .like Figure 2 As shown, when the moving object intersects the straight line l in the i-th frame, b l The value of (i) is "1" (black item), otherwise it is "0" (white item). Assuming that the moving object intersects with l in the i-th frame and does not intersect with l in the j-th frame, the motion barcode of the polar line l is 1 and 0 in the i-th and j-th frames respectively. Therefore, the motion code indicates whether the straight line in the frame intersects with the moving object, and indirectly indicates the moving object. When l is an polar line, the corresponding motion barcode indicates whether the polar line intersects with the moving object.

[0042] The motion barcode of the polar line l indicates whether l intersects with the moving object. / The sports barcode indicates l / The intersection with the moving object, then l and l / The similarity metric of the motion barcodes can be used to find the epipolar lines.

[0043] 3. Polar lines and polar planes

[0044] Define symbols p, q, r as different pixels, Represents the projection imaging of the moving object at viewpoint A at time t. The image point p of the moving object at viewpoint A at time t is imaged at viewpoint B and connected to form the epipolar line.

[0045] As shown in Figure 3(a), A and B are two different viewpoints of the image sequence or video. i The projection image p at the viewpoint A at the moment A , projection imaging at viewpoint B Moving object j Projection imaging at viewpoint B at the moment and Connect to form an epipolar line. Points on the epipolar line of viewpoint B is a moving object k The projection of the moving object at viewpoint B is p A and Connect to form the epipolar line of viewpoint A (as shown in Figure 3(b)). The epipolar line of viewpoint A and the epipolar line of viewpoint B together form the epipolar plane. As can be seen from Figure 3(a) and Figure 3(b), there is a correspondence between the epipolar lines of viewpoint A and viewpoint B and the epipolar plane they form together. This correspondence can be used to obtain corresponding epipolar line pairs of different viewpoints.

[0046] The present invention utilizes the correspondence between the epipolar line and the epipolar plane, finds two different pixels in camera (viewpoint) B from a given pixel of camera (viewpoint) A to obtain an epipolar line, and then finds the corresponding epipolar line in camera (viewpoint) A.

[0047] Finding a pair of corresponding epipolar lines mainly includes searching for two different pixels corresponding to pixel p of camera (viewpoint) A in camera (viewpoint) B to form an epipolar line in B; taking a point in B to form the epipolar line and finding the corresponding point in A to obtain the epipolar line in A.

[0048] 2. Methods

[0049] The method of the present invention mainly includes: first, using the background difference algorithm to extract the mask sequence or mask video of the moving object from a set of content-related image sequences or videos; second, determining the candidate corresponding epipolar line set of viewpoint (camera) A and viewpoint (camera) B; determining the corresponding epipolar line pair of viewpoint (camera) A and viewpoint (camera) B; determining the viewpoint-dependent pole from the corresponding epipolar line. Finally, the camera intrinsic matrix is ​​calculated from the poles and epipolar lines. The following is a detailed analysis and introduction:

[0050] 1. Determination of the viewpoint corresponding epipolar line

[0051] Assume that camera (viewpoint) A is at time t i ,t j Both frames at time contain pixels. View all pixels of the two frames in camera (viewpoint) B and search for the corresponding pixels and Ordered pixel pairs in two frames Determine a candidate epipolar line, set is the set of all epipolar lines. As shown in Figure 3(a), pixel p A The corresponding two pixels are located at and Determine the polar line.

[0052] Λ B For each epipolar line in , search for a matching epipolar line at camera (viewpoint) A. In t k (t k ≠t i ,t k ≠t j ) time frame to search for all satisfying conditions get p A and Λ A Any point Get with l B The corresponding epipolar As shown in Figure 3(b).

[0053] 2. Search for matching epipolar pairs

[0054] Let the epipolar lines l corresponding to cameras A and B (viewpoints) be A , l B , l A and l B The motion barcode is N long. and l A and l B The normalized correlation of is defined as follows

[0055]

[0056] in, The operator mean(·) calculates the mean.

[0057] By using the correlation of motion barcodes between matching epipolar lines, B The highest polar line of normalized correlation of motion barcodes A As the polar line l B The candidates with normalized correlation greater than a certain threshold are selected A and l B are considered as a pair of matching epipolar lines.

[0058] Taking the B camera (viewpoint) as the reference viewpoint, the multi-viewpoint corresponding epipolar line determination and matching epipolar line pair search algorithm is applied to the A camera (viewpoint) to determine the candidate epipolar line pairs.

[0059] 3. Determine the viewpoint-dependent pole from the corresponding epipolar line

[0060] By searching for matching epipolar pairs, we obtain two pairs of corresponding epipolar lines l A,1 , l B,1 and l A,2 , l B,2 , using the random sampling consensus algorithm RANSAC to sample and determine the pole position, the sampling probability is proportional to the normalized correlation of the matching motion barcode. Figure 4 As shown, the intersection points of the two pairs of epipolar lines of camera (viewpoint) A and camera (viewpoint) B are e A 、e B .

[0061] Randomly select t s The frame of time, Connect all the most significant pixels to the poles by putting T A and T B The normalized correlation is calculated for the motion barcodes and matched, and the best candidate barcode is used as the third pair of matching epipolar lines.

[0062] The three pairs of epipolar lines determined provide the solution conditions for calculating the camera intrinsic matrix.

[0063] From the camera (viewpoint) A depends on the pole e A Sample 20 lines uniformly around and search for matching epipolar pairs. Search for the corresponding through e at camera (viewpoint) B. B The normalized cross-correlation of motion barcodes between 20 pairs of epipolar lines is calculated, and the matching epipolar line pair with the highest normalized cross-correlation is selected to calculate the basic matrix.

[0064] 4. Calculate the camera intrinsic matrix from the poles and epipolar lines

[0065] Let the camera intrinsic matrix be F,l B =F[e A ] × l A With l A =F T [e B ] × l B If the scene contains planar objects, l B =Hl A , where H is the homography matrix. H is calculated from multiple sets of epipolar lines, and then the camera intrinsic matrix F is calculated = [e A ] × H.

[0066] In summary, the present invention is a method for quickly determining matching epipolar line pairs and calculating camera intrinsic matrices from a set of content-related image sequences or videos, which mainly includes determining viewpoint corresponding epipolar lines, searching for matching epipolar line pairs, and determining viewpoint-dependent poles from corresponding epipolar lines.

[0067] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the protection scope of the present invention in any form, and all technical solutions obtained by equivalent replacement and the like fall within the protection scope of the present invention.

Claims

1. A method for quickly determining matching epipolar pairs from an image sequence or video and calculating the camera intrinsic matrix, characterized in that The steps include: Step 1: Use background subtraction algorithm to extract the mask sequence or mask video of the moving object from a set of content-related image sequences or videos; Step 2: Determine the candidate corresponding epipolar line set of viewpoint A and viewpoint B; the specific steps are as follows: Step 2.1: Take any image from the image sequence or video and Two frames, in and Search for any corresponding point p in the two frames A , p A Set as the pixel point in viewpoint A; Step 2.2: All pixels in the frame form a pixel set All pixels in the frame form a pixel set Respectively from and Take one point each to form an epiline Λ B Considered as the epipolar line in point B; Step 2.3: Get t k (t k ≠t i ,t k ≠t j ) time frame, search for all satisfying conditions get p A and Any point Get with l B The corresponding epipolar Step 3: Determine the corresponding epipolar line pairs of viewpoint A and viewpoint B; the specific steps are as follows: Step 3.1: If there are two pairs of corresponding epipolar lines of viewpoint A and viewpoint B, go to step 4, otherwise proceed to the next step; Step 3.2: Use the random sampling consensus algorithm RANSAC from Select 20 epilines l B , using the random sampling consensus algorithm from Select 20 epilines l A ; Calculate l A and l B Movement coding; Step 3.3: Calculate l A and l B Normalized Correlation Select the l with the highest normalized correlation A and l B To match the epipolar lines; Step 3.4: Go to step 3.1 and determine a set of corresponding matching epipolar lines of a pair of viewpoints A and B; Step 4: The two epipolar lines of viewpoint A intersect to determine the pole e A , the two epipolar lines of viewpoint B intersect to determine the pole e B ; Step 5: Calculate the camera intrinsic matrix from the poles and epipolar lines; Where, let the camera intrinsic matrix be F,l B =F[e A ] × l A With l A =F T [e B ] × l B Established; when the scene contains planar objects, l B =Hl A , where H is the homography matrix; H is calculated from multiple sets of epipolar lines, and then the camera intrinsic matrix F is calculated = [e A ] × H.

2. An electronic device, characterized in that: The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the steps as claimed in claim 1 are implemented when the program or instruction is executed by the processor.

3. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps as claimed in claim 1 are implemented.

Images