Camera calibration parameter automatic correction method based on image feature matching and geometric transformation and time sequence-based bipolar matching mapping method

Abstract

The invention discloses a camera calibration parameter automatic correction method based on image feature matching and geometric transformation and a time sequence-based bipolar matching mapping method. The process of the method comprises the following steps: S1, acquiring a view angle picture which changes before and after a camera moves, and performing feature matching by using a LightGlue matching algorithm; s2, converting the matching point pair from a pixel coordinate to a normalized camera coordinate system; s3, solving a rotation matrix and a translation vector after the camera moves through the matching point pair set; s4, reading initial rotation matrixes and translation vectors of the camera A and the camera B before moving; s5, performing matrix operation on the rotation matrix and the translation vector obtained twice to obtain a basic matrix F from the camera A to the camera B after movement; s6, performing target detection on the image by using a YOLO-V5 model; s7, screening a matching target through a forward matcher and a reverse matcher; and S8, matching succeeds after the time sequence queue constraint is met.

Description

Technical Field

[0001] The present invention mainly includes computer vision, camera calibration, three-dimensional reconstruction and image processing technology. Specifically, the automatic correction method of camera calibration parameters based on image feature matching and geometric transformation belongs to the field of camera positioning and posture estimation technology. The dual epipolar matching mapping method based on time series belongs to the field of feature matching and stereo vision and multi-view geometry in computer vision. Background Art

[0002] In recent years, deep learning has made significant progress in computer vision, especially in stereo matching and 3D reconstruction. Matching methods based on deep learning use convolutional neural networks (CNNs) to automatically learn image features and obtain more robust and accurate matching strategies through end-to-end training. Although deep learning methods can learn matching relationships between image pairs on large-scale datasets, they rely on a large amount of labeled data and are prone to overfitting on the training set, resulting in performance degradation in practical applications, especially in different scenes and changes in perspective.

[0003] Existing cameras are divided into two categories: hydroxyl and mercapto. The viewing angle of hydroxyl cameras is generally adjusted manually, and will not change easily after being fixed, but position shifts may occur due to environmental factors; while the viewing angle of mercapto cameras can be adjusted electrically, and will automatically return to the initial angle after each restart, but multiple power outages may make it impossible to restore to the viewing angle position during calibration. Regardless of the type, once the camera calibration is completed, once the viewing angle changes, it needs to be recalibrated, which is a time-consuming and tedious task. Therefore, it is necessary to design a method that can automatically correct the parameters after the camera viewing angle changes.

[0004] Epipolar matching is widely used in tasks such as 3D reconstruction and depth estimation. In binocular stereo vision, the epipolar line refers to the projection line of the same 3D point in two camera images. The task of epipolar matching is to find the corresponding matching points in the two images, so as to infer the 3D information of the object. Since epipolar matching depends on calibration accuracy, it is prone to mismatching in areas with large parallax changes, especially when the number of targets is large, the mismatching problem is more prominent. To solve this problem, we propose an automatic correction method for camera calibration based on image feature matching and geometric transformation to ensure that the calibration accuracy is always accurate. Then, a dual epipolar matching mapping method based on time series is designed, which can achieve lightweight and high-precision target matching, effectively avoid the mismatching problem of traditional epipolar matching, and solve the problem of poor robustness and dependence on a large amount of labeled data in deep learning methods. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the prior art and propose an automatic correction method for camera calibration parameters based on image feature matching and geometric transformation and a dual epipolar matching mapping method based on time sequence. To this end, the present invention specifically adopts the following technical solutions:

[0006] Step S1: Obtain the image of the camera's view angle during calibration and the image of the camera's view angle after the camera moves, and use the LightGlue matching algorithm to perform feature matching on the images before and after the camera moves.

[0007] Step S2: Convert the matching point pairs from pixel coordinates to the normalized camera coordinate system to reduce the influence of distortion and noise on the results.

[0008] Step S3: Solve the essential matrix (E) by matching the point pair set. Recover the rotation matrix (R1) and translation vector (T1) before and after the camera moves from the essential matrix.

[0009] Step S4: Read the rotation matrix (R2) and translation vector (T2) of camera A and camera B during calibration before moving. R1 and T1 are synthesized with R2 and T2 to form homogeneous transformation matrices P1 and P2 respectively.

[0010] Step S5: P1 and P2 are subjected to a series of matrix operations to calculate the basic matrix F from camera A to camera B after the movement.

[0011] A further technical solution is to use a dual epipolar matching mapping method based on time sequence to ensure that the calibration parameters are corrected in time and accurately after steps S1 to S5, and the specific technical solution is as follows:

[0012] Step S6: Use the YOLO-V5 model to perform target detection on the image, obtain the target detection frame, extract the position information of the target detection frame, and obtain the target pixel coordinates.

[0013] Step S7: Screening matching targets through the forward matcher and the reverse matcher.

[0014] Step S8: Determine whether the matching target that meets the condition in S7 meets the timing queue constraint. If the timing queue constraint is met, return the successfully matched target and add the new matching target to the timing queue. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present invention, but are not intended to limit the present invention.

[0016] Figure 1 : LightGlue feature matching map of the camera before and after movement.

[0017] Figure 2 : Flowchart of the automatic correction method of camera calibration parameters based on image matching and geometric transformation.

[0018] Figure 3 : Flowchart of the timing-based dual epipolar matching mapping method. DETAILED DESCRIPTION

[0019] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0020] Reference Figure 1 , Figure 2 , Figure 3 , a camera calibration parameter automatic correction method based on image feature matching and geometric transformation and a dual epipolar matching mapping method based on time sequence, the method comprising the following steps:

[0021] Step S1: Obtain the image of the camera's view angle during calibration and the image of the camera's view angle after the camera moves, and use the LightGlue matching algorithm to perform feature matching on the images before and after the camera moves.

[0022] Step S2: Convert the matching point pairs from pixel coordinates to the normalized camera coordinate system to reduce the influence of distortion and noise on the results.

[0023] Step S3: Solve the essential matrix (E) through the set of matching point pairs, and recover the rotation matrix (R1) and translation vector (T1) before and after the camera moves from the essential matrix.

[0024] Step S31: For the image before and after the camera moves, the successfully matched feature point x1=(u1,v1,1) T and x2 = (u2, v2, 1) T , use the camera's intrinsic parameter matrices K1 and K2 for normalization transformation:

[0025]

[0026] Get the normalized coordinates of the matching points in the two images and

[0027] Step S32: normalize the matching points and Substitute the following essential matrix constraints:

[0028]

[0029] Among them, E is the essential matrix. In order to solve E, we first construct a matrix A:

[0030]

[0031] For each pair of matching points, construct each row of the matrix. Integrate all matching points into matrix A. Reduce the dimension of matrix A through singular value decomposition (SVD) and obtain the least squares solution to calculate the essential matrix E.

[0032] Step S33: Recover the rotation and translation vectors from the essential matrix E by using singular value decomposition (SVD) to obtain the decomposition of E:

[0033] E=U·∑·V T

[0034] Where U and V are orthogonal matrices, and ∑ is a diagonal matrix containing the singular values ​​of E.

[0035] According to SVD decomposition, two possible rotation matrices R1 and R2 can be constructed. The two rotation matrices are in the following form:

[0036] R1=U·W·V T

[0037] R2=U·W T ·V T

[0038] Where W is a fixed matrix, usually:

[0039]

[0040] The translation vector T can be recovered by the following formula:

[0041] T1=U[:,2]

[0042] T2=-U[:,2]

[0043] Among them, U[:,2] is the third column of matrix U.

[0044] The translation vectors T1 and T2 are two candidate values, representing two possible solutions for the relative position of the cameras.

[0045] Step S34: The decomposition of the essential matrix usually produces four solutions: the combination of two rotation matrices and two translation vectors forms four possible solutions. It is possible to verify which solutions are physically reasonable by triangulating the matching points. Select those solutions that make the three-dimensional points within the camera's field of view.

[0046] Step S4: Read the rotation matrix (R2) and translation vector (T2) of camera A and camera B during calibration before moving. R1 and T1 are synthesized with R2 and T2 to form homogeneous transformation matrices P1 and P2 respectively.

[0047] Step S5: P1 and P2 are subjected to a series of matrix operations to calculate the basic matrix F from camera A to camera B after the movement.

[0048] Step S51: P1 = [R1|T1], indicating the rotation and translation relationship between camera A before and after the movement. P2 = [R2|T2], indicating the rotation and translation relationship between camera A and camera B before and after the movement.

[0049] Step S52: Combine multiple coordinate transformations into an overall transformation through matrix multiplication. Obtain the transformation matrix P from the moved camera A to the camera B, that is: P = P1.P2

[0050] Step S53: Through the matrix P, the rotation matrix R and displacement vector T between camera A and camera B after the movement can be extracted.

[0051] Step S54: Using the cross product matrix T of the translation vector cross And the rotation matrix R, calculate the new essential matrix E. Combined with the known camera intrinsic parameters K, the basic matrix F can be finally calculated. That is, the external parameter basic matrix F from camera A to camera B after the movement is obtained.

[0052]

[0053] A further technical solution is to adopt a dual epipolar matching mapping method based on time sequence on the premise of ensuring that the calibration parameters are corrected in time and accurately after steps S1 to S5, and the method comprises the following steps:

[0054] Step S6: Use the YOLO-V5 model to perform target detection on the image, obtain the target detection frame, extract the position information of the target detection frame, and obtain the target pixel coordinates.

[0055] Step S7: Screening matching targets through the forward matcher and the reverse matcher.

[0056] Step S71: Draw an epipolar line based on the pixel coordinates of the target ID detected by camera A, and find a target near the epipolar line for matching under camera B. If the distance between the target and the epipolar line is less than d min , add the target to the queue to be matched.

[0057] Step S72: sort the targets in the queue to be matched in ascending order of distance from the extreme line, and iterate the queue to be matched.

[0058] Step S73: Draw an epipolar line to map the pixel coordinates of the target ID in the queue to be matched to the perspective of camera A, and determine whether the target closest to the epipolar line is the corresponding ID. If it is not the corresponding ID, perform the next iteration. If it is temporarily considered to be a match, then determine whether the timing queue constraints are met.

[0059] Step S8: Determine whether the matching target that meets the condition in S7 meets the timing queue constraint. If the timing queue constraint is met, return the successfully matched target and add the new matching target to the timing queue.

[0060] Step S81: The timing queue stores the matching results of each id including: its own id, the match_id of the matching target, the x and y coordinates of the matching target, and saves them in a queue with a fixed queue length of 10. When the length exceeds 10, the earliest matching result is removed and the latest matching result is inserted.

[0061] Step S82: The matching targets selected by the forward matcher and the reverse matcher in step S7 are subjected to timing queue constraints. If the following constraints are met, the target is considered to be matched successfully, the new target is added to the matching queue, and the matching result is returned.

[0062] Condition 1: When the length of the matching queue of the target is less than 3, the target selected by the matcher meets the timing queue constraint by default and is automatically added to the timing queue. When the length of the matching queue of the target is greater than 3, condition 2 must be met.

[0063] Condition 2: The number of matches between the current match_id and the match_id that already exists in the queue must be greater than the queue length - 1. If this condition is met, the queue is added to the time sequence queue and the matching result is returned. If this condition is not met, condition 3 is judged.

[0064] Condition 3: Compare the pixel coordinates of match_id with the pixel coordinates of the match_id already in the queue. The number of pixels less than 35 must be greater than or equal to the queue length - 1. If this condition is met, the new matching target is added to the timing queue and the matching result is returned. If this condition is not met, return to S82 for the next iteration.

[0065] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A camera calibration parameter automatic correction method based on image feature matching and geometric transformation and a dual epipolar matching mapping method based on time sequence, the method comprising the following steps: S1: Obtain the image of the camera's view during calibration and the image of the camera's view after the camera moves, and use the LightGlue matching algorithm to perform feature matching on the images before and after the camera moves. S2: Convert the matching point pairs from pixel coordinates to the normalized camera coordinate system to reduce the impact of distortion and noise on the results. S3: Solve the essential matrix (E) by matching the point pair set. Recover the rotation matrix (R1) and translation vector (T1) before and after the camera moves from the essential matrix. S4: Read the rotation matrix (R2) and translation vector (T2) of camera A and camera B during calibration before moving. R1 and T1 are synthesized with R2 and T2 to form homogeneous transformation matrices P1 and P2 respectively. S5: P1 and P2 are subjected to a series of matrix operations to calculate the basic matrix F from camera A to camera B after the movement. A further technical solution is to use a dual epipolar matching mapping method based on time sequence under the premise of ensuring that the calibration parameters are corrected in time and accurately after steps S1 to S5, and the method includes the following steps: S6: Use the YOLO-V5 model to perform target detection on the image, obtain the target detection frame, extract the position information of the target detection frame, and obtain the target pixel coordinates. S7: Filter matching targets through the forward matcher and the reverse matcher. S8: Determine whether the matching target that meets the S7 condition meets the timing queue constraint. If the timing queue constraint is met, the match is successful. The new matching target is added to the timing queue and the matching result is returned.

2. The method for automatically correcting camera calibration parameters based on image feature matching and geometric transformation and the method for dual epipolar matching mapping based on time sequence described in claim 1 are characterized in that: The dual epipolar matching mapping method based on time series depends on the stability and accuracy of the camera calibration accuracy. The automatic correction method of camera calibration parameters based on image matching and geometric transformation can ensure the stability and accuracy of the camera calibration accuracy. Therefore, the dual epipolar matching mapping method based on time series is completed when the automatic correction method of camera calibration parameters based on image matching and geometric transformation is implemented. Steps S1 to S5 are the automatic correction process of camera calibration parameters, and S6-S9 are the dual epipolar matching mapping process based on time series.

3. In the method for automatic correction of camera calibration parameters based on image feature matching and geometric transformation described in claim 1, it is characterized in that: In step S1, the image of the current camera angle of view and the image of the initial camera calibration or the image of the camera after the camera moves and the calibration parameters have been automatically corrected are automatically obtained at a fixed time. The pre-trained LightGlue model is used to load the pre-trained model weights and perform feature matching on the two images.

4. In the method for automatic correction of camera calibration parameters based on image feature matching and geometric transformation described in claim 1, it is characterized in that: In step S2, the matching point pairs are converted from pixel coordinates to a normalized camera coordinate system in combination with the camera's internal parameters, thereby reducing the influence of distortion and noise on the results, thereby improving the accuracy of calibration parameter correction.

5. In the method for automatic correction of camera calibration parameters based on image feature matching and geometric transformation described in claim 1, it is characterized in that: In step S3, the successfully matched feature points after normalization are constructed into a matrix, and the dimension of the matrix is ​​reduced by singular value decomposition to obtain the least squares solution to calculate the essential matrix, and then the rotation matrix and translation vector are restored from the essential matrix to obtain the rotation matrix R2 and translation vector T2 before and after the camera moves.

6. In the method for automatic correction of camera calibration parameters based on image feature matching and geometric transformation described in claim 1, it is characterized in that: In step S4, the rotation matrix R1 and the translation vector T1 currently used by a pair of cameras are read to form a transformation matrix P1, and the rotation matrix R2 and the translation vector T2 calculated before and after the camera moves are also formed into a transformation matrix P2.

7. In the method for automatic correction of camera calibration parameters based on image feature matching and geometric transformation described in claim 1, it is characterized in that: In step S5, P1 and P2 obtained in S4 are multiplied to obtain a transformation matrix P, and the rotation matrix and translation vector are restored from P. The new essential matrix E is calculated by the cross product matrix of the translation vector and the rotation matrix. Combined with the known camera intrinsic parameter K, a new basic matrix F is calculated.

8. In the timing-based dual epipolar line matching mapping method described in claim 1, it is characterized in that: In step S6, the pre-trained YOLO-V5 detection model is used to load the pre-trained weights to detect the target in the image. The position information of the target detection frame is extracted to obtain the target pixel coordinates, and then the matching target is found for each target under the corresponding viewing angle.

9. In the timing-based dual epipolar line matching mapping method described in claim 1, it is characterized in that: In step S7, the forward matcher is first used to find a target that meets the requirements of the matcher under the corresponding viewing angle, and then the reverse matcher is used to verify whether the target matches the target selected by the forward matcher.

10. In the timing-based dual epipolar line matching mapping method described in claim 1, it is characterized in that: In step S8, it is determined whether the target that meets the condition in S7 meets the constraints of the timing queue, and the target is compared with the matching target already existing in the timing queue and the pixel coordinates of the matching target. If the constraints of the timing queue are met, it is considered that the match is successful, the successfully matched target is returned, and the new matching target is added to the timing queue. Otherwise, the next round of iterative judgment is performed on the next target screened by the forward matcher.

Images