Vehicle-mounted binocular vision calibration method based on scene information understanding

Abstract

The invention discloses a vehicle-mounted binocular vision calibration method based on scene information understanding, and belongs to the technical field of vehicle-mounted vision monitoring. According to the method, external parameters, namely a rotation matrix and a translation vector, of a camera are calibrated, three-dimensional space re-projection constraint and depth information estimation are introduced in the external parameter optimization process on the basis of a traditional epipolar geometric calibration method, and the modeling capacity for long-distance point space uncertainty is enhanced; meanwhile, aiming at external parameters of multi-frame estimation, a multi-frame result clustering algorithm is designed, and estimation deviation caused by single-frame noise and a bad initial value is effectively reduced. According to the method, the defects that a traditional self-calibration method based on epipolar geometry only depends on two-dimensional projection point information, a long-distance target is difficult to process accurately, and the traditional self-calibration method depends on a better initial value are overcome.

Description

Technical Field

[0001] This invention relates to the field of vehicle-mounted visual monitoring technology, and more particularly to a vehicle-mounted binocular vision calibration method based on scene information understanding, which mainly calibrates the extrinsic parameters of the camera. Background Technology

[0002] Binocular vision is an important branch of machine vision. It involves using two cameras at different positions to capture the same scene while rotating, and then calculating the parallax of a point in space between the two images to obtain the three-dimensional coordinates of that point. Stereo vision is a method of acquiring three-dimensional geometric information of an object from multiple images (usually two). Compared to other methods of acquiring three-dimensional information, binocular vision more reliably and simply simulates how human eyes process objects, and has applications in many fields, such as vehicle vision monitoring, robot navigation, three-dimensional measurement of industrial parts, and virtual reality.

[0003] As a binocular vision system for intelligent visual surveillance (Reference 1), it should be able to perform real-time detection, localization, tracking, identification, and abnormal behavior alarm functions for targets within the monitored field of view. The basic theoretical components of a binocular vision system include... Figure 1 As shown, the process begins with image acquisition using a camera, followed by camera calibration to establish a mapping between the image captured by the camera and the image in the real scene. This also assists in the registration between two or more cameras, providing a foundation for obtaining depth information. After the binocular vision system is calibrated, a moving target detection algorithm is used to detect targets of interest (such as people, vehicles, etc.).

[0004] In the field of vision system research, the accuracy of binocular vision systems significantly depends on the accuracy of camera calibration. Existing binocular extrinsic parameter calibration methods are mainly based on epipolar geometry theory to model the mathematical relationship between projection points in the left and right images. This provides a solid theoretical foundation and support for camera (or binocular camera) self-calibration algorithms. Reference 2, based on epipolar geometry theory, models the camera's extrinsic parameters (i.e., rotation matrix R and translation vector t), obtains the associated projection points in the left and right images through a traditional binocular matching algorithm, and then substitutes the two-dimensional position information of the projection points into the epipolar geometry model, obtaining the camera's extrinsic parameters through numerical optimization.

[0005] Problem 1: When performing self-calibration using epipolar geometry constraints, only the two-dimensional information of the projected points is utilized, often neglecting their spatial information. This is particularly problematic when processing distant objects in three-dimensional space. The reason is that points far from the imaging plane often suffer from higher positioning errors. This stems from the fact that when the camera maps three-dimensional spatial information onto a two-dimensional plane, a single pixel is actually mapped to a frustum region in three-dimensional space. Furthermore, as the depth of the coordinate point increases, the range of this frustum region expands accordingly, exacerbating the uncertainty of spatial positioning. Therefore, the reliability of such imaging point pairs is relatively low, which may adversely affect the optimization process of the self-calibration algorithm, thereby weakening the accuracy and stability of the calibration.

[0006] Problem 2: In "High-Precision Online Markerless Stereo Extrinsic Calibration," the external parameter values ​​need to be continuously updated through numerical optimization to eventually converge to a better solution. However, this requires a good initial value; otherwise, the results will be unsatisfactory. The reason for this is that epipolar geometry only utilizes two-dimensional information, resulting in limited input information. Therefore, a clear optimization direction cannot be provided during the optimization process. Furthermore, existing binocular matching algorithms suffer from insufficient accuracy and mismatches, leading to noise during optimization. This, without a good initial value, may result in a suboptimal solution.

[0007] Reference 1, Binocular Vision System, Beijing: Central University for Nationalities Press, December 2020, author Liu Xiangchun, pp. 1-7.

[0008] Reference 2, Ling Y, Shen S. High-precision online markerless stereoextrinsic calibration[C] / / 2016 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2016: 1771-1778. Summary of the Invention

[0009] To address the aforementioned shortcomings, this invention proposes a vehicle-mounted binocular vision calibration method based on scene information understanding. This method improves upon traditional epipolar geometry calibration methods by introducing 3D spatial reprojection constraints and depth information estimation during extrinsic parameter optimization, enhancing the ability to model spatial uncertainties at distant points. Furthermore, a multi-frame result clustering algorithm is designed for multi-frame estimated extrinsic parameters, effectively reducing estimation bias caused by single-frame noise and poor initial values. Since image acquisition involves 3D information, the introduction of 3D reprojection and depth information estimation of pixels enhances the image understanding of 3D scene information.

[0010] See Figure 2 The present invention illustrates a vehicle-mounted binocular vision calibration system based on scene information understanding. First, images are acquired using a camera. Then, camera calibration is performed to establish a mapping relationship between the image within the camera and the image in the real scene. Simultaneously, registration between two or more cameras is assisted, providing a foundation for obtaining depth information. The process from image acquisition to target detection includes an image feature extraction unit, a feature matching unit (2), a reprojection and loss calculation unit (3), a nonlinear optimization unit (4), and a multi-frame estimation result clustering unit (5). The image feature extraction unit, due to the acquisition of two image blocks (left and right), is designed with separate left image feature extraction units (1A) and right image feature extraction units (1B). During camera calibration, only the feature points of the left image block are processed. Feature points of the right image patch , left intersection point Intersection with the right Optimize.

[0011] The image feature extraction unit is used to input the left and right binocular images acquired at the current moment into the vehicle-mounted intelligent vision monitoring center, and process them through the original binocular matching algorithm of the vehicle-mounted intelligent vision monitoring center to obtain the position of any feature point in the left and right image blocks.

[0012] Feature matching unit (2), based on camera coordinate system Draw a parallel line to the Z-axis. Project the lines originating from the optical center of the left eye camera and the optical center of the right eye camera. These two projection lines intersect the parallel line to the Z-axis at two points, namely the left intersection point. right intersection point ;Will Corresponding to the camera coordinate system middle;

[0013] The reprojection and loss calculation unit (3) introduces three-dimensional spatial reprojection constraints and depth information estimation during the optimization of the camera's extrinsic parameters, which enhances the ability to model spatial uncertainties of distant points; the camera's extrinsic parameters are rotation matrix and translation vector.

[0014] The nonlinear optimization unit (4) performs nonlinear optimization based on the Taylor expansion algorithm and the Lie algebra space method;

[0015] The multi-frame estimation results are clustered into units (5). The K-means clustering method is used to cluster the multi-frame estimation results to obtain the rotation vector cluster centers. Translation vector cluster centers .

[0016] The present invention provides a vehicle-mounted binocular vision calibration method based on scene information understanding, comprising the following steps:

[0017] Step 1: Input the left and right binocular images acquired at the current moment into the vehicle-mounted intelligent vision monitoring center;

[0018] Step 11: Obtain the location of feature points in the image patch;

[0019] The left image patch acquired from the image is denoted as The right image patch acquired by the image acquisition is denoted as .

[0020] The The coordinates of any feature point in the left image block are obtained through the original binocular matching algorithm of the vehicle-mounted intelligent vision monitoring center. , In this diagram, u represents the x-coordinate value, v represents the y-coordinate value, and the superscript i represents the feature point identifier. Let n be the total number of feature points. Then the coordinates of the last feature point in the left image patch are... .

[0021] The The coordinates of any feature point in the right image patch are obtained through the original binocular matching algorithm processed by the vehicle-mounted intelligent vision monitoring center. , In this context, u represents the x-coordinate and v represents the y-coordinate. Therefore, the coordinates of the last feature point in the right image patch are... .

[0022] In binocular vision image acquisition, the With the A pair of feature points, i.e., a feature point pair .

[0023] Step 12, obtaining the intersection of the left and right points;

[0024] like Figure 3 In the diagram showing the relationship between the binocular coordinate systems, the camera coordinate system is denoted as... The coordinate system of the left-eye camera is denoted as In coordinate system One of the two-dimensional pixel coordinates is (Also known as feature points). The coordinate system of the right eye camera is denoted as... In coordinate system One of the two-dimensional pixel coordinates is (Also known as feature points). The intrinsic parameters of the left eye camera are denoted as... The intrinsic parameters of the right eye camera are denoted as Using coordinate systems Draw a parallel line to the Z-axis. The red dashed line is the projection line originating from the optical center of the left eye camera, and the blue dashed line is the projection line originating from the optical center of the right eye camera. The two projection lines intersect the parallel line to the Z-axis at two points, namely the left intersection point. right intersection point .Will Corresponding to the camera coordinate system middle.

[0025] Obtained through step one , , and Used for calibrating cameras, i.e., camera calibration resources.

[0026] Step 2: Calculate the loss function based on the current stereo matching results;

[0027] In step two, the extrinsic parameters of the camera, namely the rotation matrix and translation vector, are calibrated. Based on the traditional epipolar geometry calibration method, three-dimensional spatial reprojection constraints and depth information estimation are introduced in the process of extrinsic parameter optimization, which enhances the ability to model spatial uncertainties of distant points.

[0028] Set internal parameters ,and for The matrix; setting internal parameters ,and for The matrix. and stated To determine the camera manufacturer.

[0029] Ideally, let's assume Through With coordinate system The association of obtaining Similarly, let's assume... Through With coordinate system The association of obtaining .

[0030] It is the left intersection point The corresponding Z-axis coordinate value in the left camera coordinate system.

[0031] It is the right intersection point The corresponding Z-axis coordinate value in the right camera coordinate system.

[0032] Through the above and stated Able to To link them together, that is R is the rotation matrix, and t is the translation vector; where R is The matrix R represents the rotation of the left and right camera coordinate systems in the coordinate system. Let t be a vector of length 3. The rotational relationships on the three axes; the translation vector t represents the coordinate systems of the left and right cameras in the coordinate system. The translation vector.

[0033] Not to be introduced for the time being To obtain the coordinate system The homogeneous coordinates below are .

[0034] It is the left intersection point In coordinate system The corresponding homogeneous coordinates in the middle.

[0035] It is the right intersection point In coordinate system The corresponding homogeneous coordinates in the middle.

[0036] When introduced In homogeneous coordinates, we have .

[0037] This is the result observed directly in the left camera coordinate system. This is the result observed by the right camera and transformed into the coordinate system of the left camera. At this point... The distance between them is denoted as Then we have:

[0038] .

[0039] The goal of optimizing the method of this invention is to make Minimize it, and then obtain the rotation matrix R and the translation vector t. For example... Figure 3 As shown, because In real space, they correspond to the same region, so when R and t are accurate, their distance is minimized. Using this information, a loss function was designed. However, formula (1) contains two incoherent ( The unknowns of () need to be represented by R and t.

[0040] Because of the need to calculate Corresponding to If the true value is obtained, then it is necessary to utilize... Ideally, these two coordinate points would coincide with this constraint, such as... Figure 3 As shown, coordinate system The blue and red dots in the diagram correspond to the same point, and ideally they should overlap.

[0041] Step 21: Derive the depth consistency and collinearity equations based on the left and right image point coincidence constraints;

[0042] (A) Due to If they overlap, then there are ,in identity matrix The third line, that is That is, the Z-axis coordinate values ​​z of the two are equal (having the same depth), which can be further expanded into formula (2).

[0043] .

[0044] For ease of calculation, multiply both sides. First, multiply both sides by... Thus, we obtain formula (3).

[0045] .

[0046] because Therefore, the result on the left side of formula (3) is You can get a result about The equation is denoted as formula (4). In formula (4), we first assume that R and t are known quantities.

[0047] .

[0048] (B) Due to They coincide, each starting from the origin (the optical center of the camera). We obtain two vectors with the endpoint as follows: . From the origin to the destination The vector, From the origin to The two vectors are ideally in the same direction, and the cross product of vectors in the same direction is 0, which leads to a new equation. ,in Indicates using Cross product The matrix corresponding to the time.

[0049] Step 22: Eliminate the depth variable, derive the distance metric between the left and right image points, and construct the matching loss function;

[0050] Will After unfolding, we get , here It can be eliminated directly, and then obtained The solution is given by formula (5).

[0051] .

[0052] Will Substituting the solution into formula (3), we can obtain the result. The solution is formula (6).

[0053] .

[0054] Thus, we have obtained Substituting the solutions for R and t into formula (1) yields formula (7).

[0055] .

[0056] It can be found (because This represents the result of the cross product of two vectors. This result must be perpendicular to the original two vectors. and Since they are perpendicular, the vector dot product is 0), so formula (7) can be simplified to formula (8).

[0057] .

[0058] Substituting all n matching results into the formula yields the loss function LOSS, which is... .

[0059] In this invention, the loss function is based on the Hinge loss function formula, referencing "Neural Networks and Deep Learning", Beijing: Machinery Industry Press, April 2020, author: Qiu Xipeng, page 29.

[0060] Step 3: Calculate the derivative of the loss function with respect to the input variables based on the loss function.

[0061] Based on the loss function Loss, the derivatives of R and t are calculated. This involves estimating the extrinsic parameters for multi-frame estimation. Two-dimensional pixels are reprojected back into three-dimensional space, and their relative depth information is estimated using the current extrinsic parameter estimates, thereby enhancing the image understanding of the current scene. Nonlinear optimization is performed using the Taylor expansion algorithm and the Lie algebra space method.

[0062] In this invention, in order to better represent the loss function The loss function was simplified by setting an input variable denoted as... ,use Replacing part of the loss function, we have .

[0063] In this invention, during the camera calibration process, the currently estimated rotation matrix is ​​denoted as... The currently estimated translation vector is denoted as Based on the Taylor expansion algorithm, It is represented by formula (9).

[0064] .

[0065] for At the current estimate Value at this location.

[0066] for At the current estimate Regarding The first derivative.

[0067] For increments, and , Let R be the increment of R in the Lie algebra space. Let t be the increment of t in the Lie algebra space; in this invention A vector of length 3.

[0068] Among them, the Lie algebra space method can be used to obtain The derivative is expressed as formula (10).

[0069] .

[0070] in, for right The derivative of .

[0071] in, for right The derivative of .

[0072] Step four: Iteratively update the input variables using numerical optimization methods;

[0073] Let s be the number of any iteration (also called the iteration number identifier), and S be the maximum number of iterations (also called the last iteration number).

[0074] Based on the derivative Calculate the rotation matrix during the s-th step iteration. Translation vector .

[0075] Based on the loss function From formula (9), the derivative of the LOSS function is expressed as formula (11).

[0076] .

[0077] The LM algorithm is used to optimize formula (11) so that the loss function value is minimized, i.e., formula (12).

[0078]

[0079] This is the value corresponding to Hinge Loss.

[0080] for The coordinate transpose.

[0081] This is the damping coefficient used in the LM algorithm.

[0082] It is an identity matrix.

[0083] For increments, and , Let R be the increment of R in the Lie algebra space. Let t be the increment of t in the Lie algebra space.

[0084] The Hinge loss function is introduced to prevent outliers from having an excessive impact on the optimization process. Therefore, it is necessary to limit the loss function value for some samples with excessively large loss values.

[0085] During the optimization process, there is a constraint on the translation vector t: its length does not change during optimization. Therefore, we restrict t to a spherical space, treating it as a vector that can rotate around the origin. Then, based on the theory of Lie groups and Lie algebras, we can... The parameterization is expressed as formula (13).

[0086] .

[0087] Similarly, regarding the rotation matrix R, we can also... The parameterization is expressed as formula (14).

[0088] .

[0089] Combining formulas (13) and (14), we obtain the solution for the increment Δ, and then use the optimization method in Lie algebra to update the current value. We obtain the iteration value of the s-th round, i.e., formula (15).

[0090] .

[0091] Let R be the increment of R in the Lie algebra space.

[0092] Let t be the increment of t in the Lie algebra space.

[0093] This is the currently estimated rotation matrix.

[0094] This is the currently estimated translation vector.

[0095] This completes one round of optimization in the LM algorithm (Levenberg-Marquardt, nonlinear least squares optimization algorithm), from the current estimate. Starting from this point, the rotation matrix in the s-th step iteration process was obtained through optimization. Translation vector .

[0096] When the maximum number of iterations S is reached, the external parameter estimation rotation matrix is ​​output. And extrinsic parameter estimation of translation vector This completes the update of the rotation matrix and translation vector in the binocular image.

[0097] Step 5: Perform multi-frame estimation optimization based on clustering;

[0098] Step 51: Convert all rotation matrices into rotation vectors;

[0099] Using a three-dimensional vector to represent a three-dimensional rotation transformation, where the direction of the vector is the rotation axis and its magnitude is the rotation angle, the transformation from a rotation matrix to a rotation vector can be obtained using the Rodriguez formula. The Rodriguez formula is referenced in "Des lois géometriques qui regissent les déplacements d' un systéme solidedans l' espace, et de la variation des coordonnées provenant de ces déplacement considérées indépendant des causes qui peuvent les produire", J.Math. Pures Appl. 5 (1840), 380–440.

[0100] The rotation matrix during the s-th step iteration process The estimated extrinsic parameter values ​​for the first frame. Based on the Rodriguez method... Convert to a rotation vector, denoted as .

[0101] The rotation matrix during the s-th step iteration process The estimated extrinsic parameter values ​​for the second frame. Based on the Rodriguez method... Convert to a rotation vector, denoted as .

[0102] The rotation matrix during the s-th step iteration process The estimated extrinsic parameter values ​​for the j-th frame. Based on the Rodriguez method... Convert to a rotation vector, denoted as .

[0103] The rotation matrix during the s-th step iteration process The estimated extrinsic parameters for the m-th frame. Based on the Rodriguez method... Convert to a rotation vector, denoted as .

[0104] The rotation vectors obtained statistically based on the Rodriguez method are denoted as the set of rotation vectors. .

[0105] The translation vector during the iteration process at step s is The estimated extrinsic values ​​for the first frame.

[0106] The translation vector during the iteration process at step s is The estimated extrinsic values ​​for the second frame.

[0107] The translation vector during the iteration process at step s is The estimated extrinsic parameter values ​​for the j-th frame.

[0108] The translation vector during the iteration process at step s is The estimated extrinsic parameter values ​​for the m-th frame.

[0109] The translation vector estimation results of multiple frames are denoted as the translation vector set. .

[0110] This is the calibration estimate. The subscript j indicates the frame identifier, and... , The total number of frames.

[0111] Step 52: Unify the vector set and calculate the initial cluster centers;

[0112] Using the K-means clustering method to Perform clustering to obtain the rotation vector cluster centers. .

[0113] Using the K-means clustering method to Clustering is performed to obtain the cluster centers of the translation vector. The aforementioned It is the final result of the translation vector in camera calibration.

[0114] The K-means clustering method is referenced from Section 1.1 of "Research on XGBoost Ensemble Algorithm Based on K-means Clustering" in the 10th issue of "Computer Era" in 2020, authored by Luo Chunfang et al.

[0115] Step 53: Use the final cluster centers to inversely calculate the rotation and translation to obtain the robust initial pose;

[0116] Using the Rodriguez method Reverse the direction to obtain the reversed rotation matrix. The aforementioned It is the final result of the rotation matrix in camera calibration.

[0117] Traditional epipolar geometry algorithms only utilize the two-dimensional information of matching points, neglecting the significant spatial uncertainty of distant points, resulting in poor optimization performance on the translation vector t. This invention, however, reprojects two-dimensional pixels back into three-dimensional space and estimates their relative depth information using current extrinsic parameter estimates, thereby deepening the image understanding of the current scene. Based on this information, this invention can restrict distant points (corresponding to higher losses) during the optimization process, preventing them from significantly impacting the optimization of the translation vector, thus reducing noise generation during optimization.

[0118] Meanwhile, addressing the issue of traditional algorithms requiring a good initial value, this invention introduces a clustering algorithm in step five. Using the K-means clustering method, the estimation results from multiple frames are clustered to obtain the final result. This way, even if a single frame lacks a good initial value, leading to a significant deviation in the final estimation result, the optimization between frames is relatively independent across multiple frames. Therefore, the directions of deviation from the true value are likely to differ, which facilitates the reduction of overall noise through clustering in this invention. Through the clustering algorithm of this invention, the noise from each frame can be mutually canceled out, resulting in a better aggregated value, which can be used as the initial value for subsequent frames.

[0119] The technical advantages of the method of this invention are as follows:

[0120] Utilization of 3D information: For the same scene image, this method introduces 3D reprojection and depth information estimation of pixels during the calibration process, which breaks through the limitation of traditional epipolar geometry relying only on 2D pixel constraints. The introduction of 3D reprojection and depth information estimation of pixels enhances the ability to model distant points.

[0121] Noise suppression mechanism: For the optimization process of camera extrinsic parameters during calibration, a loss constraint is introduced for distant feature points to avoid excessive interference with translation vector estimation.

[0122] Multi-frame clustering initial value optimization: For the extrinsic parameter results estimated in multiple frames, outliers are eliminated by clustering the independent estimation results of multiple frames to obtain more reliable initial values, thereby improving the overall optimization accuracy and robustness. Attached Figure Description

[0123] Figure 1 This is a structural block diagram of a typical binocular vision system.

[0124] Figure 2 This is a structural block diagram of the camera calibration system based on scene information understanding according to the present invention.

[0125] Figure 3 This is a schematic diagram of a camera capturing images for dual-target positioning.

[0126] Figure 4The disparity map results obtained by calculating disparity using the extrinsic parameters obtained by Zhang Zhengyou's calibration method.

[0127] Figure 5 The result is a disparity map obtained by calculating disparity using the extrinsic parameters estimated by this method.

[0128] Figure 6 This is a multi-frame rotation vector clustering result diagram obtained using the method of this invention.

[0129] Figure 7 This is a multi-frame translation vector clustering result diagram obtained using the method of this invention. Detailed Implementation

[0130] The present invention will now be described in further detail with reference to the accompanying drawings. The examples of the parameters listed are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

[0131] Figure 4 and Figure 5 Three scene images at different distances were acquired, and disparity calculations were performed on the images. (a) shows the image acquired at a long distance, (b) at a medium distance, and (c) at a short distance. Vehicle-mounted binocular cameras typically require a compromise baseline (commonly between 0.2m and 0.5m) to balance near-range obstacle detection and medium-to-long-range road target perception. The distance limits in (a), (b), and (c) are relative distances from the camera calibration process. Generally, near distance refers to a relative distance of less than 10m between the front of the vehicle and the reference object. Medium distance refers to a relative distance of 10m-50m between the front of the vehicle and the reference object. Long distance refers to a relative distance of more than 50m between the front of the vehicle and the reference object.

[0132] This embodiment uses the KITTI 2015 dataset to calculate disparity and verify the results of the extrinsic parameters obtained using the Zhang Zhengyou calibration method. Figure 4 As shown. Figure 4 The disparity map obtained using this extrinsic parameter is presented. The results show that the disparity map calculated using Zhang Zhengyou's calibration method can clearly depict the overall outlines and some local details of various objects in the scene, especially showing good edge and structural representation in near-range target areas. Furthermore, for distant targets, this method can also recover their approximate outline information, providing a benchmark reference for subsequent comparative experiments.

[0133] This embodiment also verifies the effectiveness of the method of the present invention on the KITTI 2015 dataset. Figure 5The disparity map results obtained by calculating disparity using the extrinsic parameters estimated using this method are shown. Figure 4 and Figure 5 The comparison results show that the disparity map generated by the extrinsic parameters estimated by this method is superior in both overall structure and detail (i.e., Figure 5 )and Figure 4 The results shown are largely consistent, and no significant deviation was observed in the disparity estimation for distant scene points (i.e., Figure (a)). This fully demonstrates that the proposed method has good accuracy and robustness in modeling distant targets.

[0134] This embodiment presents the results of multi-frame rotation vector clustering of this method on the KITTI 2015 dataset. Figure 6 The horizontal axis represents the number of image frames used for multi-frame estimation, and the vertical axis represents the angular error between the estimated rotation vector converted into an orientation angle and the true value. The results show that as the number of frames involved in the estimation gradually increases, the orientation angle estimation error, after processing with the clustering strategy proposed in this invention, exhibits a monotonically decreasing trend and gradually approaches the true value, eventually converging to a relatively small angular range. This demonstrates better accuracy and stability compared to single-frame estimation results.

[0135] This embodiment further presents the multi-frame translation vector clustering results of this method on the KITTI 2015 dataset. Figure 7 The horizontal axis represents the number of image frames used for multi-frame estimation, and the vertical axis represents the Euclidean distance between the estimated translation vector and the true translation vector. Experimental results show that as the number of frames involved in the estimation increases, the translation estimation error obtained by the clustering method used in this invention continuously decreases and gradually stabilizes, eventually converging to a numerical range that is better than single-frame estimation. This verifies the effectiveness of the multi-frame clustering strategy in improving the accuracy of translation extrinsic parameter estimation.

Claims

1. A vehicle-mounted binocular vision calibration method based on scene information understanding, involving the calibration and optimization of cameras; characterized in that... Includes the following steps: Step 1: Input the left and right binocular images acquired at the current moment into the vehicle-mounted intelligent vision monitoring center; Step 11: Obtain the location of feature points in the image patch; Left image block The original binocular matching algorithm of the in-vehicle intelligent visual monitoring center is processed to obtain the coordinates of any feature point in the left image block ; right image block The original binocular matching algorithm of the in-vehicle intelligent vision monitoring center is processed to obtain the coordinates of any feature point in the right image block ; In binocular vision image collection, the image collection device comprises a left camera and a right camera The left camera and the right camera are arranged on the same plane A pair of feature points is denoted as a feature point pair ; Step 12, obtaining the intersection of the left and right points; With camera coordinate system Draw a parallel line to the Z-axis. The projection lines originating from the optical center of the left eye camera and the right eye camera intersect the parallel line at two points, namely the left intersection point. right intersection point ; Step 2: Calculate the loss function based on the current stereo matching results; Step 21: Derive the depth consistency and collinearity equations based on the left and right image point coincidence constraints; Step 22: Eliminate the depth variable, derive the distance metric between the left and right image points, and construct the matching loss function; Substituting all n matching results into the formula yields the loss function LOSS, which is... ; Step 3: Calculate the derivative of the loss function with respect to the input variables based on the loss function. Step four: Iteratively update the input variables using numerical optimization methods; Update the current using optimization techniques from Lie algebras We obtain the iteration value of the s-th round, i.e. ;in, Let R be the increment of R in the Lie algebra space; Let t be the increment of t in the Lie algebra space; This is the currently estimated rotation matrix; This is the currently estimated translation vector; Step 5: Perform multi-frame estimation optimization based on clustering; Step 51: Convert all rotation matrices into rotation vectors; The set of rotation vectors obtained according to the Rodriguez method ; Multi-frame translation vector estimation results ; Step 52: Unify the vector set and calculate the initial cluster centers; The Kmeans clustering method is used for clustering to obtain a rotation vector clustering center ; The Kmeans clustering method is used for clustering to obtain a translation vector clustering center ; Step 53: Use the final cluster centers to inversely calculate the rotation and translation to obtain the robust initial pose; Using the Rodrigues method on R = RrR ; the final result of the rotation matrix in camera calibration is R = RrR 2. The vehicle-mounted binocular vision calibration method based on scene information understanding according to claim 1, characterized in that... In step two, The distance between the two points is denoted by ; (A) Due to If they overlap, then there are ,in identity matrix The third line, that is That is, the Z-axis coordinate values ​​z of the two are equal, which can be further expanded into formula (2); ； For ease of calculation, multiply both sides. First, multiply both sides by... Thus, we obtain formula (3); ； As a result , the left side of equation (3) becomes , resulting in an equation for , denoted as equation (4); ； (B) Due to Overlapping, each starting from the origin, We obtain two vectors with the endpoint as follows: ; From the origin to the destination The vector, From the origin to The two vectors are ideally in the same direction, and the cross product of vectors in the same direction is 0, which leads to a new equation. ,in Indicates using Cross product The matrix corresponding to the time.

3. The vehicle-mounted binocular vision calibration method based on scene information understanding according to claim 1, characterized in that... In step two, After unfolding, we get , here It can be eliminated directly, and then obtained The solution is formula (5); ； Substituting the solution of into equation (3), the solution of , i.e. equation (6), is obtained; ； Thus, we have Substituting the solution for R and t into equation (1) gives equation (7); ； discovery Equation (7) can be simplified to Equation (8); 。 4. The scene information understanding based vehicle-mounted binocular vision calibration method according to claim 1, characterized in that In step three, the derivatives of R and t are calculated based on the loss function Loss; To simplify , one input variable is set instead, and ; Let the currently estimated rotation matrix be denoted as The currently estimated translation vector is denoted as According to Taylor expansion Represented as formula (9); ； for At the current estimate Value at location; for At the current estimate Regarding The first derivative; For increments, and , Let R be the increment of R in the Lie algebra space. Let t be the increment of t in the Lie algebra space; This can be obtained by using the Lie algebra space. The derivative is expressed as formula (10); ； in, for right The derivative; in, for right The derivative of .

5. The vehicle-mounted binocular vision calibration system based on scene information understanding, which implements the method of claim 1, is characterized in that: First, images are acquired using a camera, then the camera is calibrated to complete the mapping relationship between the images in the camera and the images in the real scene. At the same time, it assists in the registration between two or more cameras, providing a basis for obtaining depth information. The process from image acquisition to target detection includes an image feature extraction unit, a feature matching unit (2), a reprojection and loss calculation unit (3), a nonlinear optimization unit (4), and a multi-frame estimation result clustering unit (5). The image feature extraction unit is used to input the left and right binocular images acquired at the current moment into the vehicle-mounted intelligent vision monitoring center, and process them through the original binocular matching algorithm of the vehicle-mounted intelligent vision monitoring center to obtain the position of any feature point in the left and right image blocks. Feature matching unit (2), based on coordinate system Draw a parallel line to the Z-axis. Project the lines originating from the optical center of the left eye camera and the optical center of the right eye camera. These two projection lines intersect the parallel line to the Z-axis at two points, namely the left intersection point. right intersection point ;Will Corresponding to the camera coordinate system middle; The reprojection and loss calculation unit (3) introduces three-dimensional spatial reprojection constraints and depth information estimation during the optimization of the camera's extrinsic parameters, which enhances the ability to model spatial uncertainties of distant points; the camera's extrinsic parameters are rotation matrix and translation vector. The nonlinear optimization unit (4) performs nonlinear optimization based on the Taylor expansion algorithm and the Lie algebra space method; The multi-frame estimation results are clustered into units (5). The K-means clustering method is used to cluster the multi-frame estimation results to obtain the rotation vector cluster centers. Translation vector cluster centers .

6. A vehicle-mounted computer-readable storage medium having a computer program stored thereon, characterized in that: When executed by a processor, the computer program implements the steps of the vehicle-mounted binocular vision calibration method based on scene information understanding as described in any one of claims 1-4.

7. An in-vehicle computer program product, comprising a computer program; characterized in that: When executed by a processor, the computer program implements the steps of the vehicle-mounted binocular vision calibration method based on scene information understanding as described in any one of claims 1-4.

Images