A vehicle-mounted camera calibration method and system based on epipolar constraint

Abstract

This invention discloses a vehicle-mounted camera calibration method and system based on epipolar constraints. The method includes: extracting image feature points from each frame of an acquired continuous image sequence, and obtaining image feature point pairs between consecutive frames of each frame based on the image feature points of each frame; obtaining motion prior information and a first rotational extrinsic parameter corresponding to the image sequence based on the image feature point pairs through a preset epipolar constraint; performing a rationality check on the first rotational extrinsic parameter based on the motion prior information through a preset vehicle incompleteness constraint, and generating a second rotational extrinsic parameter based on the check result, so that camera calibration based on the second rotational extrinsic parameter does not require a target and eliminates the error of the motion decomposition algorithm, thereby improving the efficiency and accuracy of camera calibration.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and system for calibrating vehicle-mounted cameras based on epipolar constraints. Background Technology

[0002] In image measurement and machine vision applications, camera calibration is a crucial step. The accuracy of the calibration results and the stability of the algorithm directly affect the accuracy of the camera's output. With the development of technology, camera technology is gradually being applied to vehicles. During vehicle parking and positioning, the more accurate the camera calibration, the more precise the recognition of the vehicle's surrounding environment.

[0003] Camera calibration is the process of determining the camera's intrinsic and extrinsic parameters, as well as distortion parameters, to establish a geometric model of the camera's imaging. Currently, the most common camera calibration method involves reconstructing the camera's motion from a series of camera images. The estimated motion of the camera and the vehicle are then used to calculate the rotational extrinsic parameters between the two rigid bodies. The calibration of the rotational extrinsic parameters between the camera and the vehicle is achieved by photographing a manually placed target while the vehicle is stationary on a plane. This method relies on the manual target, and the camera motion estimation obtained from continuous images has significant noise and is unstable. Therefore, obtaining stable extrinsic parameter estimates requires a long period of data. Furthermore, the camera motion is obtained through algorithmic processing, not direct sensor observation, thus incorporating errors from the motion decomposition algorithm. Summary of the Invention

[0004] This invention discloses a vehicle-mounted camera calibration method and system based on epipolar constraints, which does not rely on a target and eliminates the error of motion decomposition algorithms, thereby improving the efficiency and accuracy of camera calibration.

[0005] To achieve the above objectives, this invention discloses a vehicle-mounted camera calibration method based on epipolar constraints, comprising:

[0006] Extract image feature points from each frame of the acquired continuous image sequence, and obtain image feature point pairs between the preceding and following frames based on the image feature points of each frame.

[0007] Based on the image feature point pairs, the motion prior information and the first rotation extrinsic parameter corresponding to the image sequence are obtained through a preset epipolar constraint;

[0008] Based on the motion prior information, the rationality of the first rotational extrinsic parameter is checked by the preset non-integrity constraints of the vehicle, and a second rotational extrinsic parameter is generated based on the check result, so that camera calibration can be performed based on the second rotational extrinsic parameter.

[0009] This invention discloses a vehicle-mounted camera calibration method based on epipolar constraints. First, image feature points are extracted from each frame of the acquired image sequence. This allows for the acquisition of image feature point pairs between consecutive frames based on these feature points. The image feature points provide pixel-level accuracy, which is higher than the accuracy obtained through motion decomposition, thus improving the precision of camera calibration. Furthermore, the image feature point pairs allow for the acquisition of data connection relationships between consecutive frames, eliminating the need for manual target marking and improving calibration efficiency. After obtaining the image feature points and their pairs, corresponding motion prior information and preliminary first rotational extrinsic parameters are obtained through preset epipolar constraints. The rationality of these first rotational extrinsic parameters is then checked based on the motion prior information. During the check, the non-integrity constraint of the vehicle is utilized, i.e., the rationality of the first rotational extrinsic parameters is checked based on the influence of the vehicle on the camera. The results of the rationality check are then used to process the first rotational extrinsic parameters, allowing for camera calibration based on the final obtained second rotational extrinsic parameters.

[0010] As a preferred example, the extraction of image feature points for each frame in the acquired continuous image sequence, and the obtaining of image feature point pairs between preceding and following frames based on the image feature points of each frame, includes:

[0011] The acquired continuous image sequence is decomposed to obtain each frame image contained in the image sequence, and the image feature points corresponding to each frame image are extracted by a preset feature point extraction method.

[0012] Based on the image feature points corresponding to each frame, the image feature point pairs between the preceding and following frames are obtained using a preset optical flow method.

[0013] In this invention, image feature points are collected for each frame of the image, which is beneficial for data collection, greatly reduces data collection time, and improves the efficiency of camera calibration. At the same time, by using image feature point pairs between images, it is easier to obtain motion prior information corresponding to continuous images, thereby avoiding target acquisition.

[0014] As a preferred example, obtaining the motion prior information and the first rotation extrinsic parameter corresponding to the image sequence through a preset epipolar constraint includes:

[0015] The essential matrix corresponding to the image feature point pair is calculated, and the essential matrix is ​​decomposed by epipolar constraint to obtain the motion prior information corresponding to the image feature point pair.

[0016] Based on the image feature point pairs, an initial first rotation extrinsic parameter is obtained through a preset epipolar constraint formula, and the initial first rotation extrinsic parameter is optimized through a preset nonlinear optimization algorithm to obtain the first rotation extrinsic parameter.

[0017] This invention obtains motion prior information corresponding to continuous images through the image feature point pairs, avoiding target selection. Then, it uses a preset epipolar constraint formula to obtain the first rotational extrinsic parameter and optimizes the first rotational extrinsic parameter to improve its accuracy, thereby improving the accuracy of camera calibration.

[0018] As a preferred example, obtaining the initial first rotational extrinsic parameters through a preset epipolar constraint formula includes:

[0019] The polar constraint formula is obtained based on the preset first polar constraint formula and the speed conversion formula between the vehicle and the camera in the vehicle.

[0020] The first pole constraint formula is:

[0021] Wherein, X1 and X2 are the normalized coordinates of image feature points; Let I be the unit rotation matrix corresponding to the vehicle moving in a uniform linear motion; and let t be the camera's velocity V. c Multiply by time τ;

[0022] The first polar constraint formula can be converted to: X1*(V c *τ)*I*X2=0、X1*V c *X2 = 0;

[0023] The speed conversion formula between the vehicle and the camera in the vehicle is: Among them, the The speed of the vehicle;

[0024] The non-integrity constraint of the vehicle is: V v =[0,0,1] T ;

[0025] The epipolar constraint formula is then:

[0026] This invention considers the speed of the vehicle and the speed of the camera, and then combines the constructed initial epipolar constraint formula to construct an epipolar constraint formula, so as to obtain the rotational extrinsic parameters according to the epipolar constraint formula, thereby improving the accuracy of the rotational extrinsic parameters.

[0027] As a preferred example, the reasonableness check of the first rotational extrinsic parameter by means of a preset vehicle non-integrity constraint includes:

[0028] Based on the speed conversion formula between the vehicle and the camera in the vehicle and the non-integrity constraints of the vehicle, combined with the first rotational extrinsic parameter, the camera's motion speed is obtained;

[0029] Calculate the dot product of the camera motion speed and the motion prior information, and compare the dot product with a preset threshold to obtain the rationality check result of the first rotation extrinsic parameter.

[0030] This invention combines the nonholonomic constraints of the vehicle to perform a rationality check on the first rotational extrinsic parameter, so as to improve the accuracy of camera calibration by performing camera calibration based on the reasonable rotational extrinsic parameter.

[0031] As a preferred example, generating the second rotational extrinsic parameter based on the test result includes:

[0032] When the dot product is greater than the preset threshold, the first rotation extrinsic parameter is determined to be reasonable, and the first rotation extrinsic parameter is output as the second rotation extrinsic parameter.

[0033] When the dot product is less than or equal to the preset threshold, the first rotational extrinsic parameter is determined to be unreasonable, and the first rotational extrinsic parameter is rotated 180 degrees around the y-axis of the camera to obtain the second rotational extrinsic parameter.

[0034] This invention combines the first rotational extrinsic parameter and the prior motion information to perform a rationality check on the rotational extrinsic parameter, thereby eliminating ambiguity in the rotational extrinsic parameter and improving the accuracy of camera calibration.

[0035] On the other hand, the present invention also discloses a vehicle-mounted camera calibration system based on epipolar constraints, the system including a feature extraction module, an epipolar constraint module and a verification calibration module;

[0036] The feature extraction module is used to extract image feature points of each frame in the acquired continuous image sequence, and to obtain image feature point pairs between the preceding and following frames based on the image feature points of each frame.

[0037] The epipolar constraint module is used to obtain the motion prior information and the first rotation extrinsic parameter corresponding to the image sequence based on the image feature point pairs and through preset epipolar constraints.

[0038] The verification and calibration module is used to verify the rationality of the first rotational extrinsic parameter by means of a preset vehicle non-integrity constraint based on the motion prior information, and to generate a second rotational extrinsic parameter based on the verification result, so as to perform camera calibration based on the second rotational extrinsic parameter.

[0039] This invention discloses a vehicle-mounted camera calibration system based on epipolar constraints. First, image feature points are extracted from each frame of the acquired image sequence. These feature points are then used to obtain image feature point pairs between consecutive frames. The image feature points provide pixel-level accuracy, which is higher than the accuracy obtained through motion decomposition, thus improving the precision of camera calibration. Furthermore, the image feature point pairs allow for the acquisition of data connection relationships between consecutive frames, eliminating the need for manual target marking and improving calibration efficiency. After obtaining the image feature points and their pairs, corresponding motion prior information and preliminary first rotational extrinsic parameters are obtained through preset epipolar constraints. The rationality of these first rotational extrinsic parameters is then checked based on the motion prior information. During the check, the non-integrity constraint of the vehicle is utilized, i.e., the impact of the vehicle on the camera is used to check the rationality of the first rotational extrinsic parameters. The results of the rationality check are then used to process the first rotational extrinsic parameters, allowing for camera calibration based on the final obtained second rotational extrinsic parameters.

[0040] As a preferred example, the feature extraction module includes an image decomposition unit and an optical flow processing unit;

[0041] The image decomposition unit is used to decompose the acquired continuous image sequence to obtain each frame image contained in the image sequence, and extract the image feature points corresponding to each frame image by a preset feature point extraction method.

[0042] The optical flow processing unit is used to obtain the image feature point pairs between the preceding and following frames corresponding to each frame image by using a preset optical flow method based on the image feature points corresponding to each frame image.

[0043] In this invention, image feature points are collected for each frame of the image, which is beneficial for data collection, greatly reduces data collection time, and improves the efficiency of camera calibration. At the same time, by using image feature point pairs between images, it is easier to obtain motion prior information corresponding to continuous images, thereby avoiding target acquisition.

[0044] As a preferred example, the polar constraint module includes a motion information unit and a rotational extrinsic parameter unit;

[0045] The motion information unit is used to calculate the essential matrix corresponding to the image feature point pair, and to perform epipolar constraint decomposition on the essential matrix to obtain the motion prior information corresponding to the image feature point pair.

[0046] The rotation extrinsic unit is used to obtain an initial first rotation extrinsic parameter based on the image feature point pair using a preset epipolar constraint formula, and to optimize the initial first rotation extrinsic parameter using a preset nonlinear optimization algorithm to obtain the first rotation extrinsic parameter.

[0047] This invention obtains motion prior information corresponding to continuous images through the image feature point pairs, avoiding target selection. Then, it uses a preset epipolar constraint formula to obtain the first rotational extrinsic parameter and optimizes the first rotational extrinsic parameter to improve its accuracy, thereby improving the accuracy of camera calibration.

[0048] As a preferred example, the inspection and calibration module includes a rationality inspection unit and a camera calibration unit;

[0049] The rationality verification unit is used to obtain the camera motion speed based on the speed conversion formula between the vehicle and the camera in the vehicle and the non-integrity constraints of the vehicle, combined with the first rotation extrinsic parameter; calculate the dot product of the camera motion speed and the motion prior information, and compare the dot product with a preset threshold to obtain the rationality verification result of the first rotation extrinsic parameter;

[0050] The camera calibration unit is used to determine that the first rotation extrinsic parameter is reasonable when the dot product is greater than the preset threshold, and output the first rotation extrinsic parameter as the second rotation extrinsic parameter; when the dot product is less than or equal to the preset threshold, the first rotation extrinsic parameter is determined to be unreasonable, and the first rotation extrinsic parameter is rotated 180 degrees around the y-axis of the camera to obtain the second rotation extrinsic parameter.

[0051] This invention combines the nonholonomic constraints of the vehicle to perform a rationality check on the first rotational extrinsic parameter, so that camera calibration can be performed based on the rational rotational extrinsic parameter, thereby improving the accuracy of camera calibration. At the same time, by combining the first rotational extrinsic parameter and the motion prior information to perform a rationality check on the rotational extrinsic parameter, the ambiguity of the rotational extrinsic parameter is eliminated, thereby improving the accuracy of camera calibration. Attached Figure Description

[0052] Figure 1 : A schematic flowchart of a vehicle-mounted camera calibration method based on epipolar constraints provided in an embodiment of the present invention;

[0053] Figure 2 : A schematic diagram of a vehicle-mounted camera calibration system based on epipolar constraints provided in an embodiment of the present invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] Example

[0056] This invention provides a calibration method for vehicle-mounted cameras based on epipolar constraints. The specific implementation process of the calibration method can be found in [reference needed]. Figure 1 It mainly includes steps 101 to 103, which mainly include:

[0057] Step 101: Extract the image feature points of each frame in the acquired continuous image sequence, and obtain the image feature point pairs between the preceding and following frames based on the image feature points of each frame.

[0058] In this embodiment, the step mainly includes: performing image decomposition on the acquired continuous image sequence to obtain each frame image contained in the image sequence, and extracting the image feature points corresponding to each frame image respectively through a preset feature point extraction method; and obtaining the image feature point pairs between the preceding and following frames corresponding to each frame image according to the image feature points corresponding to each frame image through a preset optical flow method.

[0059] Preferably, in this embodiment, during the straight-line driving of the vehicle, feature points of continuous images obtained by the camera on the vehicle are tracked and extracted, and then motion priors are calculated. Specifically, the acquired image sequence is decomposed to obtain each frame image. The image sequence can be a continuous image sequence In, In+1..., and feature points on the In and In+1 frames are further extracted. Then, the optical flow method is used to obtain the corresponding image feature point pairs Pn and Pn+1 based on the feature points on the In and In+1 frames. Furthermore, in this embodiment, the method for extracting the feature points includes multiple methods, including manual feature extraction methods such as FAST, ORB, and SIFT, and deep learning feature point extraction methods such as super-point. Different feature extraction methods can be adopted according to different needs. Furthermore, the optical flow method can obtain the matching relationship of feature points between two frames, that is, p1 corresponds to p1', and p2 corresponds to p2'.

[0060] In this embodiment, the step of collecting image feature points for each frame of the image is beneficial for data collection, greatly reduces data collection time, and improves the efficiency of camera calibration. At the same time, by utilizing the image feature point pairs between consecutive images, it is easier to obtain motion prior information corresponding to continuous images, thereby avoiding target acquisition.

[0061] Step 102: Based on the image feature point pairs, obtain the motion prior information and the first rotation extrinsic parameter corresponding to the image sequence through a preset epipolar constraint.

[0062] In this embodiment, the step mainly includes: calculating the essential matrix corresponding to the image feature point pair, performing epipolar constraint decomposition on the essential matrix to obtain the motion prior information corresponding to the image feature point pair; obtaining the initial first rotation extrinsic parameters according to the image feature point pair through a preset epipolar constraint formula, and optimizing the initial first rotation extrinsic parameters through a preset nonlinear optimization algorithm to obtain the first rotation extrinsic parameters.

[0063] Furthermore, the epipolar constraint formula can be obtained from a preset first epipolar constraint formula and a speed conversion formula between the vehicle and the camera in the vehicle. The first epipolar constraint formula is: Wherein, X1 and X2 are the normalized coordinates of image feature points; Let I be the unit rotation matrix corresponding to the vehicle moving in a uniform linear motion; and let t be the camera's velocity V. c Multiply by time τ; where the first pole constraint formula can be converted to: X1*(V c *τ)*I*X2=0、X1*V c *X2=0; The speed conversion formula between the vehicle and the camera in the vehicle is: Among them, the Let V be the speed of the vehicle; the non-integrity constraint of the vehicle is: V v =[0,0,1] T The epipolar constraint formula is then:

[0064] Specifically, in this embodiment, based on feature point pairs Pn, Pn+1..., the motion prior information tp of the image is obtained through epipolar constraint decomposition. In this embodiment, the correspondence of the feature points means that two points correspond to points in the same three-dimensional space. The essential matrix can be calculated from the correspondence of the feature points. By performing singular value decomposition on the essential matrix, the corresponding motion information, namely translation and rotation, can be obtained. Optionally, an epipolar constraint formula is constructed based on the provided first epipolar constraint formula, the velocity conversion formula between the vehicle and the camera in the vehicle, and the vehicle's non-integrity constraint. Then, the image feature point pairs Pn, Pn+1 are processed by the epipolar constraint formula to obtain the first rotation extrinsic parameters.

[0065] In this embodiment, this step obtains motion prior information corresponding to continuous images through the image feature point pairs, avoiding target selection. Then, a first rotational extrinsic parameter is obtained using a preset epipolar constraint formula, and the first rotational extrinsic parameter is optimized to improve its accuracy, thereby improving the accuracy of camera calibration. Simultaneously, the vehicle speed and camera speed are considered, and then an epipolar constraint formula is constructed in conjunction with the constructed initial epipolar constraint formula, so that the rotational extrinsic parameter is obtained according to the epipolar constraint formula, thus improving the accuracy of the rotational extrinsic parameter.

[0066] Step 103: Based on the motion prior information, the rationality of the first rotational extrinsic parameter is checked by the preset non-integrity constraints of the vehicle, and the second rotational extrinsic parameter is generated based on the check result so that the camera calibration is performed based on the second rotational extrinsic parameter.

[0067] In this embodiment, the step mainly includes: obtaining the camera's motion speed based on the speed conversion formula between the vehicle and the camera in the vehicle and the non-integrity constraints of the vehicle, combined with the first rotational extrinsic parameter; calculating the dot product of the camera's motion speed and the motion prior information, and comparing the dot product with a preset threshold to obtain the rationality check result of the first rotational extrinsic parameter.

[0068] Furthermore, when the dot product is greater than the preset threshold, the first rotation extrinsic parameter is determined to be reasonable, and the first rotation extrinsic parameter is output as the second rotation extrinsic parameter; when the dot product is less than or equal to the preset threshold, the first rotation extrinsic parameter is determined to be unreasonable, and the first rotation extrinsic parameter is rotated 180 degrees around the y-axis of the camera to obtain the second rotation extrinsic parameter.

[0069] Specifically, in this embodiment, in order to eliminate the ambiguity of the solved first rotational extrinsic parameters, the first rotational extrinsic parameters are adjusted according to the motion prior information tp of the image. Perform a reasonableness test, and further determine the reasonableness test result T. If the check is correct, the second rotational extrinsic parameter is the first rotational extrinsic parameter; if incorrect, the first rotational extrinsic parameter is rotated 180 degrees along the camera's Y-axis. Preferably, in this embodiment, the rationality check process is as follows: because the equation corresponding to the epipolar constraint formula provided in this embodiment still holds even when one side is multiplied by any constant, the solved rotational extrinsic parameter will be ambiguous, that is, the solution will converge to one of the two solutions, and the two solutions will rotate around the camera's Y-axis by 180 degrees. The known conversion formula between vehicle speed and camera speed is... The non-integrity constraint of the vehicle is V v =[0,0,1] TSubstituting the first rotational extrinsic parameter obtained by solving the equation yields the camera motion velocity Vc. Using the prior motion information tp from the image, if the solution... If correct, then the dot product of Vc and tp is greater than 0. If the dot product is less than 0, then the solution needs to be... Simply rotate the camera 180 degrees around its y-axis.

[0070] In this embodiment, this step combines the vehicle's nonholonomic constraints to perform a rationality check on the first rotational extrinsic parameters, so that camera calibration can be performed based on reasonable rotational extrinsic parameters, thereby improving the accuracy of camera calibration. At the same time, by combining the first rotational extrinsic parameters and the motion prior information to perform a rationality check on the rotational extrinsic parameters, the ambiguity of the rotational extrinsic parameters is eliminated, thereby improving the accuracy of camera calibration.

[0071] On the other hand, embodiments of the present invention also disclose a vehicle-mounted camera calibration system based on epipolar constraints, the specific structural composition of which can be referred to... Figure 2 The system includes a feature extraction module 201, a parabola constraint module 202, and a calibration module 203.

[0072] The feature extraction module 201 is used to extract image feature points of each frame in the acquired continuous image sequence, and to obtain image feature point pairs between the preceding and following frames based on the image feature points of each frame.

[0073] The epipolar constraint module 202 is used to obtain the motion prior information and the first rotation extrinsic parameter corresponding to the image sequence based on the image feature point pairs through a preset epipolar constraint.

[0074] The verification and calibration module 203 is used to verify the rationality of the first rotational extrinsic parameter by means of a preset vehicle non-integrity constraint based on the motion prior information, and to generate a second rotational extrinsic parameter based on the verification result, so as to perform camera calibration based on the second rotational extrinsic parameter.

[0075] In this embodiment, the feature extraction module 201 includes an image decomposition unit and an optical flow processing unit.

[0076] The image decomposition unit is used to decompose the acquired continuous image sequence to obtain each frame image contained in the image sequence, and to extract the image feature points corresponding to each frame image by a preset feature point extraction method.

[0077] The optical flow processing unit is used to obtain the image feature point pairs between the preceding and following frames corresponding to each frame image by using a preset optical flow method based on the image feature points corresponding to each frame image.

[0078] In this embodiment, the polar constraint module 202 includes a motion information unit and a rotational extrinsic parameter unit.

[0079] The motion information unit is used to calculate the essential matrix corresponding to the image feature point pair, and to perform epipolar constraint decomposition on the essential matrix to obtain the motion prior information corresponding to the image feature point pair.

[0080] The rotation extrinsic unit is used to obtain an initial first rotation extrinsic parameter based on the image feature point pair using a preset epipolar constraint formula, and to optimize the initial first rotation extrinsic parameter using a preset nonlinear optimization algorithm to obtain the first rotation extrinsic parameter.

[0081] In this embodiment, the inspection and calibration module 203 includes a rationality inspection unit and a camera calibration unit.

[0082] The rationality verification unit is used to obtain the camera motion speed based on the speed conversion formula between the vehicle and the camera in the vehicle and the non-integrity constraints of the vehicle, combined with the first rotation extrinsic parameter; calculate the dot product of the camera motion speed and the motion prior information, and compare the dot product with a preset threshold to obtain the rationality verification result of the first rotation extrinsic parameter.

[0083] The camera calibration unit is used to determine that the first rotation extrinsic parameter is reasonable when the dot product is greater than the preset threshold, and output the first rotation extrinsic parameter as the second rotation extrinsic parameter; when the dot product is less than or equal to the preset threshold, the first rotation extrinsic parameter is determined to be unreasonable, and the first rotation extrinsic parameter is rotated 180 degrees around the y-axis of the camera to obtain the second rotation extrinsic parameter.

[0084] Furthermore, this embodiment also discloses a computer-readable storage medium and an electronic device. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements a vehicle-mounted camera calibration method based on epipolar constraints as described in this embodiment of the invention. The electronic device includes: at least one processor; and a storage device configured to store at least one program. When the at least one program is executed by the at least one processor, the at least one processor implements a vehicle-mounted camera calibration method based on epipolar constraints as described in this embodiment of the invention.

[0085] This invention discloses a vehicle-mounted camera calibration method and system based on epipolar constraints. First, image feature points are extracted from each frame of the acquired image sequence. This allows for the acquisition of image feature point pairs between consecutive frames based on these feature points. The image feature points provide pixel-level accuracy, which is higher than the accuracy obtained through motion decomposition, thus improving the precision of camera calibration. Furthermore, the image feature point pairs allow for the acquisition of data connection relationships between consecutive frames, eliminating the need for manual target marking and improving the efficiency of camera calibration. After obtaining the image feature points and their pairs, corresponding motion prior information and preliminary first rotational extrinsic parameters are obtained through preset epipolar constraints. The rationality of the first rotational extrinsic parameters is then checked based on the motion prior information. During the verification process, the non-integrity constraint of the vehicle is utilized, that is, the rationality of the first rotational extrinsic parameter is checked based on the influence of the vehicle on the camera. The result of the rationality check is used to process the first rotational extrinsic parameter so that the camera can be calibrated based on the finally obtained second rotational extrinsic parameter. Furthermore, the calibration method provided in this embodiment utilizes epipolar constraints and vehicle non-integrity constraints to establish the relationship between pixel feature points, vehicle motion, and camera mounting rotational extrinsic parameters. The camera extrinsic parameters can be directly solved from pixel coordinates, improving accuracy and shortening the calibration time. At the same time, the information based on the extracted image feature points is rich, and a large number of feature points can be extracted per frame, which is conducive to data collection and can significantly reduce data collection time. Moreover, no target is required, and image feature points can provide pixel-level accuracy, which is more accurate than the result obtained through motion decomposition and eliminates the error of motion decomposition algorithm.

[0086] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A vehicle-mounted camera calibration method based on epipolar constraint, characterized in that, The method comprises the following steps: extracting image feature points of each frame image in the obtained continuous image sequence, and obtaining image feature point pairs between front and rear frame images of each frame image according to the image feature points of each frame image; obtaining motion prior information and a first rotation external parameter corresponding to the image sequence according to the image feature point pairs through a preset epipolar constraint; performing rationality inspection on the first rotation external parameter through a preset nonholonomic constraint of the vehicle, and generating a second rotation external parameter according to the inspection result, so as to calibrate the camera according to the second rotation external parameter; wherein, the step of obtaining motion prior information and a first rotation external parameter corresponding to the image sequence according to the image feature point pairs through a preset epipolar constraint comprises: calculating an essential matrix corresponding to the image feature point pairs, and performing epipolar constraint decomposition on the essential matrix to obtain the motion prior information corresponding to the image feature point pairs; obtaining an initial first rotation external parameter according to the image feature point pairs through a preset epipolar constraint formula, and optimizing the initial first rotation external parameter through a preset nonlinear optimization algorithm to obtain the first rotation external parameter; The initial first rotation external parameter is obtained through a preset epipolar constraint formula, comprising: obtaining the epipolar constraint formula according to a preset first epipolar constraint formula and a speed conversion formula of the vehicle and the camera in the vehicle; the first epipolar constraint formula is: ; wherein the , is the normalized coordinate of the image feature point; the is the corresponding unit rotation matrix when the vehicle is in a uniform linear form; the is the camera motion speed multiplied by time τ; wherein the first epipolar constraint formula can be converted into: , ; the speed conversion formula of the vehicle and the camera in the vehicle is: ; wherein the is the speed of the vehicle; the nonholonomic constraint of the vehicle is: ; and the epipolar constraint formula is: .

2. The vehicle camera calibration method based on epipolar constraint according to claim 1, wherein, the step of extracting image feature points of each frame image in the obtained continuous image sequence, and obtaining image feature point pairs between front and rear frame images of each frame image according to the image feature points of each frame image comprises: performing image decomposition on the obtained continuous image sequence to obtain each frame image contained in the image sequence, and extracting image feature points corresponding to each frame image through a preset feature point extraction method; obtaining image feature point pairs between front and rear frame images corresponding to each frame image through a preset optical flow method according to the image feature points corresponding to each frame image.

3. The method of claim 1, wherein the method comprises: the step of performing rationality inspection on the first rotation external parameter through a preset nonholonomic constraint of the vehicle comprises: obtaining the camera motion speed by combining the first rotation external parameter according to a speed conversion formula of the vehicle and a camera in the vehicle and the nonholonomic constraint of the vehicle; calculating the dot product of the camera motion speed and the motion prior information, and comparing the dot product with a preset threshold to obtain the rationality inspection result of the first rotation external parameter; wherein, the preset threshold is 0.

4. The method of claim 3, wherein the method comprises: the step of generating a second rotation external parameter according to the inspection result comprises: when the dot product is greater than 0, it is determined that the first rotation external parameter is reasonable, and the first rotation external parameter is output as the second rotation external parameter; when the dot product is less than or equal to 0, it is determined that the first rotation external parameter is unreasonable, and the first rotation external parameter is rotated by 180 degrees around the y-axis of the camera to obtain the second rotation external parameter.

5. A vehicle-mounted camera calibration system based on epipolar constraint, characterized in that, The system comprises a feature extraction module, an epipolar constraint module and an inspection and calibration module; the feature extraction module is used for extracting image feature points of each frame image in the obtained continuous image sequence, and obtaining image feature point pairs between front and rear frame images of each frame image according to the image feature points of each frame image; The epipolar constraint module is configured to obtain motion prior information and a first rotation external parameter corresponding to the image sequence according to the image feature point pairs by using a preset epipolar constraint; The calibration module is configured to perform rationality inspection on the first rotation external parameter according to the motion prior information by using a preset nonholonomic constraint of the vehicle, and generate a second rotation external parameter according to an inspection result, so as to calibrate the camera according to the second rotation external parameter; The epipolar constraint module includes a motion information unit and a rotation external parameter unit; the motion information unit is configured to calculate an essential matrix corresponding to the image feature point pairs, and perform epipolar constraint decomposition on the essential matrix to obtain the motion prior information corresponding to the image feature point pairs; and the rotation external parameter unit is configured to obtain an initial first rotation external parameter according to the image feature point pairs by using a preset epipolar constraint formula, and optimize the initial first rotation external parameter by using a preset nonlinear optimization algorithm to obtain the first rotation external parameter; The initial first rotation external parameter is obtained through a preset epipolar constraint formula, including: obtaining the epipolar constraint formula according to a preset first epipolar constraint formula and a speed conversion formula of the vehicle and the camera in the vehicle. ; wherein the , is a normalized coordinate of the image feature point; the is a corresponding unit rotation matrix when the vehicle is in a uniform straight line form; the is a camera motion speed multiplied by time τ; wherein the first epipolar constraint formula can be converted into: , ; the speed conversion formula of the vehicle and the camera in the vehicle is: ; wherein the is the speed of the vehicle; the nonholonomic constraint of the vehicle is: ; and the epipolar constraint formula is: .

6. A vehicle camera calibration system based on epipolar constraint as claimed in claim 5 wherein, The feature extraction module includes an image decomposition unit and an optical flow processing unit; The image decomposition unit is configured to perform image decomposition on the obtained continuous image sequence to obtain each frame of image included in the image sequence, and extract image feature points corresponding to each frame of image by using a preset feature point extraction method; The optical flow processing unit is configured to obtain image feature point pairs between front and rear frames of image corresponding to each frame of image by using a preset optical flow method according to the image feature points corresponding to each frame of image.

7. A vehicle camera calibration system based on epipolar constraint as claimed in claim 5 wherein, The calibration module includes a rationality inspection unit and a camera calibration unit; The rationality inspection unit is configured to obtain a camera motion speed according to a speed conversion formula of the vehicle and a camera in the vehicle and a nonholonomic constraint of the vehicle, in combination with the first rotation external parameter; calculate a dot product of the camera motion speed and the motion prior information, and compare the dot product with a preset threshold to obtain a rationality inspection result of the first rotation external parameter; and the preset threshold is 0; The camera calibration unit is configured to determine that the first rotation external parameter is reasonable when the dot product is greater than 0, and output the first rotation external parameter as the second rotation external parameter; and determine that the first rotation external parameter is unreasonable when the dot product is less than or equal to 0, and rotate the first rotation external parameter by 180 degrees around a y-axis of the camera to obtain the second rotation external parameter.

Images