A method for extrinsic calibration of vehicle-mounted cameras applicable to all scenes without mapping

Abstract

The present application relates to a method for calibrating extrinsic parameters of a vehicle-mounted camera applicable to all scenes without mapping, the method comprising: collecting first image data of a vehicle in motion, and processing the first image data to obtain rotation extrinsic parameters; collecting second image data and wheel speed data of the vehicle in motion, and processing the second image data, wheel speed data, and the rotation extrinsic parameters to obtain translation extrinsic parameters. The beneficial effect of the present application is that the present application can collect multiple frames of visual information during the normal driving of the vehicle, and calibrate the camera extrinsic parameters with the help of wheel speed sensors. It does not require professional operators and special sites, and has a wide range of applications.

Description

Technical Field

[0001] The present application relates to the field of automotive electronics technology, and more specifically, to a method for calibrating extrinsic parameters of a vehicle-mounted camera applicable to all scenarios without mapping. Background Art

[0002] As an important sensor for sensing the surrounding environment of the vehicle, the vehicle camera has a wide range of applications in the fields of assisted driving and intelligent driving. All camera-based function development requires the conversion of the camera coordinate system to the vehicle body coordinate system or the earth coordinate system. Therefore, after the vehicle camera is installed, it needs to be calibrated to obtain the camera's external parameters before it can be used normally.

[0003] The current mainstream calibration solutions are mainly divided into three categories:

[0004] The first category: Calibration is performed using known specific markers, such as setting up a calibration site or laying out a calibration cloth. The transformation matrix is ​​calculated through the corner points of known physical coordinates and image corner points, and the rotation and translation of the camera are then restored.

[0005] The second category: calibration using other sensors, such as using a vehicle-mounted inertial measurement unit (IMU) for calibration. It is necessary to simultaneously calculate the posture of the camera to be calibrated and the reference sensor, and obtain the conversion relationship between the camera and reference sensor coordinate systems through multi-frame motion postures.

[0006] The third category: Use the front and rear frames of the image to adjust the external parameters. By matching the front and rear frame images of the camera, some parameters of the camera external parameters are adjusted.

[0007] The first type of solution is not only costly and complex to operate, but the vehicle cannot be calibrated once it leaves the factory or 4S shop, resulting in a poor user experience. The second solution is similar to the hand-eye calibration of a robotic arm. It not only relies on the measurement accuracy of the reference sensor and the accuracy of its installed external parameters, but most importantly, when the vehicle is driving normally, the motion state is limited to the ground plane, the imu cannot fully rotate, and the equation group for calculating the transformation matrix is ​​not full rank, so a stable solution cannot be obtained. The third type of solution relies on production line calibration, and can only adjust a limited number of degrees of freedom based on the existing relatively accurate external parameters. Summary of the invention

[0008] In order to overcome the problems in the prior art, the present application provides a method for calibrating extrinsic parameters of a vehicle-mounted camera that is applicable to all scenes without mapping.

[0009] A method for calibrating extrinsic parameters of a vehicle-mounted camera applicable to all scenes without mapping, the method comprising:

[0010] Collecting first image data of a vehicle while it is traveling, and processing the first image data to obtain an external parameter of a rotation amount;

[0011] The second image data and the wheel speed data of the vehicle are collected during driving, and the second image data, the wheel speed data and the rotation amount external parameter are processed to obtain the translation amount external parameter.

[0012] Optionally, collecting first image data of a vehicle while it is traveling includes:

[0013] S1, during the driving process of the vehicle, collecting a first image and a second image with different frame rates, and performing ORB feature extraction and matching on the first image and the second image to obtain a first point set, a second point set, and a matching set;

[0014] S2, calculating a basic matrix and restoring a pose using an epipolar constraint condition according to the first point set, the second point set, and the first matching set;

[0015] S3, repeat S1-S2, collect a set of motion postures for a preset number of times, and filter the straight-moving postures and turning postures.

[0016] Optionally, in step S1, extracting and matching ORB features on the first image and the second image includes:

[0017] Extract FAST corner points and calculate the BRIEF descriptors at the corner points; obtain the first matching set through the brute force matching algorithm.

[0018] Optionally, in step S2, the epipolar constraint is used to calculate the basic matrix and restore the pose, and the formula is as follows:

[0019] p jk T Fp ik =0 (k∈(1…n)) (1)

[0020] F=K -T t ji ∧R ji K -1 (2)

[0021] where p ik and p jk The point set P i and P j The kth pair of matching points in , K is the camera internal parameter matrix, R ji and t ji are the rotation matrix and translation vector in the camera coordinate system from the i-th frame to the j-th frame respectively.

[0022] Optionally, in step S3, collecting a preset number of motion gesture sets and screening straight-moving gestures and turning gestures includes:

[0023] S31, collect m motion postures, recorded as R ji , tji (i∈(0…m-1),j∈(1…m));

[0024] S32, select the vehicle straight-moving posture t cji , where R ji Approximately the unit matrix, collect the translation vector t at this time ji And normalized to a unit vector, recorded as:

[0025]

[0026] S33, select vehicle turning posture w cji Calculate R at this time ji The rotation axis w ji And normalized to a unit vector, recorded as:

[0027]

[0028] Optionally, the processing the first image data to obtain a rotation amount extrinsic parameter includes:

[0029] S4, collect and process a preset number of normalized straight-line postures t cji , turning posture w cji , get the first translation vector t c , the first rotation axis vector w c ;

[0030] S5, when the vehicle is moving straight, obtain the normalized second translation vector of the vehicle coordinate system and record it as t w =[0 1 0] T When the vehicle turns, the normalized second rotation axis vector of the vehicle coordinate system is obtained and recorded as w w =[0 0 1] T ;

[0031] S6, through the first translation vector t c , the first rotation axis vector w c , the second translation vector t w , the second rotation axis vector w w , calculate the external parameters of the rotation.

[0032] Optionally, in step S4, the collecting and processing of a preset number of normalized straight-ahead postures t cji , turning posture w cji , get the first translation vector t c , the first rotation axis vector w c ,include:

[0033] When a sufficient number of normalized straight-line poses t are collected cji , turning posture w cjiAfter that, outliers outside the range of mean ±3 times standard deviation are removed, the mean of the retained data is calculated, and the first translation vector t is obtained. c , the first rotation axis vector w c .

[0034] Optionally, in step S6, the rotation amount extrinsic parameter is calculated by the following formula, including:

[0035] R cw ·t w =t c (6)

[0036] R cw ·w w =w c (7)

[0037] The rotation external parameter R is calculated by combining formulas (6) and (7): cw .

[0038] Optionally, collecting second image data and wheel speed data of the vehicle while it is traveling, and processing the second image data, wheel speed data, and the rotation amount external parameter to obtain the translation amount external parameter includes:

[0039] S7, collecting and processing the third image and the fourth image whose interval is greater than the preset distance when the vehicle is traveling straight, and obtaining the third point set and the fourth point set;

[0040] S8, estimate the rotation matrix R from frame i to frame j in the vehicle coordinate system through the wheel speed sensor wji and the translation vector t wji , and according to the rotation matrix R wji and the translation vector t wji Calculating a second matching point set of the third image and the fourth image;

[0041] S9, obtaining a ground point pair set according to the second matching point set of the third image and the fourth image, and calculating a rotation matrix R′ from the third image frame rate to the fourth image frame rate in the vehicle coordinate system wji and the translation vector t′ wji ;

[0042] S10, translation vector t calculated by wheel speed sensor wji The translation vector t′ calculated modulo the second matching point set wji The ratio of the modules gives the scale s;

[0043] S11, repeat steps S7-S10, obtain multiple groups of scales s, and calculate the scale average value; obtain the translation external parameter through the installation position and the scale average value.

[0044] Optionally, in step S7, the collecting and processing of the third image and the fourth image separated by a distance greater than a preset distance when the vehicle is traveling straight, and obtaining the third point set and the fourth point set, includes:

[0045] In the third image and the fourth image, the FAST corner points of the ground area are extracted and the BRIEF descriptor is calculated to obtain the third point set and the fourth point set, which are denoted as P respectively. i and P j .

[0046] Optionally, the rotation matrix R from frame i to frame j in the vehicle coordinate system is estimated by the wheel speed sensor wji and the translation vector t wji ,include:

[0047]

[0048] Among them, l r and l l is the distance moved by the right rear wheel and the left rear wheel, tr is the wheelbase of the left and right wheels, θ is the angle of movement, P wji is the position of the vehicle at the fourth image frame rate relative to the third image frame rate.

[0049] Optionally, the rotation matrix R wji and the translation vector t wji Calculating the second matching point set of the third image and the fourth image, and extracting the second matching point set by the following formula, including:

[0050]

[0051] p′ jk =K·[R cw |t cw ]·P′ jk (13)

[0052] P′ ik =R wji ·P′ jk +t wji (14)

[0053]

[0054] Where R cw is the external parameter of the rotation; t cw is the translation extrinsic parameter, that is, the translation vector calculated using the installation position; at this time, the z component scale s of the camera installation position is set to an initial value of 1.0:

[0055]

[0056] At position p′ ikSet ROI nearby and select the fourth point set P j The points within the ROI are matched to obtain the second matching point set, denoted as (p ik , p jk )k∈1…n.

[0057] Optionally, in step S9, a ground point pair set is obtained according to the second matching point set of the third image and the fourth image, and a rotation matrix R′ from the third image frame rate to the fourth image frame rate in the vehicle coordinate system is calculated. wji and the translation vector t′ wji ,include:

[0058] According to the first matching point set (p ik , p jk )k∈1…n can get the ground point pair (P ik , P jk )k∈1…n;

[0059] Calculate the rotation matrix R′ from the third image frame rate to the fourth image frame rate in the vehicle coordinate system wji and the translation vector t′ wji The formula includes:

[0060]

[0061] P jk =[R′ wji |t′ wji ]·P ik (20).

[0062] Optionally, the translation vector t calculated by the wheel speed sensor wji The translation vector t′ calculated from the second matching point set wji The ratio of the modulus can be used to obtain the scale s, which is calculated using the following formula:

[0063] Optionally, in step S11, the translation extrinsic parameter is calculated by the following formula:

[0064] Among them, the installation position of the camera is (Pos′ x , Pos′ y , Pos′ z ), is the average value of multiple groups of scales s.

[0065] Compared with the prior art, the beneficial effects of the present application are as follows: the present application can collect multiple frames of visual information during the normal driving of the vehicle, and calibrate the camera extrinsic parameters with the help of wheel speed sensors. No professional operators and special sites are required, and the application range is wide; the present application re-derives the extrinsic parameter calculation formula, and does not rely on the diversified movements of multiple rotation axes, so that the vehicle can also be successfully calibrated during normal flat driving, greatly increasing the success rate. The present application does not need to rely on other reference sensors to give accurate vehicle postures. According to the characteristics of the wheel speed sensor, only its high-precision mileage information is used to assist in external point elimination and iterative optimization of camera height. The present application normalizes all vectors involved in the calculation before processing, effectively avoiding problems such as scale divergence in traditional multi-frame correlation scale normalization, and improving accuracy. The present application pre-eliminates most of the visual mismatch errors through wheel speed sensors, thereby improving the convergence speed and solution accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 The present invention is a flowchart of a method according to an embodiment of the present invention.

[0067] Figure 2 This is a flowchart of collecting first image data of a vehicle while it is traveling according to an embodiment of the present application.

[0068] Figure 3 This is a flow chart of obtaining the external parameters of the rotation amount according to an embodiment of the present application.

[0069] Figure 4 This is a flow chart of obtaining the translation extrinsic parameter according to an embodiment of the present application. DETAILED DESCRIPTION

[0070] The present application is further described below in conjunction with specific implementation methods.

[0071] The same or similar numbers in the drawings of the embodiments of the present application correspond to the same or similar parts; in the description of the present application, it should be understood that if the terms "upper", "lower", "left", "right", "top", "bottom", "inside", "outside", etc. indicate the orientation or position relationship, it is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as a limitation on this patent.

[0072] In addition, if there are terms such as "first" and "second", they are only used for descriptive purposes, mainly to distinguish different devices, elements or components (the specific types and structures may be the same or different), and are not used to indicate or imply the relative importance and quantity of the indicated devices, elements or components, and cannot be understood as indicating or implying relative importance.

[0073] Embodiment 1,

[0074] See also Figure 1-4 , the present application provides a method for calibrating the extrinsic parameters of a vehicle-mounted camera applicable to all scenes without mapping, the method comprising: 100, collecting the first image data of the vehicle while it is traveling, and processing the first image data to obtain the rotation extrinsic parameter; 200, collecting the second image data and wheel speed data of the vehicle while it is traveling, and processing the second image data, wheel speed data, and rotation extrinsic parameter to obtain the translation extrinsic parameter. In this embodiment, the present application can collect multiple frames of visual information during the normal driving of the vehicle, and calibrate the camera extrinsic parameters with the help of a wheel speed sensor. No professional operators and special sites are required, and the scope of application is wide; the present application re-derives the extrinsic parameter calculation formula, and does not rely on the diversified movement of multiple rotation axes, so that the vehicle can also be successfully calibrated during normal flat driving, greatly increasing the success rate. The present application does not need to rely on other reference sensors to give an accurate vehicle posture. In view of the characteristics of the wheel speed sensor, only its high-precision mileage information is used to assist in the elimination of external points and iterative optimization of the camera height. This application normalizes all vectors involved in the calculation before processing, effectively avoiding problems such as scale divergence in traditional multi-frame correlation scale normalization and improving accuracy. This application pre-eliminates most of the visual mismatch errors through wheel speed sensors, improving convergence speed and solution accuracy.

[0075] In some embodiments, collecting first image data of a vehicle while it is traveling includes:

[0076] S1, during vehicle driving, a first image and a second image with different frame rates are collected, and ORB feature extraction and matching are performed on the first image and the second image to obtain a first point set, a second point set, and a matching set; in step S1, ORB feature extraction and matching are performed on the first image and the second image, including: extracting FAST corner points and calculating BRIEF descriptors at the corner points; and obtaining a first matching set through a brute force matching algorithm.

[0077] S2, according to the first point set, the second point set, and the first matching set, the basic matrix is ​​calculated using the epipolar constraint condition and the pose is restored; In step S2, the basic matrix is ​​calculated using the epipolar constraint condition and the pose is restored, and the formula is as follows: jk T Fp ik =0(k∈(1…n))(1)F=K -T t ji ∧R ji K -1 (2); where p ik and p jk The point set P i and P jThe kth pair of matching points in , K is the camera internal parameter matrix, R ji and t ji are the rotation matrix and translation vector in the camera coordinate system from the i-th frame to the j-th frame respectively.

[0078] S3, repeat S1-S2, collect the preset number of motion posture sets, and filter the straight-moving posture and the turning posture. In step S3, collect the preset number of motion posture sets, and filter the straight-moving posture and the turning posture, including: S31, collect m motion postures, recorded as R ji , t ji (i∈(0…m-1),j∈(1…m)); S32, select the vehicle straight-moving posture t cji , where R cji Approximately the unit matrix, collect the translation vector t at this time cji And normalized to a unit vector, recorded as: S33, select vehicle turning posture w cji Calculate R at this time ji The rotation axis w ji And normalized to a unit vector, denoted as

[0079]

[0080] In some embodiments, processing the first image data to obtain the rotation amount extrinsic parameter includes:

[0081] S4, collect and process a preset number of normalized straight-line postures t cji , turning posture w cji , get the first translation vector t c , the first rotation axis vector w c In step S4, a preset number of normalized straight-line postures t are collected and processed cji , turning posture w cji , get the first translation vector t c , the first rotation axis vector w c , including: when a sufficient number of normalized straight-line postures t are collected cji , turning posture w cji After that, outliers outside the range of mean ±3 times standard deviation are removed, the mean of the retained data is calculated, and the first translation vector t is obtained. c , the first rotation axis vector w c .

[0082] S5, when the vehicle is moving straight, obtain the normalized second translation vector of the vehicle coordinate system and record it as t w =[0 1 0] T When the vehicle turns, the normalized second rotation axis vector of the vehicle coordinate system is obtained and recorded as w w=[0 0 1] T ;

[0083] S6, through the first translation vector t c , the first rotation axis vector w c , the second translation vector t w , the second rotation axis vector w w , calculate the rotation amount external parameter. In step S6, the rotation amount external parameter is calculated by the following formula, including: R cw ·t w =t c (6); R cw ·w w =w c (7); Calculate the rotation external parameter R by combining formulas (6) and (7) cw .

[0084] In some embodiments, collecting the second image data and wheel speed data of the vehicle while it is traveling, and processing the second image data, wheel speed data, and rotation extrinsic parameter to obtain the translation extrinsic parameter includes:

[0085] S7, collecting and processing the third image and the fourth image whose interval is greater than the preset distance when the vehicle is moving straight, and obtaining the third point set and the fourth point set; In step S7, collecting and processing the third image and the fourth image whose interval is greater than the preset distance when the vehicle is moving straight, and obtaining the third point set and the fourth point set, including extracting the FAST corner points of the ground area in the third image and the fourth image respectively and calculating the BRIEF descriptor, obtaining the third point set and the fourth point set, respectively denoted as P i and P j

[0086] S8, estimate the rotation matrix R from frame i to frame j in the vehicle coordinate system through the wheel speed sensor wji and the translation vector t wji , and according to the rotation matrix R wji and the translation vector t wji Calculate the second matching point set of the third image and the fourth image; in step S8, estimate the rotation matrix R from frame i to frame j in the vehicle coordinate system through the wheel speed sensor wji and the translation vector t wji ,include:

[0087]

[0088] Among them, l r and l l is the distance moved by the right rear wheel and the left rear wheel, tr is the wheelbase of the left and right wheels, θ is the angle of movement, P wji is the position of the vehicle at the fourth image frame rate relative to the third image frame rate.

[0089] According to the rotation matrix R wji and the translation vector t wji Calculate the second matching point set of the third image and the fourth image, and extract the second matching point set by the following formula, including:

[0090]

[0091] p′ jk =K·[R cw |t cw ]·P′ jk (13)

[0092] P′ ik =R wji ·P′ jk +t wji …(14)

[0093]

[0094] Among them, R cw is the external parameter of the rotation, t cw is the translation vector calculated using the installation position. At this time, the z-component scale s of the camera installation position is set to an initial value of 1.0:

[0095]

[0096] At position p′ ik Set ROI nearby and select the fourth point set P j The points within the ROI are matched to obtain the second matching point set, denoted as (p ik , p jk ),k∈(1…n).

[0097] S9, obtaining a ground point pair set according to the second matching point set of the third image and the fourth image, and calculating a rotation matrix R′ from the third image frame rate to the fourth image frame rate in the vehicle coordinate system wji and the translation vector t′ wji In step S9, a ground point pair set is obtained based on the second matching point set of the third image and the fourth image, and a rotation matrix R′ from the third image frame rate to the fourth image frame rate in the vehicle coordinate system is calculated. wji and the translation vector t′ wji ,include:

[0098] According to the first matching point set (p ik , p jk ), k∈(1…n) can get the ground point pair (P ik , P jk ),k∈(1…n);

[0099] Calculate the rotation matrix R′ from the third image frame rate to the fourth image frame rate in the vehicle coordinate system wji and the translation vector t′ wji The formula includes:

[0100]

[0101] P jk =[R′ wji |t′ wji ]·P ik (20).

[0102] S10, translation vector t calculated by wheel speed sensor wji The translation vector t′ calculated modulo the second matching point set wji The ratio of the modulus can be used to obtain the scale s; in step S10, the translation vector t is calculated by the wheel speed sensor wji The translation vector t′ calculated from the second matching point set wji The ratio of the modulus can be used to obtain the scale s, which is calculated using the following formula:

[0103] S11, repeat steps S7-S10 to obtain multiple groups of scales s, calculate the scale average value; obtain the translation external parameter through the installation position and the scale average value. In step S11, the translation external parameter is calculated by the following formula: Among them, the installation position of the camera is (Pos′ x , Pos′ y , Pos′ z ), is the average value of multiple groups of scales s.

[0104] Among them, the rotation extrinsic parameters and the translation extrinsic parameters are the extrinsic parameters for camera calibration in this application.

[0105] Embodiment 2,

[0106] The present application provides a method for calibrating the external parameters of a vehicle-mounted camera applicable to all scenes without mapping, and the specific steps are as follows:

[0107] Part I: Information Collection

[0108] During the vehicle driving, two frames of images i and j are collected, denoted as Img i and Img j ; for Img i and Img j Perform ORB feature extraction and matching to obtain the point set P i and P jThe matching relationship is divided into the following steps: extract FAST corner points; calculate the BRIEF descriptor at the corner points; Brute-Froce Matcher to obtain n pairs of matching points, denoted as (p ik , p jk )k∈1…n. Use the epipolar constraint to calculate the basic matrix F and restore the pose:

[0109] p jk T Fp ik =0,(k∈(1…n))

[0110] F=K -T t ji ∧R ji K -1

[0111] where p ik and p jk The point set P i and P j The kth pair of matching points in , K is the camera internal parameter matrix, R cji and t cji are the rotation matrix and translation vector in the camera coordinate system from the i-th frame to the j-th frame respectively.

[0112] Repeat the above steps to collect m motion postures, recorded as R ji , t ji ,(i∈(0…m-1),j∈(1…m)). Select the vehicle’s straight-moving posture t cji (R ji Approximately the unit matrix), collect the translation vector t at this time ji And normalized to a unit vector, recorded as: Select the vehicle's turning posture w cji Calculate R at this time cji The rotation axis w ji And normalized to a unit vector, recorded as:

[0113] The second part is to solve the camera external parameter rotation - rotation matrix R cw

[0114] Since we can directly solve R cw The solution can only be obtained by rotating the vehicle along at least two non-parallel rotation axes. Therefore, by screening the translation vectors or rotation axes in different motion processes and normalizing them as matching point pairs for calculating the rotation relationship between the two coordinate systems, the purpose of solving the external parameter rotation quantity when the vehicle moves in a single plane can be achieved.

[0115] When a sufficient number of normalized tcji and w cji After that, outliers outside the range of mean ± 3 times standard deviation are removed, and the mean of the retained data is calculated and recorded as t c and w c , which represents the translation vector and rotation axis vector of the vehicle's straight and turning motion in the camera coordinate system.

[0116] When the vehicle is moving straight, the normalized translation vector of the vehicle coordinate system is denoted as t w =[0 1 0] T ;

[0117] When the vehicle turns, the normalized rotation axis vector of the vehicle coordinate system is recorded as w w =[0 0 1] T ;

[0118] By R cw ·t w =t c and R cw ·w w =w c Solving the problem yields the unique rotation matrix R cw .

[0119] The third part is to solve the camera's external parameter translation - the translation vector t cw

[0120] The installation position of the vehicle-mounted camera is recorded as Pos = [Pos x Pos y Pos z ] T , where the installation position Pos′ in the mechanical structure is used x , Pos′ y , Pos′ z is the initial value. x , Pos′ y The installation tolerance is generally no more than 5mm, so it can be assigned to Pos x ,Pos y .Pos′ z Affected by vehicle load and tire pressure, the range of variation is large, so it cannot be directly assigned to Pos z , the following steps are required to solve it.

[0121] When the vehicle is traveling straight, select the two frames before and after with a driving distance > 5m, and record them as Img i and Img j ;

[0122] In Img i and Img jIn the above example, the FAST corner points of the ground area are extracted and the BRIEF descriptor is calculated, which is recorded as the point set P i and P j ;

[0123] The rotation matrix R from frame i to frame j in the vehicle coordinate system is estimated by the wheel speed sensor wji and the translation vector t wji ;

[0124]

[0125] Among them l r and l l is the distance moved by the right rear wheel and the left rear wheel, tr is the wheelbase of the left and right wheels, θ is the angle of movement, P wji is the position of the vehicle at frame j relative to frame i.

[0126] The vehicle pose R estimated by wheel speed wji , t wji Calculate the Img extracted from 2 i The characteristic point p ik ∈P i In Img j The position p′ in jk ,

[0127]

[0128] p′ jk =K·[R cw |t cw ]·P′ jk

[0129] P′ ik =R wji ·P′ jk +t wji

[0130]

[0131]

[0132] Where R cw is the camera extrinsic rotation matrix obtained in step 2, t cw is the translation vector calculated using the installation position, where the z-component scale s of the installation position is set to an initial value of 1.0:

[0133]

[0134] At position p′ ik Set ROI nearby and select point set P j Match the points within the ROI range to obtain n pairs of matching points, denoted as (pik , p jk ),k∈(1…n);

[0135] According to the image point pair (p ik , p jk ), k∈(1…n) can get the ground point pair (P ik , P jk ), k∈(1…n), and the rotation matrix R′ from frame i to frame j in the vehicle coordinate system can be calculated wji and the translation vector t′ wji :

[0136]

[0137] P jk =[R′ wji |t′ wji ]·P ik

[0138] The translation vector t calculated by the wheel speed sensor wji and the translation vector t′ calculated from the matching point set wji The ratio of the moduli gives the scale s:

[0139]

[0140] Repeat the above steps to get multiple groups of s and find the mean The translation vector can be solved by the installation position and is expressed as:

[0141]

[0142] Embodiment 3:

[0143] The present application also provides a vehicle-mounted system, wherein the vehicle-mounted system is provided with a vehicle-mounted integrated terminal that can execute a vehicle-mounted camera extrinsic parameter calibration method applicable to the full scene without mapping in Example 1 or Example 2. The present application can collect multiple frames of visual information during normal vehicle driving, and calibrate the camera extrinsic parameters with the help of wheel speed sensors. No professional operators and special sites are required, and the scope of application is wide; the present application re-derives the extrinsic parameter calculation formula, does not rely on the diversified movement of multiple rotation axes, so that the vehicle can also be successfully calibrated during normal flat driving, greatly increasing the success rate. The present application does not need to rely on other reference sensors to give accurate vehicle postures. According to the characteristics of the wheel speed sensor, only its high-precision mileage information is used to assist in external point elimination and iterative optimization of camera height. The present application normalizes all vectors involved in the calculation before processing, effectively avoiding problems such as scale divergence in traditional multi-frame correlation scale normalization, and improving accuracy. The present application pre-eliminates most of the errors of visual mismatching through wheel speed sensors, thereby improving convergence speed and solution accuracy.

[0144] Obviously, the above embodiments of the present application are merely examples for clearly illustrating the present application, and are not intended to limit the implementation methods of the present application. For those of ordinary skill in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all implementation methods here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present application should be included in the protection scope of the claims of the present application.

Claims

1. A method for extrinsic calibration of vehicle-mounted cameras applicable to all scenes without mapping, characterized in that: The method comprises: Collecting first image data of the vehicle while it is traveling, and processing the first image data to obtain an external parameter of the rotation amount; The second image data and wheel speed data of the vehicle are collected during driving, and the second image data, wheel speed data and the rotation amount external parameter are processed to obtain the translation amount external parameter, specifically including: S7, collecting and processing the third image and the fourth image whose interval is greater than the preset distance when the vehicle is moving straight, and obtaining the third point set and the fourth point set, S8, estimate the rotation matrix R from frame i to frame j in the vehicle coordinate system through the wheel speed sensor wji and the translation vector t wji , and according to the rotation matrix R wji and the translation vector t wji Calculating a second matching point set of the third image and the fourth image, S9, obtaining a ground point pair set according to the second matching point set of the third image and the fourth image, and calculating a rotation matrix R′ from the third image frame rate to the fourth image frame rate in the vehicle coordinate system wji and the translation vector t′ wji ; S10, translation vector t calculated by wheel speed sensor wji The translation vector t′ calculated modulo the second matching point set wji The ratio of the moduli gives the scale s, S11, repeat steps S7-S10, obtain multiple groups of scales s, and calculate the scale average value; obtain the translation external parameter through the installation position and the scale average value.

2. According to the method for extrinsic calibration of vehicle-mounted cameras applicable to all scenes without mapping construction as claimed in claim 1, it is characterized in that: The collecting of first image data of a vehicle while it is traveling comprises: S1, during the driving process of the vehicle, collecting a first image and a second image with different frame rates, and performing ORB feature extraction and matching on the first image and the second image to obtain a first point set, a second point set, and a matching set; S2, calculating a basic matrix and restoring a pose using an epipolar constraint condition according to the first point set, the second point set, and the first matching set; S3, repeat S1-S2, collect a set of motion postures for a preset number of times, and filter the straight-moving postures and turning postures.

3. The method for calibrating extrinsic parameters of a vehicle-mounted camera applicable to all scenes without mapping according to claim 2, characterized in that: In the step S1, the ORB feature extraction and matching of the first image and the second image includes: Extract FAST corner points and calculate the BRIEF descriptors at the corner points; obtain the first matching set through the brute force matching algorithm.

4. The method for extrinsic calibration of a vehicle-mounted camera applicable to all scenes without mapping according to claim 2, characterized in that: In step S2, the epipolar constraint is used to calculate the basic matrix and restore the pose, and the formula is as follows: p jk T Fp ik =0,(k∈(1…n))(1) F=K -T t ji ∧ R ji K -1 (2) where p ik and p jk The point set P i and P j The kth pair of matching points in the matrix, F represents the basic matrix, K represents the camera internal parameter matrix, K -1 represents the inverse matrix of the camera intrinsic parameter matrix, K -T Represents the inverse transposed matrix of the camera intrinsic parameter matrix, R ji and t ji are the rotation matrix and translation vector in the camera coordinate system from the i-th frame to the j-th frame respectively.

5. The method for calibrating extrinsic parameters of a vehicle-mounted camera applicable to all scenes without mapping according to claim 2, characterized in that: In step S3, the step of collecting a preset number of motion gesture sets and screening the straight-moving gestures and turning gestures includes: S31, collect m motion postures, denoted as R ji , t ji ,(i∈(0…m-1),j∈(1…m)); S32, selecting the straight-ahead posture t of the vehicle cji , where R ji is the unit matrix, and the translation vector t is collected at this time ji And normalized to a unit vector, recorded as: S33, selecting the turning posture w of the vehicle cji Calculate R at this time ji The rotation axis w ji And normalized to a unit vector, recorded as: Among them, r 11 ~r 33 For R ji The matrix elements of .

6. The method for extrinsic calibration of a vehicle-mounted camera applicable to all scenes without mapping according to claim 2, characterized in that: The step of processing the first image data to obtain a rotation amount external parameter includes: S4, collect and process a preset number of normalized straight-line postures t cji , turning posture w cji , get the first translation vector t c , the first rotation axis vector w c ; S5, when the vehicle is moving straight, obtain the normalized second translation vector of the vehicle coordinate system and record it as t w =[0 1 0] T When the vehicle turns, the normalized second rotation axis vector of the vehicle coordinate system is obtained and recorded as w w =[0 0 1] T ; S6, through the first translation vector t c , the first rotation axis vector w c , the second translation vector t w , the second rotation axis vector w w , calculate the external parameters of the rotation.

7. The method for extrinsic calibration of a vehicle-mounted camera applicable to all scenes without mapping according to claim 6, characterized in that: In step S4, a preset number of normalized straight-line postures t are collected and processed. cji , turning posture w cji , get the first translation vector t c , the first rotation axis vector w c ,include: When a preset number of normalized straight-line postures t are collected cji , turning posture w cji After that, outliers outside the range of mean ±3 times standard deviation are removed, the mean of the retained data is calculated, and the first translation vector t is obtained. c , the first rotation axis vector w c .

8. The method for calibrating extrinsic parameters of a vehicle-mounted camera applicable to all scenes without mapping according to claim 6, characterized in that: In step S6, the rotation amount external parameter is calculated by the following formula, including: R cw ·t w =t c (6) R cw ·w w =w c (7) The rotation external parameter R is calculated by combining formulas (6) and (7): cw .

9. The method for calibrating extrinsic parameters of a vehicle-mounted camera applicable to all scenes without mapping according to claim 4, characterized in that: In step S7, the collecting and processing of the third image and the fourth image separated by a distance greater than a preset distance when the vehicle is traveling straight, and obtaining the third point set and the fourth point set, includes: In the third image and the fourth image, the FAST corner points of the ground area are extracted and the BRIEF descriptor is calculated to obtain the third point set and the fourth point set, which are denoted as P respectively. i and P j .

10. The method for calibrating extrinsic parameters of a vehicle-mounted camera applicable to all scenes without mapping according to claim 9, characterized in that: The rotation matrix R from frame i to frame j in the vehicle coordinate system is estimated by the wheel speed sensor wji and the translation vector t wji , include: Among them, l r and l l is the distance moved by the right rear wheel and the left rear wheel, tr is the wheelbase of the left and right wheels, θ is the angle of movement, P wji is the position of the vehicle at the fourth image frame rate relative to the third image frame rate.

11. The method for calibrating extrinsic parameters of a vehicle-mounted camera applicable to all scenes without mapping according to claim 10, characterized in that: According to the rotation matrix R wji and the translation vector t wji Calculating the second matching point set of the third image and the fourth image, and extracting the second matching point set by the following formula, including: p ′ jk =K·[R cw |t cw ]·P ′ jk (13) P ′ ik =R wji ·P′ jk +t wji (14) Where R cw is the external parameter of the rotation; t cw is the translation extrinsic parameter, that is, the translation vector calculated using the installation position; at this time, the z component scale s of the camera installation position is set to an initial value of 1.0: At position p′ ik Set ROI nearby and select the fourth point set P j The points within the ROI are matched to obtain the second matching point set, denoted as (p ik , p jk )k∈1…n, Pos is the camera installation position coordinate, its coordinate is (Pos ′ x ,Pos ′ y ,Pos ′ z ).

12. The method for calibrating extrinsic parameters of a vehicle-mounted camera applicable to all scenes without mapping according to claim 11, characterized in that: In step S9, a ground point pair set is obtained according to the second matching point set of the third image and the fourth image, and a rotation matrix R′ from the third image frame rate to the fourth image frame rate in the vehicle coordinate system is calculated. wji and the translation vector t′ wji ,include: According to the first matching point set (p ik , p jk ), k∈(1…n) to get the ground point pair (P ik , P jk ),k∈(1…n); Calculate the rotation matrix R′ from the third image frame rate to the fourth image frame rate in the vehicle coordinate system wji and the translation vector t′ wji The formula includes: P jk =[R′ wji |t′ wji ]·P ik (20)。 13. The method for calibrating extrinsic parameters of a vehicle-mounted camera applicable to all scenes without mapping according to claim 12, characterized in that: The translation vector t calculated by the wheel speed sensor wji The translation vector t′ calculated modulo the second matching point set wji The ratio of the modulus to obtain the scale s is calculated by the following formula:

14. The method for extrinsic calibration of a vehicle-mounted camera applicable to all scenes without mapping according to claim 13, characterized in that: In step S11, the translation extrinsic parameter is calculated by the following formula: Among them, the installation position of the camera is (Pos ′ x ,Pos ′ y ,Pos ′ z ), is the average value of multiple groups of scales s.

Images