On-board camera online calibration method, device and computer-readable storage medium

Abstract

The embodiment of the present invention provides a method, device and computer-readable storage medium for online calibration of a vehicle-mounted camera, the method comprising: acquiring a real-time image frame; calculating the observed noise variance and predicted noise variance of the image frame based on the wheel speed pulse of the motor vehicle, and calculating the rotation matrix and translation matrix of the image frame; extracting the BEBLID feature points in each image frame, obtaining the descriptor matching feature points of the BEBLID feature points, and extracting a pair of matching point pairs from the descriptor matching feature points based on the principle of minimizing the Hamming distance; obtaining the three-dimensional coordinate points corresponding to the matching point pairs in the normalized three-dimensional space, and obtaining the key point clustering plane; constructing a distance loss function from the three-dimensional coordinate points to the key point clustering plane, solving the minimum distance loss function, and calculating the lens intrinsic parameters of the vehicle-mounted camera based on the minimum distance loss function. This embodiment can effectively improve the accuracy of the lens intrinsic parameter calibration of the vehicle-mounted camera.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of camera parameter calibration, and in particular to an on-line calibration method, device and computer-readable storage medium for a vehicle-mounted camera. Background Art

[0002] Before a vehicle-mounted camera installed on a motor vehicle can work normally, it is usually necessary to accurately calibrate the lens internal parameters of the vehicle-mounted camera to ensure the accuracy of the image captured by the vehicle-mounted camera during actual work, so as to facilitate the stable and effective operation of the vehicle-mounted system using the vehicle-mounted camera (for example: vehicle-mounted positioning and navigation system, vehicle-mounted visual measurement system, etc.).

[0003] An existing on-board camera online calibration method is to use the feature point matching principle based on SLAM (simultaneous localization and mapping). However, the above calibration method requires the pre-setting of specific markers, and the size and spatial relationship of the markers need to be known, and the calibration conditions are complex. In addition, the online calibration method based on SLAM requires image matching of key points in the image. However, image distortion caused by the lens of the on-board camera (for example: fisheye lens), changes in the ambient brightness of the image taken by the on-board camera, and nonlinear movement of the motor vehicle can all lead to key point matching errors, which ultimately results in low accuracy of the calculated lens intrinsic parameters of the on-board camera. Summary of the invention

[0004] The technical problem to be solved by the embodiments of the present invention is to provide an on-board camera online calibration method, which can effectively improve the accuracy of the lens intrinsic parameter calibration of the on-board camera.

[0005] A further technical problem to be solved by the embodiments of the present invention is to provide an on-board camera online calibration device, which can effectively improve the accuracy of the lens intrinsic parameter calibration of the on-board camera.

[0006] A further technical problem to be solved by an embodiment of the present invention is to provide a computer-readable storage medium that can effectively improve the accuracy of lens intrinsic parameter calibration of a vehicle-mounted camera.

[0007] In order to solve the above technical problems, the embodiment of the present invention first provides the following technical solution: an online calibration method for a vehicle-mounted camera, comprising the following steps:

[0008] Obtain real-time image frames from the original video images captured by the vehicle-mounted camera;

[0009] Calculating the observed noise variance and predicted noise variance of each image frame based on the wheel speed pulse of the motor vehicle, and obtaining the rotation matrix and translation matrix of each image frame based on the observed noise variance and predicted noise variance;

[0010] Extracting BEBLID feature points in each image frame in turn, and then obtaining descriptor matching feature points of each BEBLID feature point, and extracting a pair of matching point pairs from each descriptor matching feature point in the image frame and the previous frame based on the principle of minimizing the Hamming distance starting from the second image frame;

[0011] triangulate the matching point pairs based on the rotation matrix and the translation matrix to obtain the three-dimensional coordinate points corresponding to each matching point pair in the normalized three-dimensional space, cluster the three-dimensional coordinate points located in the same space plane to obtain a key point clustering plane; and

[0012] Based on the distance principle from a spatial point to a spatial plane, a distance loss function from a three-dimensional coordinate point to the key point clustering plane is constructed. Based on a predetermined nonlinear optimization algorithm model, the algebraic expression of the distance loss function is iteratively optimized to solve the minimum distance loss function. Then, based on the minimum distance loss function and the ideal imaging model of the lens of the vehicle camera, the lens intrinsic parameters of the vehicle camera are calculated.

[0013] Furthermore, starting from the second image frame, extracting a pair of matching point pairs from the image frame and each descriptor matching feature point in the previous frame based on the Hamming distance minimization principle specifically includes:

[0014] Calculating the Hamming distance between each of the descriptor matching feature points in the image frame and the previous frame;

[0015] Determine the minimum value of the Hamming distance of each of the descriptor matching feature points; and

[0016] When the minimum value of the Hamming distance is less than a predetermined Hamming distance threshold, a pair of the descriptor matching feature points corresponding to the minimum value of the Hamming distance is determined as a matching point pair of two adjacent image frames. Further, clustering the three-dimensional coordinate points located in the same spatial plane to obtain a key point clustering plane specifically includes:

[0017] Arbitrarily select three non-collinear three-dimensional coordinate points to construct an initial space plane;

[0018] Calculating the point-to-surface distance between each of the three-dimensional coordinate points and the initial spatial plane;

[0019] When the point-to-plane distance corresponding to the three-dimensional coordinate point is less than a predetermined point-to-plane distance, classifying the three-dimensional coordinate point as a valid coordinate point; and

[0020] The actual number of valid coordinate points corresponding to the same initial spatial plane is counted, and when the actual number is greater than a predetermined number, the valid coordinate points are stopped from being classified and the initial spatial plane is determined as a key point clustering plane.

[0021] Furthermore, the calculation of the observed noise variance and the predicted noise variance of each image frame based on the wheel speed pulse of the motor vehicle specifically includes:

[0022] Calculating the observed three-dimensional motion parameters of the motor vehicle within the frame difference time length corresponding to the image frame in sequence according to the motor vehicle wheel speed pulses; performing Kalman filtering on the observed three-dimensional motion parameters of the image frame to obtain optimized three-dimensional motion parameters;

[0023] Combining the actual three-dimensional motion parameters of the image frame and the optimized three-dimensional motion parameters and calculating based on the standard normal distribution principle to obtain the observed noise variance corresponding to each image frame;

[0024] estimating predicted 3D motion parameters of the image frame based on the optimized 3D motion parameters of a previous frame of the image frame and based on a rigid body motion assumption of the vehicle; and

[0025] The predicted three-dimensional motion parameter of the image frame and the optimized three-dimensional motion parameter are combined and predicted based on the standard normal distribution principle to obtain the predicted noise variance.

[0026] Furthermore, the calculation of the rotation matrix and the translation matrix of each image frame based on the observed noise variance and the predicted noise variance specifically includes:

[0027] The Kalman fusion motion vector of the image frame is calculated based on the observed noise variance and the predicted noise variance; and the rotation matrix and the translation matrix of the image frame are calculated based on the Kalman fusion motion vector.

[0028] Furthermore, after obtaining the real-time image frame from the original video image captured by the vehicle-mounted camera, the image frame is preprocessed, and the preprocessing at least includes image noise reduction and histogram equalization.

[0029] Furthermore, the histogram equalization is adaptive histogram equalization.

[0030] Furthermore, the predetermined nonlinear optimization algorithm model is a Levenberg-Marquardt algorithm model.

[0031] On the other hand, in order to solve the above-mentioned technical problems, an embodiment of the present invention first provides the following technical solutions: an online calibration device for a vehicle-mounted camera, which is respectively connected to a vehicle-mounted camera for shooting video images around a motor vehicle and providing original video images, a wheel pulse sensor for detecting wheel pulses of a motor vehicle and providing wheel speed pulses of a motor vehicle, and a parameter storage device for storing lens intrinsic parameters of the vehicle-mounted camera. The online calibration device for the vehicle-mounted camera includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the online calibration method for the vehicle-mounted camera as described above is implemented.

[0032] On the other hand, in order to solve the above-mentioned further technical problems, an embodiment of the present invention further provides the following technical solutions: a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the vehicle-mounted camera online calibration method as described above.

[0033] After adopting the above technical scheme, the embodiment of the present invention has at least the following beneficial effects: the embodiment of the present invention extracts image frames from the original video images, calculates the observed noise variance and the predicted noise variance of each image frame based on the wheel speed pulse of the motor vehicle, further estimates the movement of the motor vehicle, calculates the rotation matrix and the translation matrix of each image frame, and estimates the movement of the motor vehicle by using the three-dimensional spatial movement of the motor vehicle, which can effectively simplify the processing steps, and then extracts a pair of matching point pairs from the image frame and each descriptor matching feature point in the previous frame based on the principle of Hamming distance minimization. The matching process is simple, and the matching failure caused by the lens distortion of the vehicle-mounted camera is avoided. The key point clustering plane is further determined by classifying the coplanar coordinate points, and finally the distance loss function from the three-dimensional coordinate point to the key point clustering plane is constructed based on the distance principle from the spatial point to the spatial plane, and then the minimum distance loss function is solved by iterative optimization, and then the lens intrinsic parameters of the vehicle-mounted camera are calculated based on the minimum distance loss function and the ideal imaging model of the lens of the vehicle-mounted camera. The best lens intrinsic parameters of the vehicle-mounted camera can be calculated, and the accuracy of the lens intrinsic parameter calibration of the vehicle-mounted camera is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 The present invention is a flowchart of the steps of an optional embodiment of the on-line calibration method for a vehicle-mounted camera.

[0035] Figure 2 This is a flowchart of the steps of extracting matching point pairs in an optional embodiment of the online calibration method for a vehicle-mounted camera of the present invention.

[0036] Figure 3This is a flow chart of the steps of calculating and obtaining a key point clustering plane in an optional embodiment of the on-line calibration method for a vehicle-mounted camera of the present invention.

[0037] Figure 4 This is a flow chart of the steps of calculating and obtaining the observed noise variance and the predicted noise variance in an optional embodiment of the on-line calibration method for a vehicle-mounted camera of the present invention.

[0038] Figure 5 This is a flow chart of the steps of calculating and obtaining the rotation matrix and translation matrix of each image frame in an optional embodiment of the online calibration method for a vehicle-mounted camera of the present invention.

[0039] Figure 6 The schematic diagram is a functional block diagram of an optional embodiment of the on-board camera online calibration device of the present invention.

[0040] Figure 7 This is a functional module diagram of an optional embodiment of the on-board camera online calibration device of the present invention. DETAILED DESCRIPTION

[0041] The present application is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following illustrative embodiments and descriptions are only used to explain the present invention and are not intended to limit the present invention. In addition, the embodiments and features in the embodiments of the present application may be combined with each other without conflict.

[0042] like Figure 1 As shown, an optional embodiment of the present invention provides an on-board camera online calibration method, comprising the following steps:

[0043] S1: Obtain real-time image frames from the original video image captured by the vehicle-mounted camera 1;

[0044] S2: calculating the observed noise variance and predicted noise variance of each image frame based on the wheel speed pulse of the motor vehicle, and obtaining the rotation matrix and translation matrix of each image frame based on the observed noise variance and predicted noise variance;

[0045] S3: extracting BEBLID (Boosted Efficient Binary Local Image Descriptor) feature points in each image frame in turn, and then obtaining descriptor matching feature points of each BEBLID feature point, and extracting a pair of matching point pairs from each descriptor matching feature point in the current image frame and the previous image frame based on the principle of minimizing the Hamming distance starting from the second image frame;

[0046] S4: triangulating the matching point pairs based on the rotation matrix and the translation matrix to obtain the three-dimensional coordinate points corresponding to each matching point pair in the normalized three-dimensional space, and clustering the three-dimensional coordinate points located in the same space plane to obtain a key point clustering plane; and

[0047] S5: Based on the distance principle from a spatial point to a spatial plane, a distance loss function from a three-dimensional coordinate point to the key point clustering plane is constructed; based on a predetermined nonlinear optimization algorithm model, an algebraic expression of the distance loss function is iteratively optimized to solve the minimum distance loss function; and then based on the minimum distance loss function and the ideal imaging model of the lens of the vehicle-mounted camera 1, the lens intrinsic parameters of the vehicle-mounted camera 1 are calculated.

[0048] The embodiment of the present invention extracts image frames from the original video images, calculates the observed noise variance and the predicted noise variance of each image frame based on the wheel speed pulse of the motor vehicle, further estimates the movement of the motor vehicle, calculates the rotation matrix and the translation matrix of each image frame, and estimates the movement of the motor vehicle by using the three-dimensional spatial movement of the motor vehicle, which can effectively simplify the processing steps, and then extracts a pair of matching point pairs from each descriptor matching feature point in two adjacent image frames based on the principle of Hamming distance minimization. The matching process is simple, and the matching failure caused by the lens distortion of the vehicle-mounted camera 1 is avoided. The key point clustering plane is further determined by classifying the coplanar coordinate points, and finally the distance loss function from the three-dimensional coordinate point to the key point clustering plane is constructed based on the distance principle from the spatial point to the spatial plane, and then the minimum distance loss function is solved by iterative optimization, and then the lens intrinsic parameters of the vehicle-mounted camera 1 are calculated based on the minimum distance loss function and the ideal imaging model of the lens of the vehicle-mounted camera 1. The best lens intrinsic parameters of the vehicle-mounted camera 1 can be calculated, and the accuracy of the lens intrinsic parameter calibration of the vehicle-mounted camera is improved.

[0049] In an optional embodiment of the present invention, Figure 2 As shown, starting from the second image frame, extracting a pair of matching point pairs from the image frame and each descriptor matching feature point in the previous frame based on the principle of minimizing the Hamming distance specifically includes:

[0050] S31: Calculating the Hamming distance between each of the descriptor matching feature points in the image frame and the previous frame;

[0051] S32: Determine the minimum value of the Hamming distance of each of the descriptor matching feature points; and

[0052] S33: When the minimum value of the Hamming distance is less than a predetermined Hamming distance threshold, a pair of the descriptor matching feature points corresponding to the minimum value of the Hamming distance is determined as a matching point pair of two adjacent image frames.

[0053] In this embodiment, by calculating the Hamming distance between each of the descriptor matching feature points, the Hamming distance of the descriptor matching feature points can effectively measure the similarity of the descriptor matching feature points. By determining the minimum value of the Hamming distance and comparing it with a predetermined Hamming distance threshold, matching errors can be avoided, and the matching point pairs of two adjacent image frames can be accurately obtained, which is convenient for calculation.

[0054] In another optional embodiment of the present invention, Figure 3 As shown, clustering the three-dimensional coordinate points located in the same spatial plane to obtain the key point clustering plane specifically includes:

[0055] S41: arbitrarily selecting three non-collinear three-dimensional coordinate points to construct an initial space plane;

[0056] S42: Calculating the point-to-plane distance between each of the three-dimensional coordinate points and the initial spatial plane;

[0057] S43: when the point-to-plane distance corresponding to the three-dimensional coordinate point is less than a predetermined point-to-plane distance, classifying the three-dimensional coordinate point as a valid coordinate point; and

[0058] S44: Counting the actual number of valid coordinate points corresponding to the same initial spatial plane, and when the actual number is greater than a predetermined number, stopping the classification of the valid coordinate points and determining the initial spatial plane as a key point clustering plane.

[0059] In this embodiment, the initial spatial plane is constructed by first selecting three non-collinear three-dimensional coordinate points, and the distance from the spatial point to the spatial plane is calculated. Due to the inevitable existence of calculation errors, the point-to-plane distance is judged to be the same as the predetermined point-to-plane distance, rather than judging the point-to-plane distance to be zero, so as to avoid the situation where the three-dimensional coordinate point cannot be classified as a valid coordinate point. Finally, by counting the actual number of valid coordinate points corresponding to the same initial spatial plane, the key point clustering plane is finally determined. The calculation steps are relatively simple and the data processing efficiency is high.

[0060] In another optional embodiment of the present invention, Figure 4 As shown, the calculation of the observed noise variance and predicted noise variance of each image frame based on the wheel speed pulse of the motor vehicle specifically includes:

[0061] S21: Calculating and obtaining the observed three-dimensional motion parameters of the motor vehicle within the frame difference time length corresponding to the image frames according to the wheel speed pulses of the motor vehicle;

[0062] S22: performing Kalman filtering on the observed three-dimensional motion parameters of the image frame to obtain optimized three-dimensional motion parameters;

[0063] S23: combining the actual 3D motion parameters of the image frame and the optimized 3D motion parameters and calculating based on the standard normal distribution principle to obtain the observed noise variance corresponding to each image frame;

[0064] S24: estimating predicted three-dimensional motion parameters of the image frame according to the optimized three-dimensional motion parameters of the previous frame of the image frame and based on the rigid body motion assumption of the motor vehicle; and

[0065] S25: Combining the predicted 3D motion parameter of the image frame with the optimized 3D motion parameter and predicting based on the standard normal distribution principle to obtain the predicted noise variance.

[0066] In this embodiment, the observed three-dimensional motion parameters of the image frame are first calculated based on the wheel speed pulse of the motor vehicle, and then Kalman filtering is performed on them to obtain the optimized three-dimensional motion parameters, so as to calculate the observed noise variance corresponding to the image frame, and the calculation process is very simple; correspondingly, the predicted three-dimensional motion parameters of the image frame are obtained by making a corresponding prediction based on the optimized three-dimensional motion parameters of the previous frame of the image frame, so as to quickly infer the predicted noise variance of the image frame.

[0067] In another optional embodiment of the present invention, Figure 5 As shown, the calculation based on the observed noise variance and the predicted noise variance to obtain the rotation matrix and the translation matrix of each image frame specifically includes:

[0068] S26: Obtaining a Kalman fusion motion vector of the image frame based on the observed noise variance and the predicted noise variance; and

[0069] S27: Obtain the rotation matrix and translation matrix of the image frame based on the Kalman fusion motion vector calculation.

[0070] In this embodiment, the Kalman fusion motion vector of the image frame is obtained by calculating based on the observed noise variance and the predicted noise variance, and then the rotation matrix and the translation matrix of the image frame are further obtained by using the Kalman fusion motion vector calculation. This makes the motion estimation of the motor vehicle relatively more accurate, which is beneficial to improving the accuracy of the internal parameter calibration.

[0071] In another optional embodiment of the present invention, after obtaining the real-time image frame from the original video image captured by the vehicle-mounted camera 1, the image frame is also preprocessed, and the preprocessing at least includes image noise reduction and histogram equalization. In this embodiment, by performing corresponding image noise reduction and histogram equalization processing on the image frame, subsequent image information calculation and recognition are facilitated, which is conducive to improving the accuracy of intrinsic parameter calibration.

[0072] In another optional embodiment of the present invention, the histogram equalization is an adaptive histogram equalization. In this embodiment, the image frame is pre-processed by the adaptive histogram equalization, so that when the brightness of the shooting environment of the image frame is relatively low (for example: underground parking lot, night driving environment, etc.), the image frame can still be used for online calibration of the vehicle camera 1; in addition, the image can be de-noised by high boost filtering.

[0073] In another optional embodiment of the present invention, the predetermined nonlinear optimization algorithm model is a Levenberg-Marquardt (LM algorithm for short) algorithm model. In this embodiment, the Levenberg-Marquardt algorithm model is used to iteratively optimize the distance loss function, which is insensitive to over-parameterization problems, can effectively handle redundant parameter problems, greatly reduces the chance of the loss function falling into a local minimum, and improves calibration efficiency.

[0074] In the specific implementation, in step S21, when the motor vehicle is traveling on a structured road, the original spatial motion of the motor vehicle in six degrees of freedom can be approximately equivalent to a planar motion with three degrees of freedom. Therefore, by pulse integration of the wheel speed pulse data, the observed three-dimensional motion parameter M of the motor vehicle in the current frame relative to the previous frame can be obtained, which can be expressed as:

[0075] M={mx,my,0,mtz,0,0} (Formula 1)

[0076] Among them, mx and my represent the displacement of the motor vehicle in the X direction and Y direction respectively, and mtz represents the rotation angle of the motor vehicle.

[0077] In step S22, the observed three-dimensional motion parameters mx, my and mtz of the image frame are subjected to Kalman filtering to obtain the optimized three-dimensional motion parameter mx i ∧ 、my i ∧ and mtz i ∧ , i represents the i-th image frame.

[0078] In step S23, since the observation noise of the image frame obeys the standard normal distribution assumption, the corresponding observation noise obeys the normal distribution with a mean of 0 and observation noise variances of δ_mx, δ_my, and δ_mtz, respectively. The observation noise variances in each direction corresponding to the image frame can be expressed as:

[0079]

[0080]

[0081]

[0082] Where i represents the i-th image frame, N represents the total number of image frames, mx i 、my i and mtz i They respectively represent the actual motion parameters in each direction corresponding to the i-th image frame.

[0083] In step S24, based on the rigid body motion assumption of the motor vehicle, the predicted 3D motion parameter mx of the image frame can be estimated according to the optimized 3D motion parameter corresponding to the previous frame of the image frame. i ∨ 、my i ∨ and mtz i ∨ , and are expressed as: mx i ∨ =mx i-1 ∧ +△t*mx i ′ (Formula 5)

[0084] my i ∨ =my i-1 ∧ +△t*my i ′ (Formula 6)

[0085] mtz i ∨ =mtz i-1 ∧ +△t*mtz i ′ (Formula 7)

[0086] Where △t represents the frame difference between the i-th image frame and the i-1-th image frame, mx i '、my i ′ and mtz i ′ respectively represent the derivatives of the actual motion parameters in each direction corresponding to the i-th image frame.

[0087] In step S25, since the predicted noise of the image frame also obeys the standard normal distribution assumption, the corresponding predicted noise obeys the mean value of 0, and the predicted noise variance is δ_mx ∨ , δ_my ∨ ,δ_mtz ∨ The normal distribution of , where the predicted noise variance in each direction corresponding to the image frame can be expressed as:

[0088]

[0089]

[0090]

[0091] In step S26, the noise variance δ_mx is predicted based on the sum of the observed noise variances δ_mx, δ_my, and δ_mtz. ∨ , δ_my ∨ ,δ_mtz ∨ The Kalman gains Kx, Ky and Ktz in each direction corresponding to the image frame can be calculated and expressed as:

[0092] Kx=δ mx *δ mx / (δ mx *δ mx +δ mx ∨ *δ mx ∨ ) (Formula 11)

[0093] Ky=δ my *δ my / (δ my *δ my +δ mu ∨ *δ my ∨ ) (Formula 12)

[0094] Ktz=δ mtz *δ mtz / (δ mtz *δ mtz +δ mtz ∨ *δ mtz ∨ ) (Formula 13);

[0095] The Kalman fusion motion vector M corresponding to the image frame can be obtained from the above Kalman gains Kx, Ky and Ktz K for:

[0096] M K ={mx ∧ ,my ∧ ,mtz ∧} (Formula 14)

[0097] Among them, mx ∧ =Kx*mx+(1-Kx*mx ∨ ) (Formula 15)

[0098] my ∧ =Ky*my+(1-Ky*my ∨ ) (Formula 16)

[0099] mtz ∧ =Ktz*mtz+(1-Ktz*mtz ∨ ) (Equation 17).

[0100] In step S27, the Kalman fusion motion vector M K The rotation matrix R and translation matrix T of the image frame can be calculated.

[0101] In step S31, when calculating the Hamming distance between the descriptor matching feature points, the specific calculation formula is:

[0102]

[0103] Among them, x and y are descriptor matching feature points in binary form, and XOR is an exclusive OR operation.

[0104] In step S32, the Hamming distances between the descriptor matching feature points are compared to obtain the minimum Hamming distance.

[0105] In step S33, the minimum Hamming distance is compared with a predetermined Hamming distance threshold D. Since the minimum Hamming distance represents the closest relationship between the sample and the comparison sample library, there may be false matches. Comparing with the predetermined Hamming distance threshold D can effectively filter out false matches and improve the accuracy of matching. Thus, the matching point pair (p i p j ) matched , where p i represents the matching point in the previous frame of the image frame, p j It represents the matching point in the image frame.

[0106] In step S4, the Kalman fusion motion vector M obtained by the above calculation is combined K , and then based on the principle of epipolar geometry, the matching points in the image frame and its previous frame satisfy the following relationship:

[0107] S1*P1 ∧ *p1=0=S2*P1 ∧ *R*P2+P1 ∧ *T (Formula 19)

[0108] Among them, S1 and S2 are the depths of field of the same spatial point (the same three-dimensional coordinate point corresponding to the matching point pair) under different vehicle camera viewing angles, p1 and p2 are the image coordinates (matching point pairs) corresponding to the two images (two adjacent image frames) corresponding to the spatial point, R is the rotation matrix obtained by the above calculation, and T is the translation matrix obtained by the above calculation. The depths of field S1 and S2 can be calculated through the above calculation relationship, thereby realizing the triangulation of the matching point pairs:

[0109] (p i p j ) matched →(P x P y P z ) (Formula 20)

[0110] Among them, (P x P y P z ) represents the three-dimensional coordinate points corresponding to each of the matching point pairs in the normalized three-dimensional space.

[0111] In step S41, by arbitrarily selecting three non-collinear three-dimensional coordinate points as seed points p1 (x1, y1, z1), p2 (x2, y2, z2) and p3 (x3, y3, z3), the initial spatial plane Ax+By+Cz+D=0 can be constructed.

[0112] In step S42, the point-to-plane distances between other three-dimensional coordinate points and the initial space plane are calculated using a spatial point-to-plane distance calculation formula.

[0113] In step S43, the point-to-surface distance is compared with the predetermined point-to-surface distance threshold, and thus summarized as valid coordinate points. When the actual number of valid coordinate points is greater than the predetermined number, the target space plane and its specific expression (determination of the values ​​of A, B, C, and D) are finally determined.

[0114] In step S5, the matching point pair (p i p j ) matched Corrected to the ideal imaging model of the lens of the vehicle-mounted camera 1 to obtain the corresponding algebraic expression of the three-dimensional coordinate points in the lens of the vehicle-mounted camera 1. Taking the fisheye camera as an example, the specific expression is:

[0115]

[0116]

[0117] in, and are the horizontal and vertical coordinates of the image point of the fisheye camera, X0 and Y0 are the horizontal and vertical coordinates of the center point of the fisheye camera, fc is the focal length of the fisheye camera, k1 and k2 are the polynomial parameters of the lens distortion curve of the fisheye camera. It can be seen from the above that the lens intrinsic parameters of the fisheye camera are X0, Y0, fc, k1 and k2.

[0118] Then, the three-dimensional coordinate point (P x P y P x) and the point-to-surface distance of the key point clustering plane can be expressed as:

[0119]

[0120] Among them, since the denominator in the distance algebraic expression is a constant, the distance loss function algebraic expression can be expressed as:

[0121]

[0122] Among them, P zi , P yi and P zi They are the three-dimensional coordinate points corresponding to the i-th image frame. By iteratively optimizing the nonlinear optimization algorithm model, the minimum distance loss function can be obtained. Then, according to the three-dimensional coordinate points corresponding to the minimum distance loss function, the lens intrinsic parameters X0, Y0, fc, k1 and k2 can be calculated.

[0123] On the other hand, Figure 6 As shown, an embodiment of the present invention further provides a vehicle-mounted camera online calibration device 5, which is respectively connected to a vehicle-mounted camera 1 for shooting video images around a motor vehicle and providing original video images, a wheel pulse sensor 3 for detecting wheel pulses of a motor vehicle and providing wheel speed pulse data, and a parameter storage device 7 for storing lens intrinsic parameters of the vehicle-mounted camera 1. The vehicle-mounted camera online calibration device 5 includes a processor 50, a memory 52, and a computer program stored in the memory 52 and configured to be executed by the processor. When the processor 50 executes the computer program, the vehicle-mounted camera online calibration method described in the above embodiment is implemented.

[0124] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory 52 and executed by the processor to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the vehicle-mounted camera online calibration device 5. For example, the computer program may be divided into Figure 7 The functional modules in the on-board camera online calibration device 5, wherein the frame extraction module 61, the matrix calculation module 62, the feature point matching module 63, the plane determination module 64 and the parameter determination module 65 respectively execute the above steps S1 to S5.

[0125] The vehicle-mounted camera online calibration device 5 can be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The vehicle-mounted camera online calibration device 5 can include, but is not limited to, a processor 50 and a memory 52. ​​Those skilled in the art can understand that the schematic diagram is only an example of the vehicle-mounted camera online calibration device 1, and does not constitute a limitation on the vehicle-mounted camera online calibration device 5, and can include more or fewer components than shown in the figure, or combine certain components, or different components. For example, the vehicle-mounted camera online calibration device 1 can also include input and output devices, network access devices, buses, etc.

[0126] The processor 50 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 50 is the control center of the vehicle-mounted camera online calibration device 5, and uses various interfaces and lines to connect various parts of the entire vehicle-mounted camera online calibration device 5.

[0127] The memory 52 can be used to store the computer program and / or module. The processor 50 realizes various functions of the vehicle-mounted camera online calibration device 5 by running or executing the computer program and / or module stored in the memory 52 and calling the data stored in the memory 52. ​​The memory 52 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a graphic recognition function, a graphic overlay function, etc.); the data storage area can store data created according to the use of the recognition device (such as graphic data, etc.). In addition, the memory 52 can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0128] If the functions described in the embodiments of the present invention are implemented in the form of software function modules or units and sold or used as independent products, they can be stored in a storage medium readable by a computing device. Based on this understanding, the embodiments of the present invention implement all or part of the processes in the above-mentioned embodiments, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor 50, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0129] On the other hand, an embodiment of the present invention further provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the online calibration method for the vehicle-mounted camera as described above.

[0130] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0131] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, which all belong to the protection scope of the present invention.

Claims

1. A vehicle camera online calibration method, characterized in that: The method comprises the following steps: Obtain real-time image frames from the original video images captured by the vehicle-mounted camera; Calculating the observed noise variance and predicted noise variance of each image frame based on the wheel speed pulse of the motor vehicle, and obtaining the rotation matrix and translation matrix of each image frame based on the observed noise variance and predicted noise variance; Extracting BEBLID feature points in each image frame in turn, and then obtaining descriptor matching feature points of each BEBLID feature point, and extracting a pair of matching point pairs from each descriptor matching feature point in the image frame and the previous frame based on the principle of minimizing the Hamming distance starting from the second image frame; triangulate the matching point pairs based on the rotation matrix and the translation matrix to obtain the three-dimensional coordinate points corresponding to each matching point pair in the normalized three-dimensional space, cluster the three-dimensional coordinate points located in the same space plane to obtain a key point clustering plane; and Based on the distance principle from a spatial point to a spatial plane, a distance loss function from a three-dimensional coordinate point to the key point clustering plane is constructed. Based on a predetermined nonlinear optimization algorithm model, the algebraic expression of the distance loss function is iteratively optimized to solve the minimum distance loss function. Then, based on the minimum distance loss function and the ideal imaging model of the lens of the vehicle camera, the lens intrinsic parameters of the vehicle camera are calculated.

2. The on-line calibration method for a vehicle camera according to claim 1, characterized in that: The extracting a pair of matching point pairs from the image frame and each descriptor matching feature point in the previous frame based on the principle of Hamming distance minimization starting from the second image frame specifically includes: Calculating the Hamming distance between each of the descriptor matching feature points in the image frame and the previous frame; Determine the minimum value of the Hamming distance of each of the descriptor matching feature points; and When the minimum value of the Hamming distance is less than a predetermined Hamming distance threshold, a pair of the descriptor matching feature points corresponding to the minimum value of the Hamming distance is determined as a matching point pair of two adjacent image frames.

3. The on-line calibration method for a vehicle camera according to claim 1, characterized in that: The clustering of the three-dimensional coordinate points located in the same spatial plane to obtain the key point clustering plane specifically includes: Arbitrarily select three non-collinear three-dimensional coordinate points to construct an initial space plane; Calculating the point-to-surface distance between each of the three-dimensional coordinate points and the initial spatial plane; When the point-to-plane distance corresponding to the three-dimensional coordinate point is less than a predetermined point-to-plane distance, classifying the three-dimensional coordinate point as a valid coordinate point; and The actual number of valid coordinate points corresponding to the same initial spatial plane is counted, and when the actual number is greater than a predetermined number, the valid coordinate points are stopped from being classified and the initial spatial plane is determined as a key point clustering plane.

4. The on-line calibration method for a vehicle camera according to claim 1, characterized in that: The calculation of the observed noise variance and the predicted noise variance of each image frame based on the wheel speed pulse of the motor vehicle specifically includes: Calculating the observed three-dimensional motion parameters of the motor vehicle within the frame difference time length corresponding to the image frame according to the wheel speed pulses of the motor vehicle in sequence; performing Kalman filtering on the observed three-dimensional motion parameters of the image frame to obtain optimized three-dimensional motion parameters; Combining the actual three-dimensional motion parameters of the image frame and the optimized three-dimensional motion parameters and calculating based on the standard normal distribution principle to obtain the observed noise variance corresponding to each image frame; estimating predicted 3D motion parameters of the image frame based on the optimized 3D motion parameters of a previous frame of the image frame and based on a rigid body motion assumption of the vehicle; and The predicted three-dimensional motion parameter of the image frame and the optimized three-dimensional motion parameter are combined and predicted based on the standard normal distribution principle to obtain the predicted noise variance.

5. The on-line calibration method for a vehicle camera according to claim 1 or 4, characterized in that: The calculation based on the observed noise variance and the predicted noise variance to obtain the rotation matrix and the translation matrix of each image frame specifically includes: Obtaining a Kalman fusion motion vector of the image frame based on the observed noise variance and the predicted noise variance; and The rotation matrix and the translation matrix of the image frame are obtained based on the Kalman fusion motion vector calculation.

6. The on-line calibration method for a vehicle camera according to claim 1, characterized in that: After obtaining the real-time image frame from the original video image captured by the vehicle-mounted camera, the image frame is also preprocessed, and the preprocessing at least includes image noise reduction and histogram equalization.

7. The on-line calibration method for a vehicle-mounted camera according to claim 6, characterized in that: The histogram equalization is an adaptive histogram equalization.

8. The on-line calibration method for a vehicle camera according to claim 1, characterized in that: The predetermined nonlinear optimization algorithm model is a Levenberg-Marquardt algorithm model.

9. An on-board camera online calibration device, connected to an on-board camera for capturing video images around a motor vehicle and providing original video images, a wheel pulse sensor for detecting wheel pulses of a motor vehicle and providing wheel speed pulses of the motor vehicle, and a parameter storage device for storing lens internal parameters of the on-board camera, characterized in that: The on-board camera online calibration device includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the on-board camera online calibration method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the on-line calibration method for a vehicle-mounted camera as described in any one of claims 1 to 8.

Images