Camera calibration method, system and vehicle

Abstract

The present invention discloses a camera calibration method, system, and vehicle, wherein the method comprises: when the vehicle's air suspension is in a normal state and the exterior rearview mirror is in an unfolded state, acquiring a first image captured by a camera, selecting feature points from the first image, and estimating the theoretical motion trajectory of the feature points within a preset calibration time period during the air suspension adjustment process; during the air suspension adjustment process, controlling the camera to capture a second image within a preset calibration time period during the adjustment process, identifying feature points in the second image, and determining the actual motion trajectory of the feature points within the calibration time period; and calibrating the position and posture of the camera in a world coordinate system based on the theoretical and actual motion trajectories of the feature points. The camera calibration method provided by the present invention can complete dynamic calibration of the camera in an environment that does not require manual arrangement, saving calibration space and calibration time.

Description

Technical Field

[0001] The present invention relates to the field of vehicle technology, and in particular to a camera calibration method, system and vehicle. Background Art

[0002] Streaming media exterior mirrors are an emerging automotive technology that replaces traditional exterior mirrors by using cameras mounted on the mirrors to capture images and then display them on a screen inside the vehicle. This technology expands the mirror's field of view, reduces wind resistance, and enhances visibility in challenging conditions such as rain, snow, and at night.

[0003] The current streaming media exterior rearview mirrors mainly rely on the camera's installation position and the preliminary field of view calibration parameters to control the field of view. However, during the installation of the electronic exterior rearview mirror, due to the accumulation of dimensional chain tolerances and assembly errors, the streaming media camera on the electronic rearview mirror often deviates from the theoretical position, resulting in a field of view that cannot meet regulatory requirements. In the prior art, the calibration method for streaming media cameras on exterior rearview mirrors often adopts static calibration. After the vehicle is parked, the calibration cloth is manually laid out, the vehicle is connected to the diagnostic instrument, and the calibration program is run through the diagnostic instrument to extract the characteristic points of the checkerboard on the calibration cloth for calculation and calibration. However, the manual laying of the calibration cloth has a large error, which will affect the calibration results, and it requires a larger site and takes a longer time for calibration.

[0004] Therefore, it is necessary to provide an improved technical solution to overcome the above technical problems existing in the prior art. Summary of the Invention

[0005] In view of the above problems, the object of the present invention is to provide a camera calibration method, system and vehicle, which can complete dynamic calibration of the camera in an environment without relying on manual arrangement, saving calibration site and calibration time.

[0006] The present invention provides a camera calibration method for calibrating a camera of a vehicle's exterior rearview mirror, the method comprising:

[0007] When the air suspension of the vehicle is in a normal state and the exterior rearview mirror is in an unfolded state, obtaining a first image captured by the camera, selecting feature points from the first image, and estimating theoretical motion trajectories of the feature points within a preset calibration time period during adjustment of the air suspension;

[0008] During the adjustment process of the air suspension, controlling the camera to capture a second image within the preset calibration time period during the adjustment process, identifying the feature point in the second image, and determining an actual motion trajectory of the feature point within the calibration time period;

[0009] Based on the theoretical motion trajectory and the actual motion trajectory of the feature points, the position and posture of the camera in the world coordinate system are calibrated.

[0010] Furthermore, the adjustment process of the air suspension includes: adjusting the air suspension from a normal state to a state in which the front axle of the air suspension is fixed and the rear axle is raised to a preset height, or adjusting the air suspension from a normal state to a state in which the rear axle of the air suspension is fixed and the front axle is raised to a preset height.

[0011] Furthermore, the estimating of a theoretical motion trajectory of the characteristic point within a preset calibration time period during the adjustment process of the air suspension includes:

[0012] During the adjustment process of the air suspension, obtaining the adjusted height of the front axle or rear axle of the air suspension and the adjustment time corresponding to each height;

[0013] According to the adjusted heights and the adjustment time corresponding to each height, a theoretical motion trajectory of the characteristic point within a preset calibration time period during the adjustment process of the air suspension is estimated.

[0014] Furthermore, based on the theoretical motion trajectory and the actual motion trajectory of the feature point, the position and posture of the camera in the world coordinate system are calibrated, including:

[0015] Obtaining a position change relationship of the feature point in a world coordinate system according to a theoretical motion trajectory of the feature point;

[0016] Obtaining a position change relationship of the feature point in a camera coordinate system according to the actual motion trajectory of the feature point;

[0017] Constructing a calibration equation according to the position change relationship of the feature point in the world coordinate system and the camera coordinate system;

[0018] The position and posture of the camera in the world coordinate system are calibrated using the calibration equation.

[0019] Furthermore, constructing a calibration equation according to the position change relationship of the feature point in the world coordinate system and the camera coordinate system includes:

[0020] Construct the calibration equation based on the world coordinates and camera coordinates at the mth position: A m X=XB m ;

[0021] Construct a calibration equation based on the world coordinates and camera coordinates at the nth position: A n X=XB n ;

[0022] Among them, A mand A n are the world coordinates of the mth and nth positions respectively, m and B n are the camera coordinates at the mth position and the nth position in the camera coordinate system respectively;

[0023] According to the calibration equations at the mth position and the nth position, the position change equation of the feature point from the mth position to the nth position is obtained: AX=XB, where

[0024]

[0025] R A and T A is the rotation matrix and translation matrix of the feature point from the mth position to the nth position in the world coordinate system, R B and T B is the rotation matrix and translation matrix of the feature point from the mth position to the nth position in the camera frame;

[0026] The conversion relationship X from the world coordinate system to the camera coordinate system is obtained by solving the calibration equation, wherein X includes the rotation matrix R representing the camera coordinate system relative to the world coordinate system. X , and the translation matrix T representing the origin of the camera coordinate system relative to the origin of the world coordinate system X .

[0027] Furthermore, the method further comprises the step of: X and the translation matrix T X Determine the position coordinates of the point in the environment in the camera coordinate system. The conversion relationship is:

[0028]

[0029] Among them, (x c ,y c ,z c ,1) T is the homogeneous coordinate in the camera coordinate system, (x w ,y w ,z w ,1) T is the homogeneous coordinate in the world coordinate system.

[0030] Furthermore, selecting feature points from the first image includes: selecting points on the vehicle body in the first image as feature points, or selecting points in the first image that are stationary relative to the vehicle body as feature points.

[0031] The present invention also provides a camera calibration system, comprising a camera, a processing module and a display module;

[0032] The camera is connected to the processing module and is configured to capture a first image when the air suspension of the vehicle is in a normal state and the exterior rearview mirror is in an unfolded state, and to capture a second image within a preset calibration time period during adjustment of the air suspension;

[0033] The display module is connected to the processing module and is used to display the first image and the second image captured by the camera and select feature points for camera calibration;

[0034] The processing module is used to estimate the theoretical motion trajectory of the feature point within a preset calibration time period during the adjustment process of the air suspension; during the adjustment process of the air suspension, control the camera to capture a second image within the preset calibration time period during the adjustment process, identify the feature point in the second image, and determine the actual motion trajectory of the feature point within the calibration time period; based on the theoretical motion trajectory and actual motion trajectory of the feature point, calibrate the position and posture of the camera in the world coordinate system.

[0035] Furthermore, the processing module is also used to obtain the adjusted height of the front axle or rear axle of the air suspension and the adjustment time corresponding to each height during the adjustment process of the air suspension; and estimate the theoretical motion trajectory of the feature point within a preset calibration time period during the adjustment process of the air suspension based on the adjusted height and the adjustment time corresponding to each height.

[0036] The present invention also provides a vehicle, comprising the camera calibration system as described above.

[0037] The camera calibration method provided by the present invention adjusts the air suspension to change the relative position of the feature point and the camera, and uses the position change relationship of the same feature point in the world coordinate system and the position change relationship in the camera coordinate system to construct a calibration equation, thereby obtaining the conversion relationship from the world coordinate system to the camera coordinate system, and completing the calibration of the camera without relying on an artificially arranged environment, saving calibration site and calibration time, and improving calibration accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 A schematic diagram of a flow chart of a camera calibration method provided by one embodiment of the present invention;

[0040] Figure 2 A schematic diagram of the theoretical motion trajectory and actual motion trajectory of a feature point provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The aforementioned and other technical contents, features, and effects of the present invention will be clearly presented in the following detailed description of the preferred embodiments with reference to the drawings. Through the description of the specific embodiments, a deeper and more specific understanding of the technical means and effects adopted by the present invention to achieve the intended purpose can be obtained. However, the accompanying drawings are only for reference and illustration purposes and are not intended to limit the present invention. Some well-known parts may not be shown. In the various drawings, the same elements are represented by similar reference numerals. For the sake of clarity, the various parts in the drawings are not drawn to scale.

[0042] An embodiment of the present invention provides a camera calibration method for calibrating a camera of a vehicle's exterior rearview mirror, such as Figure 1 As shown, the method includes the following steps:

[0043] Step S110: When the air suspension of the vehicle is in a normal state and the exterior rearview mirror is in an unfolded state, obtaining a first image captured by a camera, selecting feature points from the first image, and estimating theoretical motion trajectories of the feature points within a preset calibration time period during adjustment of the air suspension;

[0044] Specifically, the vehicle's air suspension is routinely adjusted to maintain, normal, and awakened states. In this embodiment, before calibration begins, the vehicle's air suspension must be adjusted to its normal state and the exterior mirrors adjusted to the default regulatory field of view (i.e., with the exterior mirrors unfolded). A background with a significant color difference from the vehicle body is selected to accurately identify the selected feature points during subsequent calibration. During calibration, the relative position of the selected feature points and the camera is altered by adjusting the height of the air suspension's front or rear axle.

[0045] Specifically, a point on the vehicle body in the first image, or a point that is stationary relative to the vehicle body in the first image, can be selected as a feature point. The feature point should be easily identifiable from surrounding pixels and remain within the camera's field of view during air suspension adjustment. In other embodiments, manual marking can be used to facilitate identification of feature points in images captured by the camera. In one embodiment, if a point on the vehicle body in the first image is selected as a feature point, the relative position of the selected feature point and the camera can be changed by adjusting the air suspension from a normal state to a state where the front axle is fixed and the rear axle is raised to a preset height, or by adjusting the air suspension from a normal state to a state where the rear axle is fixed and the front axle is raised to a preset height. In another embodiment, if a point on the vehicle body in the first image is selected as a feature point, i.e., the feature point is not located on the vehicle body, calibration can be performed using points in the vehicle body's surroundings or manually marked points. In this case, the front and rear axles of the air suspension can also be raised to a preset height to change the relative position of the selected feature point and the camera.

[0046] During air suspension adjustment, the adjusted height of the front or rear axle of the air suspension and the corresponding adjustment time for each height are obtained. Based on the adjusted height and the corresponding adjustment time for each height, the theoretical motion trajectory of the characteristic point within a preset calibration time period during the air suspension adjustment process is estimated. The adjusted height information and the corresponding adjustment time for each height can be recorded in real time by the vehicle's electronic control unit.

[0047] Step S120: During the adjustment process of the air suspension, controlling the camera to capture a second image within a preset calibration time period during the adjustment process, identifying feature points in the second image, and determining actual motion trajectories of the feature points within the calibration time period;

[0048] Specifically, during calibration in one embodiment, the front axle of the air suspension is controlled to be fixed and the rear axle is raised by 50 mm. At the same time, the camera is controlled to capture images during the raising of the rear axle, and the rear axle height information corresponding to the timestamp of each captured frame of image is recorded. Feature point recognition is performed on each captured frame of image, and the position information of the same feature point in multiple frames of the second image can be obtained. The actual motion trajectory of the feature point can be obtained based on the obtained position information of the feature point.

[0049] Step S130: Calibrate the position and posture of the camera in the world coordinate system based on the theoretical motion trajectory and the actual motion trajectory of the feature points.

[0050] Specifically, the position change relationship of the feature points in the world coordinate system is obtained according to the theoretical motion trajectory of the feature points; the position change relationship of the feature points in the camera coordinate system is obtained according to the actual motion trajectory of the feature points; the calibration equation is constructed according to the position change relationship of the feature points in the world coordinate system and the camera coordinate system; the position and posture of the camera in the world coordinate system is calibrated by the calibration equation.

[0051] For example, when the front axle of the air suspension is fixed and the rear axle is raised by 50 mm, the theoretical motion trajectory of the feature point is matched with the actual motion trajectory according to the timestamp corresponding to each frame image. The matching results are as follows: Figure 2 As shown, Figure 2 The trajectory A in the figure is the theoretical motion trajectory of the feature point, and the trajectory B is the actual motion trajectory of the feature point. According to the theoretical motion trajectory of the feature point, the world coordinates of each position of the feature point in the world coordinate system can be obtained, and according to the actual motion trajectory of the feature point, the camera coordinates of each position of the feature point in the camera coordinate system can be obtained.

[0052] Specifically, taking the feature point from the mth position to the nth position as an example, the calibration equation is constructed according to the world coordinates and camera coordinates at the mth position: m X=XB m ; Construct the calibration equation based on the world coordinates and camera coordinates at the nth position: A n X=XB n Among them, A m and A n are the world coordinates of the mth and nth positions respectively, m and B n are the camera coordinates at the mth position and the nth position in the camera coordinate system respectively;

[0053] Through the above two calibration equations, we can get: A m XB m =A n XB n ;

[0054] The above formula can be converted to:

[0055] Therefore, according to the calibration equations at the mth position and the nth position, the position change equation of the feature point from the mth position to the nth position is obtained: AX=XB, where

[0056]

[0057] R A and T Ais the rotation matrix and translation matrix of the feature point from the mth position to the nth position in the world coordinate system, R B and T B is the rotation matrix and translation matrix of the feature point from the mth position to the nth position in the camera frame;

[0058] According to the above position change equation, we can get:

[0059]

[0060] By solving the above equation, we can obtain the transformation relationship X from the world coordinate system to the camera coordinate system, which includes the rotation matrix R representing the camera coordinate system relative to the world coordinate system. X , and the translation matrix T representing the origin of the camera coordinate system relative to the origin of the world coordinate system X ; Rotation matrix R X and the translation matrix T X The specific solution process will not be introduced in detail in this case.

[0061] The rotation matrix R obtained by solving X and the translation matrix T X This can determine the camera's installation position and angle in the world coordinate system, and can also map points in the vehicle's driving environment from the world coordinate system to the camera coordinate system:

[0062]

[0063] Among them, (x c ,y c ,z c ,1) T is the homogeneous coordinate in the camera coordinate system, (x w ,y w ,z w ,1) T is the homogeneous coordinate in the world coordinate system.

[0064] The present invention also provides a camera calibration system, comprising a camera, a processing module and a display module.

[0065] Among them, the camera is connected to the processing module, which is used to collect a first image when the vehicle's air suspension is in a normal state and the exterior rearview mirror is in an unfolded state, and to collect a second image within a preset calibration time period during the adjustment process of the air suspension; the display module is connected to the processing module, which is used to display the first image and the second image collected by the camera, and select feature points for camera calibration. The display module has a human-computer interaction function, such as a touch display screen, and the user can select appropriate feature points for calibration through the human-computer interaction function of the display module; the processing module is used to estimate the theoretical motion trajectory of the feature points within a preset calibration time period during the adjustment process of the air suspension; during the adjustment process of the air suspension, the camera is controlled to collect a second image within a preset calibration time period during the adjustment process, identify feature points in the second image, and determine the actual motion trajectory of the feature points within the calibration time period; based on the theoretical motion trajectory and actual motion trajectory of the feature points, the position and posture of the camera in the world coordinate system are calibrated.

[0066] Specifically, the processing module is also used to obtain the adjusted height of the front axle or rear axle of the air suspension and the adjustment time corresponding to each height during the adjustment process of the air suspension; based on the adjusted height and the adjustment time corresponding to each height, the theoretical motion trajectory of the feature point within a preset calibration time period during the adjustment process of the air suspension is estimated.

[0067] The present invention also provides a vehicle, comprising the camera calibration system as described above.

[0068] In summary, the camera calibration method provided by the present invention integrates the adjustment parameters of the air suspension into the camera calibration, changes the relative position of the feature point and the camera by adjusting the air suspension, and constructs a calibration equation using the position change relationship of the same feature point in the world coordinate system and the position change relationship in the camera coordinate system to obtain the conversion relationship from the world coordinate system to the camera coordinate system, thereby completing the calibration of the camera without relying on an artificially arranged environment, saving calibration site and calibration time, and improving calibration accuracy.

[0069] In the description of the present invention, unless otherwise expressly specified and limited, the terms "setting", "installation", "connection" and the like should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms can be understood according to the specific circumstances. The terms "first", "second", "third" and the like are only used to distinguish elements with similar properties, and do not indicate or imply relative importance or a specific order. The terms "include", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, and in addition to including those listed elements, may also include other elements that are not explicitly listed.

[0070] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with this technical field within the technical scope disclosed in the present invention should be covered by the scope of protection of the present invention.

Claims

1. A camera calibration method for calibrating a camera of a vehicle's exterior rearview mirror, characterized in that: The method comprises: When the air suspension of the vehicle is in a normal state and the exterior rearview mirror is in an unfolded state, obtaining a first image captured by the camera, and selecting feature points from the first image; Adjusting the air suspension from a normal state to a state in which the front axle of the air suspension is fixed and the rear axle is raised to a preset height, or adjusting the air suspension from a normal state to a state in which the rear axle of the air suspension is fixed and the front axle is raised to a preset height; During the adjustment process of the air suspension, estimating a theoretical motion trajectory of the characteristic point within a preset calibration time period based on the adjusted height of the rear axle or the front axle of the air suspension and the adjustment time corresponding to each height; controlling the camera to capture a second image within the calibration time period during the adjustment process, identifying the feature point in the second image, and determining an actual motion trajectory of the feature point within the calibration time period; Obtaining a position change relationship of the feature point in a world coordinate system according to a theoretical motion trajectory of the feature point, obtaining a position change relationship of the feature point in a camera coordinate system according to an actual motion trajectory of the feature point, and constructing a calibration equation according to the position change relationship of the feature point in the world coordinate system and the camera coordinate system; The position and posture of the camera in the world coordinate system are calibrated using the calibration equation.

2. The camera calibration method according to claim 1, wherein: The constructing of a calibration equation according to the position change relationship of the feature point in the world coordinate system and the camera coordinate system includes: Construct the calibration equation based on the world coordinates and camera coordinates at the mth position: A m X=XB m ; Construct a calibration equation based on the world coordinates and camera coordinates at the nth position: A n X=XB n ; Among them, A m and A n are the world coordinates of the mth and nth positions respectively, m and B n are the camera coordinates at the mth position and the nth position in the camera coordinate system respectively; According to the calibration equations at the mth position and the nth position, the position change equation of the feature point from the mth position to the nth position is obtained: AX=XB, where ， R A and T A is the rotation matrix and translation matrix of the feature point from the mth position to the nth position in the world coordinate system, R B and T B is the rotation matrix and translation matrix of the feature point from the mth position to the nth position in the camera frame; The conversion relationship X from the world coordinate system to the camera coordinate system is obtained by solving the calibration equation, wherein X includes the rotation matrix R representing the camera coordinate system relative to the world coordinate system. X , and the translation matrix T representing the origin of the camera coordinate system relative to the origin of the world coordinate system X .

3. The camera calibration method according to claim 2, wherein: The method further comprises the step of solving the rotation matrix R X and the translation matrix T X Determine the position coordinates of the point in the environment in the camera coordinate system. The conversion relationship is: in, is the homogeneous coordinate in the camera coordinate system, is the homogeneous coordinate in the world coordinate system.

4. The camera calibration method according to claim 1, wherein: The selecting of feature points from the first image includes: selecting points on the vehicle body in the first image as feature points, or selecting points in the first image that are stationary relative to the vehicle body as feature points.

5. A camera calibration system, characterized in that: Used to implement the camera calibration method according to any one of claims 1 to 4, the system includes a camera, a processing module and a display module; The camera is connected to the processing module and is configured to capture a first image when the air suspension of the vehicle is in a normal state and the exterior rearview mirror is in an unfolded state, and to capture a second image within a preset calibration time period during adjustment of the air suspension; The display module is connected to the processing module and is used to display the first image and the second image captured by the camera and select feature points for camera calibration; The processing module is used to estimate the theoretical motion trajectory of the feature point within a preset calibration time period during the adjustment process of the air suspension; during the adjustment process of the air suspension, control the camera to capture a second image within the preset calibration time period during the adjustment process, identify the feature point in the second image, and determine the actual motion trajectory of the feature point within the calibration time period; based on the theoretical motion trajectory and actual motion trajectory of the feature point, calibrate the position and posture of the camera in the world coordinate system.

6. A vehicle, characterized in that: The vehicle includes the camera calibration system as claimed in claim 5.

Images