A method and device for calibrating a vehicle-mounted camera

Abstract

The present invention provides a calibration method and device for a vehicle-mounted camera, which obtains ground calibration results and vehicle driving direction calibration results by processing current frame vehicle driving image data, the previous frame vehicle driving image data of the current frame vehicle driving image data, and the next frame vehicle driving image data of the current frame vehicle driving image data, and thereby obtains the transformation relationship between the camera coordinate system and the world coordinate system defined on the vehicle, so that the transformation relationship between the camera coordinate system and the world coordinate system defined on the vehicle can be calibrated according to actual conditions, thereby improving the calibration accuracy, simplifying the calibration process, and promoting the application of vehicle-mounted cameras.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle reversing, and in particular to a calibration method and device for a vehicle-mounted camera. Background Art

[0002] At present, with the development of digital image processing technology, panoramic imaging technology has gradually become popular. The panoramic imaging system collects images in real time through multiple cameras, and splices the images to generate a panoramic image of the surrounding environment, forming a panoramic imaging system. Of course, the panoramic imaging system can also use a single camera to collect images, and obtain a panoramic image after processing the collected images.

[0003] In order to obtain a panoramic image, usually the positional relationship between the camera, vehicle and ground is calibrated using a static physical marker to obtain the positional relationship between the camera, vehicle and ground in some specific situations. Then, the panoramic image is obtained by using the obtained positional relationship between the camera, vehicle and ground.

[0004] During vehicle driving, the positional relationship between the camera, vehicle and the ground often changes. Therefore, the result of calibrating the positional relationship between the camera, vehicle and the ground using static physical signs cannot be applied under any road conditions, resulting in the panoramic imaging system often obtaining erroneous panoramic image results during actual use. Summary of the invention

[0005] To solve the above problems, an object of the embodiments of the present invention is to provide a method and device for calibrating a vehicle-mounted camera.

[0006] In a first aspect, an embodiment of the present invention provides a calibration method for a vehicle-mounted camera, comprising:

[0007] Processing the vehicle driving image data of the current frame and the vehicle driving image data of the previous frame of the vehicle driving image data of the current frame to obtain a first position change when the vehicle moves from a first position indicated by the previous frame of the vehicle driving image data to a second position indicated by the current frame of the vehicle driving image data, and calibrating the ground according to the obtained first position change to obtain a calibration result of the ground;

[0008] Processing a current frame of vehicle driving image data and a next frame of vehicle driving image data of the current frame of vehicle driving image data to obtain a second posture change when the vehicle moves from a second position indicated by the current frame of vehicle driving image data to a third position indicated by a next frame of vehicle driving image data of the current frame of vehicle driving image data, and calibrating the vehicle driving direction according to the obtained second posture change and the calibration result of the ground to obtain a calibration result of the vehicle driving direction;

[0009] According to the calibration results of the ground and the calibration results of the vehicle's driving direction, the transformation relationship between the camera coordinate system and the world coordinate system defined on the vehicle is obtained.

[0010] In a second aspect, an embodiment of the present invention further provides a calibration device for a vehicle-mounted camera, comprising:

[0011] A first processing module is used to process the current frame vehicle driving image data and the previous frame vehicle driving image data of the current frame vehicle driving image data, obtain a first position change when the vehicle moves from a first position indicated by the previous frame vehicle driving image data to a second position indicated by the current frame vehicle driving image data, and calibrate the ground according to the obtained first position change to obtain a ground calibration result;

[0012] A second processing module is used to process the current frame vehicle driving image data and the next frame vehicle driving image data of the current frame vehicle driving image data to obtain a second posture change when the vehicle moves from a second position indicated by the current frame vehicle driving image data to a third position indicated by the next frame vehicle driving image data of the current frame vehicle driving image data, and calibrate the vehicle driving direction according to the obtained second posture change and the calibration result of the ground to obtain a calibration result of the vehicle driving direction;

[0013] The third processing module is used to obtain the transformation relationship between the camera coordinate system and the world coordinate system defined on the vehicle according to the calibration results of the ground and the calibration results of the vehicle's driving direction.

[0014] In the solutions provided in the first aspect to the second aspect of the embodiments of the present invention, by processing the current frame vehicle driving image data, the previous frame vehicle driving image data of the current frame vehicle driving image data, and the next frame vehicle driving image data of the current frame vehicle driving image data, the calibration results of the ground and the calibration results of the vehicle driving direction are obtained, and the transformation relationship between the camera coordinate system and the world coordinate system defined on the vehicle is obtained. Compared with the method in the related art that needs to use physical marks to calibrate the positional relationship between the camera, the vehicle and the ground, the transformation relationship between the camera coordinate system and the world coordinate system defined on the vehicle can be calibrated according to actual conditions, thereby improving the calibration accuracy. Therefore, when the panoramic imaging system is actually used, an accurate panoramic image result can be obtained, thereby improving the application experience of the panoramic image.

[0015] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 A schematic structural diagram of a panoramic imaging system applicable to an embodiment of the present invention is shown;

[0018] Figure 2 A relationship diagram between a world coordinate system, a ground coordinate system, and a camera coordinate system applicable to an embodiment of the present invention is shown;

[0019] Figure 3 A flow chart of a vehicle-mounted camera calibration method provided in Embodiment 1 of the present invention is shown;

[0020] Figure 4 A schematic structural diagram of a vehicle-mounted camera calibration device provided in Embodiment 2 of the present invention is shown. DETAILED DESCRIPTION

[0021] At present, with the development of digital image processing technology, panoramic imaging technology has gradually become popular. The panoramic imaging system collects images in real time through multiple cameras, and splices the images to generate a panoramic image of the surrounding environment, forming a panoramic imaging system. Of course, the panoramic imaging system can also use a single camera to collect images, and obtain a panoramic image after processing the collected images.

[0022] In order to obtain a panoramic image, usually the positional relationship between the camera, vehicle and ground is calibrated using a static physical marker to obtain the positional relationship between the camera, vehicle and ground in some specific situations. Then, the panoramic image is obtained by using the obtained positional relationship between the camera, vehicle and ground.

[0023] During vehicle driving, the positional relationship between the camera, vehicle and the ground often changes. Therefore, the result of calibrating the positional relationship between the camera, vehicle and the ground using static physical signs cannot be applied under any road conditions, resulting in the panoramic imaging system often obtaining erroneous panoramic image results during actual use.

[0024] Based on this, the present embodiment proposes a calibration method and device for a vehicle-mounted camera, which obtains ground calibration results and vehicle driving direction calibration results by processing the current frame vehicle driving image data, the previous frame vehicle driving image data of the current frame vehicle driving image data, and the next frame vehicle driving image data of the current frame vehicle driving image data, and thereby obtains the transformation relationship between the camera coordinate system and the world coordinate system defined on the vehicle. The transformation relationship between the camera coordinate system and the world coordinate system defined on the vehicle can be calibrated according to the actual scene, thereby improving the calibration accuracy, so that when the panoramic imaging system is actually used, an accurate panoramic image result can be obtained.

[0025] See also Figure 1 The panoramic imaging system installed on the vehicle shown includes a camera 100, an image processing unit 102, an image storage unit 104 and an image display unit 106. The camera 100 is installed at the rear of the vehicle and is used to obtain images behind the vehicle. The image processing unit 102 is mainly responsible for image processing tasks such as image distortion correction, image stitching, image perspective transformation, and calibration of the relationship between the camera, the ground and the vehicle. It is the core computing unit of this solution. The image storage unit 104 is used to store images. The image display unit 106 is used to display the panoramic image to the driver in real time, and visually show the driver the surrounding and bottom images of the vehicle in all directions without blind spots.

[0026] In order to calibrate the relationship between the camera, the ground and the vehicle, first, the camera is calibrated to obtain the camera parameters such as the focal length, principal point, lens distortion coefficient, etc. of the camera; the camera parameters are the internal parameters related to the characteristics of the camera itself. Then, the calibrated camera is installed at a suitable position at the rear of the vehicle, the vehicle is started to reverse, and the image processing unit 102 enters the automatic calibration program. After entering the automatic calibration program, the vehicle is kept moving straight, and any two frames of images with similar time (i.e., the current frame of vehicle driving image data and the previous frame of vehicle driving image data of the current frame of vehicle driving image data) are selected for processing to ensure that the two frames of images have enough common feature points for ground calibration to obtain the normal vector of the ground in the camera coordinate system. If the ground calibration fails, a scene with rich ground texture is selected and the ground calibration is performed again until the ground calibration is successful. After completing the ground calibration, the vehicle continues to move straight until the positive direction of the vehicle is obtained, and the vehicle positive direction calibration is completed. Finally, the conversion matrix of the camera and the vehicle world coordinate system is calculated according to the ground calibration results and the vehicle positive direction calibration results to complete the automatic calibration.

[0027] In order to facilitate the calibration of the relationship between the camera, the ground and the vehicle, this solution sets the world coordinate system, the ground coordinate system and the camera coordinate system to illustrate how to calibrate the relationship between the camera, the ground and the vehicle.

[0028] See also Figure 2 The relationship between the world coordinate system, ground coordinate system and camera coordinate system shown in the figure, OX W Y W Z W Represents the world coordinate system; OZ W The positive direction of the axis is opposite to the direction in which the vehicle moves forward; when facing the vehicle moving forward, OY W The positive direction of the axis is perpendicular to the left side of the vehicle; plane OY W Z W Parallel to the ground, OX W The positive direction of the axis is vertical to the ground and upward. C Y C Z C is the camera coordinate system, where OZ C The axis is parallel to the camera's principal optical axis, and its positive direction points to the camera's imaging plane; OX C The axis is parallel to the upper boundary of the camera imaging plane, and the positive direction points to the right; OY C The axis is parallel to the left edge of the camera imaging plane, with the positive direction pointing downward. G Y G Z G is the ground coordinate system, which is the intermediate variable between the world coordinate system and the camera coordinate system and is determined by the world coordinate system and the camera coordinate system; G Axis and world coordinate system OX W Y W Z W OX W The axes coincide and have the same direction; let the plane OX pass through the origin O and be perpendicular to it. C X G The axis is the vector v of the rotation axis of the camera coordinate system and the ground coordinate system, and the camera coordinate system OX C Y C Z C After rotating around the v axis by an angle θ, the camera coordinate system OX C Axis and ground coordinate system OX G The axes coincide and have the same direction; at this time, OY of the camera coordinate system C Axis and OZ C Axis and ground coordinate system OY G Axis and OZ G The axes are coincident and in the same direction.

[0029] In the following embodiments, the term "positive direction of the vehicle" has the same meaning as the term "driving direction of the vehicle".

[0030] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0031] Example 1

[0032] This embodiment provides a calibration method for a vehicle-mounted camera, and the execution subject is the image processing unit 102 in the panoramic imaging system.

[0033] The image processing unit 102 may adopt any image processor in the prior art that can process images and calibrate the relationship between the camera, the ground and the vehicle, which will not be described in detail here.

[0034] See also Figure 3 The process of the vehicle camera calibration method shown in the figure, the vehicle camera calibration method proposed in this embodiment may include the following steps:

[0035] Step 200: Process the current frame vehicle driving image data and the previous frame vehicle driving image data of the current frame vehicle driving image data to obtain a first position change when the vehicle moves from a first position indicated by the previous frame vehicle driving image data to a second position indicated by the current frame vehicle driving image data, and calibrate the ground according to the obtained first position change to obtain a ground calibration result.

[0036] In the above step 200, first, the camera 100 in the panoramic imaging system collects the current frame of vehicle driving image data and the previous frame of vehicle driving image data during the vehicle driving process, and then sends the collected current frame of vehicle driving image data and the previous frame of vehicle driving image data to the image processing unit 102 for processing.

[0037] The current frame of vehicle driving image data and the previous frame of vehicle driving image data of the current frame of vehicle driving image data may be two adjacent frames of image data or may not be two adjacent frames of image data.

[0038] In order to obtain the first pose change, the image processing unit 102 obtains the camera at the first position L 1 The previous frame of vehicle driving image data f of the current frame of vehicle driving image data obtained at 1 , and in the second position L 2 The current frame vehicle driving image data f 2 Two frames of images. After image processing, feature point extraction and matching, we get f 1 and f 2 The common feature point set Ω 12 , and get Ω 12 Then, the image coordinates of all common feature points in the image are calculated by combining the camera parameters using the five-point method or the eight-point method and other publicly available methods. 1 To the second position L2 The first pose change at: the first rotation matrix R and the first translation vector T; then, calculate the set Ω 12 The three-dimensional coordinates of all feature points in the camera coordinate system, which is the camera at position L 1 The camera coordinate system at this time is called the base camera coordinate system; finally, the camera parameters, the first rotation matrix R and the first translation vector T are used to calculate the plane equation of the ground in the camera coordinate system using the random sampling consistency algorithm.

[0039] The image coordinates are the position coordinates of the pixels where the common feature points are located in the image, including: image horizontal coordinates and image vertical coordinates.

[0040] The plane equation of the ground in the camera coordinate system can be expressed by the following formula 1:

[0041] A Gc ×X G +B GC ×Y G +C GC ×Z G =D GC (1)

[0042] Among them, A GC express Coordinate on the x-axis; B GC express Coordinate on the y-axis; C GC express The coordinate on the y-axis; Represents the normal vector of the ground in the camera coordinate system; D GC Indicates the distance from the ground to the origin of the camera coordinate system in the camera coordinate system.

[0043] The above set Ω 12 The image coordinates of all feature points, the first pose change, and the process of obtaining the plane equation of the ground in the camera coordinate system are all existing technologies and will not be repeated here.

[0044] In the obtained set Ω 12 After obtaining the image coordinates of all feature points in the camera coordinate system, the first pose change, and the plane equation of the ground in the camera coordinate system, the ground can be calibrated according to the first pose change to obtain the calibration result of the ground. In order to calibrate the ground according to the first pose change to obtain the calibration result of the ground, the above step 200 can also include the following steps (1) to (5):

[0045] (1) obtaining camera parameters, and calculating a plane equation of the ground in the camera coordinate system according to the camera parameters, the first rotation matrix, and the first translation vector;

[0046] (2) Obtain the normal vector of the ground in the camera coordinate system based on the obtained plane equation of the ground in the camera coordinate system;

[0047] (3) Obtain the vector of the X-axis of the camera coordinate system, and calculate the vector of the rotation axis between the camera coordinate system and the ground coordinate system based on the normal vector of the ground in the camera coordinate system and the vector of the X-axis of the camera coordinate system;

[0048] (4) Calculate the rotation angle of the camera coordinate system and the ground coordinate system based on the normal vector of the ground in the camera coordinate system;

[0049] (5) According to the vector of the rotation axis and the rotation angle, a transformation matrix between the camera coordinate system and the ground coordinate system is obtained, thereby obtaining a ground calibration result.

[0050] In the above step (1), the process of calculating the plane equation of the ground in the camera coordinate system according to the camera parameters, the first rotation matrix and the first translation vector is a prior art and will not be repeated here.

[0051] The camera parameters are input into the cache of the image processing unit 102 in the panoramic imaging system by the technician after the camera calibration is completed. Therefore, the image processing unit 102 can obtain the camera parameters from the image processing unit 102 itself.

[0052] In the above step (2), the normal vector OX of the ground in the camera coordinate system can be determined by the above formula 1: G The value in the camera coordinate system is (A GC , B GC , C GC ).

[0053] In the above step (3), the vector of the rotation axis between the camera coordinate system and the ground coordinate system is calculated by the following formula 2:

[0054]

[0055] Among them, v T Represents the transposed vector of the rotation axis v between the camera coordinate system and the ground coordinate system; The vector representing the X axis of the camera coordinate system; Represents the normal vector of the ground in the camera coordinate system; A GG express Coordinate on the x-axis; B GC express Coordinate on the y-axis; C GC express The coordinate on the y-axis.

[0056] The vector of the rotation axis between the camera coordinate system and the ground coordinate system is v.

[0057] in, The specific vector value of can be stored in the image processing unit 102 after the camera is calibrated.

[0058] Therefore, when the image processing unit 102 calculates the vector of the rotation axis of the camera coordinate system and the ground coordinate system, it can obtain the following information from the image processing unit 102 itself: The specific vector value of .

[0059] In the above step (4), the rotation angle between the camera coordinate system and the ground coordinate system is calculated by the following formula 3:

[0060]

[0061] Among them, A GC express Coordinate on the x-axis; B GC express Coordinate on the y-axis; C GC express The coordinate on the y-axis.

[0062] In the above step (5), the transformation matrix between the camera coordinate system and the ground coordinate system is calculated by the following formula 4:

[0063]

[0064] Among them, R GC represents the transformation matrix between the camera coordinate system and the ground coordinate system; I represents the unit matrix; v u A vector representing the normalized rotation axis; represents the normalized rotation axis v u The transposed vector of the camera coordinate system; θ represents the rotation angle between the camera coordinate system and the ground coordinate system; x u The coordinate of the vector of the rotation axis after normalization on the x-axis; u The coordinate of the vector of the rotation axis after normalization on the y-axis; u The normalized coordinate of the vector of the rotation axis on the z-axis.

[0065] The normalized vector of the rotation axis is v u .

[0066] Here, in order to get v u , the image processing unit 102 needs to perform unitization processing on v.

[0067] v is normalized by the following formula 5:

[0068]

[0069] The unit matrix is ​​stored in the image processing unit 102 .

[0070] After obtaining the calibration result of the ground through step 200, the following step 202 may be performed to calibrate the driving direction of the vehicle.

[0071] Step 202: Process the current frame vehicle driving image data and the next frame vehicle driving image data of the current frame vehicle driving image data to obtain a second posture change when the vehicle moves from a second position indicated by the current frame vehicle driving image data to a third position indicated by the next frame vehicle driving image data of the current frame vehicle driving image data, and calibrate the vehicle driving direction based on the obtained second posture change and the calibration result of the ground to obtain a calibration result of the vehicle driving direction.

[0072] In the above step 202, the current frame of vehicle driving image data and the next frame of vehicle driving image data of the current frame of vehicle driving image data may be two adjacent frames of image data or two non-adjacent frames of image data.

[0073] The second posture change includes: a second rotation matrix and a second translation vector when the vehicle moves from a second position indicated by the current frame vehicle driving image data to a third position indicated by the next frame vehicle driving image data of the current frame vehicle driving image data.

[0074] In order to calibrate the vehicle's driving direction, the above step 202 may perform the following steps (1) to (6):

[0075] (1) Determine the current frame vehicle driving image data f 2 , the next frame of vehicle driving image data of the current frame of vehicle driving image data f 3 , and the previous frame of vehicle driving image data f of the current frame of vehicle driving image data 1 The common features of

[0076] (2) using the camera parameters, the image coordinates of the common feature points, and the three-dimensional coordinates in the camera coordinate system, a second rotation matrix and a second translation vector are calculated;

[0077] (3) calculating a first coordinate of the optical center of the camera at the third position in the camera coordinate system based on the second rotation matrix and the second translation vector;

[0078] (4) converting the first coordinate of the camera into a second coordinate of the optical center of the camera at the third position in the ground coordinate system;

[0079] (5) calculating the rotation angle at which the ground coordinate system and the world coordinate system coincide with each other based on the second coordinate;

[0080] (6) Obtain the vectors of the rotation axes of the ground coordinate system and the world coordinate system, and obtain the transformation matrix from the ground coordinate system to the world coordinate system based on the rotation angle of overlap between the ground coordinate system and the world coordinate system and the vectors of the rotation axes of the ground coordinate system and the world coordinate system, thereby obtaining the calibration result of the vehicle's driving direction.

[0081] In the above step (1), the process of determining the common feature points of the current frame vehicle driving image data, the next frame vehicle driving image data, and the previous frame vehicle driving image data is a prior art and will not be described in detail here.

[0082] In the previous step 200, f has been obtained 1 and f 2 The common feature point set Ω 12 The three-dimensional coordinates and image coordinates of all common feature points in. 1 、f 2 and f 3 The common feature points of must be in the feature point set Ω 12 So we can get f 1 、f 2 and f 3 The three-dimensional coordinates and image coordinates of the common feature points.

[0083] In the above step (2), the second rotation matrix and the second translation vector are calculated by the following formulas 6 to 8:

[0084]

[0085]

[0086]

[0087] Wherein, a represents the image horizontal coordinate of the common feature points of the current frame vehicle driving image data, the next frame vehicle driving image data of the current frame vehicle driving image data, and the previous frame vehicle driving image data of the current frame vehicle driving image data; b represents the image vertical coordinate of the common feature points of the current frame vehicle driving image data, the next frame vehicle driving image data of the current frame vehicle driving image data, and the previous frame vehicle driving image data of the current frame vehicle driving image data; brepresents the coordinates of the common feature points of the current frame of vehicle driving image data, the next frame of vehicle driving image data of the current frame of vehicle driving image data, and the previous frame of vehicle driving image data of the current frame of vehicle driving image data on the x-axis in the camera coordinate system; b Represents the coordinate of the common feature point on the y-axis in the camera coordinate system; Z b represents the coordinates of the common feature points of the current frame vehicle driving image data, the next frame vehicle driving image data of the current frame vehicle driving image data, and the previous frame vehicle driving image data of the current frame vehicle driving image data on the z-axis in the camera coordinate system; K represents the camera parameter matrix; I represents the unit matrix; R 31 represents the second rotation matrix; t 31 Represents the second translation vector.

[0088] In the above step (3), the optical center is the origin of the camera coordinate system.

[0089] The optical center of the camera at the third position is the origin of the camera coordinate system when the camera is at the third position.

[0090] The first coordinate of the optical center of the camera at the third position in the camera coordinate system is calculated by the following formula:

[0091]

[0092] Among them, R 31 represents the second rotation matrix; t 31 represents the second translation vector; X bc Indicates the coordinate of the first coordinate on the x-axis; Y bc Represents the coordinate of the first coordinate on the y-axis; Z bc Indicates the coordinate of the first coordinate on the z-axis.

[0093] In the above step (4), the first coordinate of the camera is converted to the second coordinate of the optical center of the camera at the third position in the ground coordinate system by the following formula 9:

[0094]

[0095] in, Represents the inverse matrix of the transformation matrix between the camera coordinate system and the ground coordinate system; X bcg Indicates the coordinate of the second coordinate on the x-axis; Y bcg Represents the coordinate of the second coordinate on the y-axis; Z bcg Indicates the coordinate of the second coordinate on the z-axis.

[0096] In the above step (5), the rotation angle of the ground coordinate system and the world coordinate system is calculated by the following formula 10:

[0097]

[0098] Among them, X bc Indicates the coordinate of the second coordinate on the x-axis; Y bcg Represents the coordinate of the second coordinate on the y-axis; Z bcg represents the coordinate of the second coordinate on the z-axis; α represents the rotation angle at which the ground coordinate system coincides with the world coordinate system.

[0099] In the above step (6), the specific value of the vector of the rotation axis between the ground coordinate system and the world coordinate system may be stored in the image processing unit 102 .

[0100] When it is necessary to obtain a transformation matrix from the ground coordinate system to the world coordinate system, the image processing unit 102 may obtain specific values ​​of vectors of the rotation axes of the ground coordinate system and the world coordinate system from the image processing unit 102 itself.

[0101] After obtaining the vectors of the rotation axes of the ground coordinate system and the world coordinate system, the transformation matrix from the ground coordinate system to the world coordinate system can be obtained by the following formula 11:

[0102]

[0103] Among them, R WG represents the transformation matrix from the ground coordinate system to the world coordinate system; I represents the unit matrix; α represents the rotation angle of the ground coordinate system and the world coordinate system; v WG The vector representing the rotation axis of the ground coordinate system and the world coordinate system; The transposed vector of the vector representing the rotation axis of the ground coordinate system with respect to the world coordinate system.

[0104] After obtaining the calibration results of the ground and the vehicle's driving direction through the above steps 200 to 202, the following step 204 may be continued to obtain the transformation relationship between the camera coordinate system and the world coordinate system defined on the vehicle.

[0105] Step 204: Obtain the transformation relationship between the camera coordinate system and the world coordinate system defined on the vehicle according to the calibration results of the ground and the calibration results of the vehicle's driving direction.

[0106] The transformation relationship between the camera coordinate system and the world coordinate system defined on the vehicle is expressed by the following formula 12:

[0107] R WC =R WG *R GC (12)

[0108] Among them, RWC Represents the transformation relationship between the camera coordinate system and the world coordinate system defined on the vehicle; R GC Represents the transformation matrix between the camera coordinate system and the ground coordinate system; R WG Represents the transformation matrix from the ground coordinate system to the world coordinate system.

[0109] In summary, the present embodiment proposes a method for calibrating a vehicle-mounted camera, which processes the current frame vehicle driving image data, the previous frame vehicle driving image data of the current frame vehicle driving image data, and the next frame vehicle driving image data of the current frame vehicle driving image data, to obtain the calibration results of the ground and the calibration results of the vehicle driving direction, and thereby obtain the transformation relationship between the camera coordinate system and the world coordinate system defined on the vehicle. Compared with the method in the related art that requires the use of physical marks to calibrate the positional relationship between the camera, the vehicle and the ground, the transformation relationship between the camera coordinate system and the world coordinate system defined on the vehicle can be calibrated according to actual conditions, thereby improving the calibration accuracy. Therefore, when the panoramic imaging system is actually used, an accurate panoramic image result can be obtained, thereby improving the application experience of the panoramic image.

[0110] Example 2

[0111] This embodiment provides a calibration device for a vehicle-mounted camera, which is used to execute the calibration method for a vehicle-mounted camera provided in the above-mentioned embodiment 1.

[0112] See also Figure 4 The structural diagram of a calibration device for a vehicle-mounted camera shown in FIG. 1 is a schematic diagram of a calibration device for a vehicle-mounted camera, and the calibration device for a vehicle-mounted camera in this embodiment includes:

[0113] The first processing module 300 is used to process the current frame vehicle driving image data and the previous frame vehicle driving image data of the current frame vehicle driving image data, obtain the first position change when the vehicle moves from the first position indicated by the previous frame vehicle driving image data to the second position indicated by the current frame vehicle driving image data, and calibrate the ground according to the obtained first position change to obtain a ground calibration result.

[0114] The second processing module 302 is used to process the current frame vehicle driving image data and the next frame vehicle driving image data of the current frame vehicle driving image data to obtain a second posture change when the vehicle moves from a second position indicated by the current frame vehicle driving image data to a third position indicated by the next frame vehicle driving image data of the current frame vehicle driving image data, and calibrate the vehicle driving direction according to the obtained second posture change and the calibration result of the ground to obtain a calibration result of the vehicle driving direction.

[0115] The third processing module 304 is used to obtain the transformation relationship between the camera coordinate system and the world coordinate system defined on the vehicle according to the calibration result of the ground and the calibration result of the vehicle's driving direction.

[0116] In summary, the present embodiment proposes a calibration device for a vehicle-mounted camera, which processes the current frame vehicle driving image data, the previous frame vehicle driving image data of the current frame vehicle driving image data, and the next frame vehicle driving image data of the current frame vehicle driving image data, to obtain the calibration results of the ground and the calibration results of the vehicle driving direction, and thereby obtain the transformation relationship between the camera coordinate system and the world coordinate system defined on the vehicle. Compared with the method in the related art that requires the use of physical marks to calibrate the positional relationship between the camera, the vehicle and the ground, the transformation relationship between the camera coordinate system and the world coordinate system defined on the vehicle can be calibrated according to actual conditions, thereby improving the calibration accuracy. Therefore, when the panoramic imaging system is actually used, an accurate panoramic image result can be obtained, thereby improving the application experience of the panoramic image.

[0117] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for calibrating a vehicle-mounted camera, characterized in that: include: Processing the vehicle driving image data of the current frame and the vehicle driving image data of the previous frame of the vehicle driving image data of the current frame to obtain a first position change when the vehicle moves from a first position indicated by the previous frame of the vehicle driving image data to a second position indicated by the current frame of the vehicle driving image data, and calibrating the ground according to the obtained first position change to obtain a calibration result of the ground; Processing a current frame of vehicle driving image data and a next frame of vehicle driving image data of the current frame of vehicle driving image data to obtain a second posture change when the vehicle moves from a second position indicated by the current frame of vehicle driving image data to a third position indicated by a next frame of vehicle driving image data of the current frame of vehicle driving image data, and calibrating the vehicle driving direction according to the obtained second posture change and the calibration result of the ground to obtain a calibration result of the vehicle driving direction; According to the calibration results of the ground and the calibration results of the vehicle's driving direction, the transformation relationship between the camera coordinate system and the world coordinate system defined on the vehicle is obtained.

2. The method according to claim 1, characterized in that The first posture change includes: a first rotation matrix and a first translation vector when the vehicle moves from a first position indicated by the previous frame of vehicle driving image data to a second position indicated by the current frame of vehicle driving image data; The ground is calibrated according to the obtained first pose change to obtain the ground calibration result, including: Obtaining camera parameters, and calculating a plane equation of the ground in the camera coordinate system according to the camera parameters, the first rotation matrix, and the first translation vector; According to the obtained plane equation of the ground in the camera coordinate system, the normal vector of the ground in the camera coordinate system is obtained; Get the vector of the X axis of the camera coordinate system, and calculate the vector of the rotation axis between the camera coordinate system and the ground coordinate system based on the normal vector of the ground in the camera coordinate system and the vector of the X axis of the camera coordinate system; According to the normal vector of the ground in the camera coordinate system, the rotation angle of the camera coordinate system and the ground coordinate system is calculated; According to the vector of the rotation axis and the rotation angle, a conversion matrix between the camera coordinate system and the ground coordinate system is obtained, thereby obtaining a ground calibration result.

3. The method according to claim 2, characterized in that According to the normal vector of the ground in the camera coordinate system and the vector of the X axis of the camera coordinate system, the vector of the rotation axis of the camera coordinate system and the ground coordinate system is calculated, including: The vector of the rotation axis between the camera coordinate system and the ground coordinate system is calculated using the following formula: Among them, v T Represents the transposed vector of the rotation axis v between the camera coordinate system and the ground coordinate system; The vector representing the X axis of the camera coordinate system; Represents the normal vector of the ground in the camera coordinate system; A GC express Coordinate on the x-axis; B GC express Coordinate on the y-axis; C GC express The coordinate on the z-axis.

4. The method according to claim 3, characterized in that According to the normal vector of the ground in the camera coordinate system, the rotation angle of the camera coordinate system and the ground coordinate system is calculated, including: The rotation angle between the camera coordinate system and the ground coordinate system is calculated using the following formula: Among them, A GC express Coordinate on the x-axis; B GC express Coordinate on the y-axis; C GC express The coordinate on the y-axis.

5. The method according to claim 3, characterized in that: According to the vector of the rotation axis and the rotation angle, a transformation matrix between the camera coordinate system and the ground coordinate system is obtained, including: The transformation matrix between the camera coordinate system and the ground coordinate system is calculated using the following formula: Among them, R GC represents the transformation matrix between the camera coordinate system and the ground coordinate system; I represents the unit matrix; v u A vector representing the normalized rotation axis; represents the normalized rotation axis v u The transposed vector of the camera coordinate system; θ represents the rotation angle between the camera coordinate system and the ground coordinate system; x u The coordinate of the vector of the rotation axis after normalization on the x-axis; u The coordinate of the vector of the rotation axis after normalization on the y-axis; u The normalized coordinate of the vector of the rotation axis on the z-axis.

6. The method according to claim 5, characterized in that The second posture change includes: a second rotation matrix and a second translation vector when the vehicle moves from a second position indicated by the current frame of vehicle driving image data to a third position indicated by a next frame of vehicle driving image data of the current frame of vehicle driving image data; Processing the current frame of vehicle driving image data and the next frame of vehicle driving image data of the current frame of vehicle driving image data to obtain a second posture change when the vehicle moves from a second position indicated by the current frame of vehicle driving image data to a third position indicated by the next frame of vehicle driving image data of the current frame of vehicle driving image data, and calibrating the vehicle driving direction according to the obtained second posture change and the calibration result of the ground to obtain the calibration result of the vehicle driving direction, including: Determine common feature points of a current frame of vehicle driving image data, a next frame of vehicle driving image data of the current frame of vehicle driving image data, and a previous frame of vehicle driving image data of the current frame of vehicle driving image data; The second rotation matrix and the second translation vector are calculated using the camera parameters, the image coordinates of the common feature points and the three-dimensional coordinates in the camera coordinate system; Calculate a first coordinate of the optical center of the camera at the third position in the camera coordinate system based on the second rotation matrix and the second translation vector; Converting the first coordinate of the camera to the second coordinate of the optical center of the camera at the third position in the ground coordinate system; According to the second coordinate, calculating the rotation angle at which the ground coordinate system coincides with the world coordinate system; The vectors of the rotation axes of the ground coordinate system and the world coordinate system are obtained, and the transformation matrix from the ground coordinate system to the world coordinate system is obtained according to the rotation angle of the ground coordinate system and the world coordinate system and the vectors of the rotation axes of the ground coordinate system and the world coordinate system, thereby obtaining the calibration result of the vehicle's driving direction.

7. The method according to claim 6, characterized in that The second rotation matrix and the second translation vector are calculated using the camera parameters, the image coordinates of the common feature points, and the three-dimensional coordinates in the camera coordinate system, including: The second rotation matrix and the second translation vector are calculated by the following formula: Wherein, a represents the image horizontal coordinate of the common feature points of the current frame vehicle driving image data, the next frame vehicle driving image data of the current frame vehicle driving image data, and the previous frame vehicle driving image data of the current frame vehicle driving image data; b represents the image vertical coordinate of the common feature points of the current frame vehicle driving image data, the next frame vehicle driving image data of the current frame vehicle driving image data, and the previous frame vehicle driving image data of the current frame vehicle driving image data; b represents the coordinates of the common feature points of the current frame of vehicle driving image data, the next frame of vehicle driving image data of the current frame of vehicle driving image data, and the previous frame of vehicle driving image data of the current frame of vehicle driving image data on the x-axis in the camera coordinate system; b Represents the coordinate of the common feature point on the y-axis in the camera coordinate system; Z b represents the coordinates of the common feature points of the current frame vehicle driving image data, the next frame vehicle driving image data of the current frame vehicle driving image data, and the previous frame vehicle driving image data of the current frame vehicle driving image data on the z-axis in the camera coordinate system; K represents the camera parameter matrix; I represents the unit matrix; R 31 represents the second rotation matrix; t 31 Represents the second translation vector.

8. The method according to claim 7, characterized in that Based on the second rotation matrix and the second translation vector, a first coordinate of the optical center of the camera at the third position in the camera coordinate system is calculated, including: The first coordinate of the optical center of the camera at the third position in the camera coordinate system is calculated by the following formula: Among them, R 31 represents the second rotation matrix; t 31 represents the second translation vector; X bc Indicates the coordinate of the first coordinate on the x-axis; Y bc Represents the coordinate of the first coordinate on the y-axis; Z bc Indicates the coordinate of the first coordinate on the z-axis.

9. The method according to claim 8, characterized in that The first coordinate of the camera is converted into the second coordinate of the optical center of the camera at the third position in the ground coordinate system, including: The first coordinate of the camera is converted to the second coordinate of the optical center of the camera at the third position in the ground coordinate system by the following formula: in, Represents the inverse matrix of the transformation matrix between the camera coordinate system and the ground coordinate system; X bcg Indicates the coordinate of the second coordinate on the x-axis; Y bcg Represents the coordinate of the second coordinate on the y-axis; Z bcg Indicates the coordinate of the second coordinate on the z-axis.

10. The method according to claim 9, characterized in that Calculating the rotation angle of the ground coordinate system and the world coordinate system according to the second coordinate, including: The rotation angle of the ground coordinate system and the world coordinate system is calculated using the following formula: Among them, X bcg Indicates the coordinate of the second coordinate on the x-axis; Y bcg Represents the coordinate of the second coordinate on the y-axis; Z bcg represents the coordinate of the second coordinate on the z-axis; α represents the rotation angle at which the ground coordinate system coincides with the world coordinate system.

11. The method according to claim 10, characterized in that According to the rotation angle of the ground coordinate system and the world coordinate system, and the vector of the rotation axis of the ground coordinate system and the world coordinate system, the transformation matrix from the ground coordinate system to the world coordinate system is obtained, including: The transformation matrix from the ground coordinate system to the world coordinate system is obtained by the following formula: Among them, R WG represents the transformation matrix from the ground coordinate system to the world coordinate system; I represents the unit matrix; α represents the rotation angle of the ground coordinate system and the world coordinate system; v WG The vector representing the rotation axis of the ground coordinate system and the world coordinate system; The transposed vector of the vector representing the rotation axis of the ground coordinate system with respect to the world coordinate system.

12. The method according to claim 11, characterized in that According to the calibration results of the ground and the calibration results of the vehicle's driving direction, the transformation relationship between the camera coordinate system and the world coordinate system defined on the vehicle is obtained, including: The transformation relationship between the camera coordinate system and the world coordinate system defined on the vehicle is expressed by the following formula: R WC =R WG *R GC Among them, R WC Represents the transformation relationship between the camera coordinate system and the world coordinate system defined on the vehicle; R GC Represents the transformation matrix between the camera coordinate system and the ground coordinate system; R WG Represents the transformation matrix from the ground coordinate system to the world coordinate system.

13. A calibration device for a vehicle-mounted camera, characterized in that: include: A first processing module is used to process the current frame vehicle driving image data and the previous frame vehicle driving image data of the current frame vehicle driving image data, obtain a first position change when the vehicle moves from a first position indicated by the previous frame vehicle driving image data to a second position indicated by the current frame vehicle driving image data, and calibrate the ground according to the obtained first position change to obtain a ground calibration result; A second processing module is used to process the current frame vehicle driving image data and the next frame vehicle driving image data of the current frame vehicle driving image data to obtain a second posture change when the vehicle moves from a second position indicated by the current frame vehicle driving image data to a third position indicated by the next frame vehicle driving image data of the current frame vehicle driving image data, and calibrate the vehicle driving direction according to the obtained second posture change and the calibration result of the ground to obtain a calibration result of the vehicle driving direction; The third processing module is used to obtain the transformation relationship between the camera coordinate system and the world coordinate system defined on the vehicle according to the calibration results of the ground and the calibration results of the vehicle's driving direction.

Images