Camera extrinsic parameter calibration method and device, vehicle and computer storage medium

Abstract

This invention provides a method, apparatus, vehicle, and computer storage medium for calibrating camera extrinsic parameters. The method involves acquiring a first image captured by a camera and a second image captured by an auxiliary camera. The first image includes a calibration object and an auxiliary camera, while the second image includes a display device. The auxiliary camera is located within the field of view of the camera, and the camera has a first camera coordinate system. The method determines a first coordinate transformation relationship between the camera and the calibration object, and a second coordinate transformation relationship between the auxiliary camera and the display device. Based on the first and second coordinate transformation relationships, a final coordinate transformation relationship between the camera and the display device is determined, and the camera extrinsic parameters are determined based on the final coordinate transformation relationship. This application simplifies the camera extrinsic parameter calibration process.

Description

Technical Field

[0001] This invention relates to the field of camera calibration technology, and in particular to a camera extrinsic parameter calibration method, a camera extrinsic parameter calibration device, a vehicle, and a computer-readable storage medium. Background Technology

[0002] There are two main methods for calibrating camera extrinsic parameters: the specular reflection method and the auxiliary camera method. The specular reflection method requires establishing a complex mathematical model for solution, making its approach very complicated and resulting in inaccurate calibration results. The auxiliary camera method requires an auxiliary camera to observe multiple markers, acting as a medium multiple times. Furthermore, to observe multiple markers, the auxiliary camera must be placed at a sufficient distance, which can lead to very edge-heavy images and incomplete, unclear image acquisition. Therefore, although the auxiliary camera method has a simpler computational approach, its implementation is very difficult, resulting in lower calibration accuracy. Summary of the Invention

[0003] In view of the above problems, embodiments of the present invention are proposed to provide a method for calibrating camera extrinsic parameters, a device for calibrating camera extrinsic parameters, a vehicle, and a computer-readable storage medium to overcome or at least partially solve the above problems.

[0004] In a first aspect, embodiments of the present invention disclose a method for calibrating camera extrinsic parameters. The method includes: acquiring a first image captured by the camera, and a second image captured by an auxiliary camera; the first image includes a calibration object and an auxiliary camera, the second image includes a display device, and the auxiliary camera is located within the field of view of the camera; determining a first coordinate transformation relationship between the camera and the calibration object, and determining a second coordinate transformation relationship between the auxiliary camera and the display device; determining a final coordinate transformation relationship between the camera and the display device based on the first coordinate transformation relationship and the second coordinate transformation relationship, and determining the camera extrinsic parameters based on the final coordinate transformation relationship.

[0005] Optionally, the camera has a first camera coordinate system; determining the first coordinate transformation relationship between the camera and the calibration object includes: determining the first coordinate transformation relationship between the camera and the calibration object based on the calibration object and the first camera coordinate system.

[0006] Optionally, the auxiliary camera has a second camera coordinate system; the display device displays a preset reference object; determining the second coordinate transformation relationship between the auxiliary camera and the display device includes: determining the second coordinate transformation relationship between the auxiliary camera and the display device based on the reference object and the second camera coordinate system.

[0007] Optionally, the calibration object includes a checkerboard pattern; determining the first coordinate transformation relationship between the camera and the calibration object based on the calibration object and the first camera coordinate system of the camera includes: obtaining the first coordinate point of any corner point in the checkerboard pattern; determining the second coordinate point corresponding to the first coordinate point in the first camera coordinate system; calculating the first coordinate point and the second coordinate point using a first preset algorithm to obtain the first coordinate transformation relationship between the auxiliary camera and the camera; the first coordinate transformation relationship includes a first rotation matrix and a first translation matrix for the coordinate transformation between the auxiliary camera and the camera.

[0008] Optionally, determining the second coordinate transformation relationship between the auxiliary camera and the display device based on the reference object and the second camera coordinate system of the auxiliary camera includes: obtaining the third coordinate point of any marker point in the reference object in the reference object coordinate system; determining the fourth coordinate point corresponding to the third coordinate point in the second camera coordinate system; and using the first preset algorithm to calculate the fourth coordinate point and the third coordinate point to obtain the second coordinate transformation relationship between the display device and the auxiliary camera; the second coordinate transformation relationship includes a second rotation matrix and a second translation matrix for the coordinate transformation between the display device and the auxiliary camera.

[0009] Optionally, determining the final coordinate transformation relationship between the camera and the display device based on the first coordinate transformation relationship and the second coordinate transformation relationship includes: using the first preset algorithm to calculate the third coordinate point of any marker point in the reference object, the first coordinate transformation relationship, and the second coordinate transformation relationship to obtain the final coordinate transformation relationship between the display screen and the camera; the final coordinate transformation relationship includes the third rotation matrix and the third translation matrix of the coordinate transformation between the display device and the camera.

[0010] Optionally, the steps of determining the first coordinate transformation relationship between the camera and the calibration object based on the calibration object and the first camera coordinate system, and determining the second coordinate transformation relationship between the auxiliary camera and the display device based on the reference object and the second camera coordinate system, further include: obtaining at least one historical third coordinate transformation relationship; calculating the at least one historical third coordinate transformation relationship using a second preset algorithm to obtain the average value of the historical third coordinate transformation relationship; and calculating the final coordinate transformation relationship between the camera and the display device using the average value of the historical third coordinate transformation relationship.

[0011] Secondly, embodiments of the present invention disclose a camera extrinsic parameter calibration device, the device comprising:

[0012] An acquisition module is configured to acquire a first image captured by the camera and a second image captured by an auxiliary camera; the first image includes a calibration object and an auxiliary camera, the second image includes a display device, and the auxiliary camera is located within the field of view of the camera; a first determination module is configured to determine a first coordinate transformation relationship between the camera and the calibration object, and to determine a second coordinate transformation relationship between the auxiliary camera and the display device; a second determination module is configured to determine a final coordinate transformation relationship between the camera and the display device based on the first coordinate transformation relationship and the second coordinate transformation relationship, and to determine the camera's extrinsic parameters based on the final coordinate transformation relationship.

[0013] Optionally, the first determining module includes: a first determining submodule, used to determine the first coordinate transformation relationship between the camera and the calibration object based on the calibration object and the first camera coordinate system.

[0014] Optionally, the first determining module further includes a second determining submodule, used to determine the second coordinate transformation relationship between the auxiliary camera and the display device based on the reference object and the second camera coordinate system.

[0015] Optionally, the first determining submodule includes: a first acquiring unit, used to acquire the first coordinate point of any corner point in the chessboard; a first determining unit, used to determine the second coordinate point corresponding to the first coordinate point in the second camera coordinate system; and a first calculating unit, used to calculate the first coordinate point and the second coordinate point using a first preset algorithm to obtain the first coordinate transformation relationship between the auxiliary camera and the camera; the first coordinate transformation relationship includes a first rotation matrix and a first translation matrix for the coordinate transformation between the auxiliary camera and the camera.

[0016] Optionally, the second determining submodule further includes: a second acquiring subunit, used to acquire the third coordinate point of any marker point in the reference object in the reference object coordinate system; a second determining unit, used to determine the fourth coordinate point corresponding to the third coordinate point in the second camera coordinate system; and a second calculation submodule, used to calculate the fourth coordinate point and the third coordinate point using the first preset algorithm to obtain the second coordinate transformation relationship between the display device and the auxiliary camera; the second coordinate transformation relationship includes a second rotation matrix and a second translation matrix for the coordinate transformation between the display device and the auxiliary camera.

[0017] Optionally, the second determining module includes: a first calculation submodule, used to calculate the third coordinate point of any marker point in the reference object, the first coordinate transformation relationship and the second coordinate transformation relationship using the first preset algorithm, to obtain the final coordinate transformation relationship between the display screen and the camera; the final coordinate transformation relationship includes the third rotation matrix and the third translation matrix of the coordinate transformation between the display device and the camera.

[0018] Optionally, the second determining module further includes: a first acquisition submodule, used to acquire at least one historical third coordinate transformation relationship; a second calculation submodule, used to calculate the at least one historical third coordinate transformation relationship using a second preset algorithm to obtain the average value of the historical third coordinate transformation relationship; and a third calculation submodule, used to calculate the final coordinate transformation relationship between the camera and the display device using the average value of the historical third coordinate transformation relationship.

[0019] Thirdly, embodiments of the present invention disclose a vehicle, including: a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, it implements the various steps of the camera extrinsic parameter calibration method embodiments as described in the first aspect.

[0020] Fourthly, embodiments of the present invention disclose a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the various steps of the camera extrinsic parameter calibration method embodiments as described in the first aspect.

[0021] The embodiments of the present invention have the following advantages:

[0022] The method acquires a first image captured by the camera and a second image captured by an auxiliary camera. The first image includes a calibration object and an auxiliary camera, and the second image includes a display device. The auxiliary camera is located within the field of view of the camera. A first coordinate transformation relationship between the camera and the calibration object is determined, and a second coordinate transformation relationship between the auxiliary camera and the display device is determined. Based on the first and second coordinate transformation relationships, a final coordinate transformation relationship between the camera and the display device is determined, and the camera's extrinsic parameters are determined based on the final coordinate transformation relationship. In this embodiment, the camera acquires images of the auxiliary camera and the calibration object, and the auxiliary camera acquires images of the display device. By calculating the coordinate relationship between the calibration object and the coordinate system, the transformation relationship between different coordinate systems can be obtained, thereby indirectly achieving the calibration of the camera's extrinsic parameters. This application uses two cameras to independently capture images, and with simple operation, complete and clear images of the display screen and the calibration object can be acquired. By performing simple calculations to obtain the transformation relationship between different coordinate systems and the calibration object, the camera's extrinsic parameters can be calibrated. This method solves the problems of the specular reflection method, which suffers from calculation failure due to its complex calculation approach, and the auxiliary camera method, which suffers from incomplete image acquisition due to operational difficulties.

[0023] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating the steps of a camera extrinsic parameter calibration method provided in an embodiment of the present invention;

[0025] Figure 2 This is a schematic diagram of coordinate transformation between the camera and the reference object provided in an embodiment of the present invention;

[0026] Figure 3A This is a schematic diagram illustrating the coordinate transformation between a display device and a calibration object according to an embodiment of the present invention;

[0027] Figure 3B This is a schematic diagram illustrating another coordinate transformation between a display device and a calibration object provided in an embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram of coordinate transformation between the camera and the reference object provided by existing technology;

[0029] Figure 5 This is a schematic diagram illustrating the coordinate transformation between display devices and markers provided by existing technology;

[0030] Figure 6This is a structural block diagram of a camera extrinsic parameter calibration device provided in an embodiment of the present invention. Detailed Implementation

[0031] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] Reference Figure 1 The diagram illustrates a step-by-step flowchart of a camera extrinsic parameter calibration method according to the present invention, which may specifically include the following steps:

[0033] Step 101: Acquire a first image captured by the camera and a second image captured by the auxiliary camera; the first image includes a calibration object and the auxiliary camera, the second image includes a display device, and the auxiliary camera is located within the field of view of the camera.

[0034] Specifically, the camera takes one picture of the calibration object and the auxiliary camera, obtaining a first image containing both the calibration object and the auxiliary camera; the auxiliary camera takes one picture of the display device and the reference object, obtaining a second image containing both the display device and the reference object. The calibration system acquires the first and second images to obtain the coordinates of points in the images used for calibration, which are then used by the calibration system to calculate the point coordinates. In this embodiment of the invention, the camera can be an infrared camera, a laser rangefinder, etc., the auxiliary camera can be a regular camera, and the display device can be any screen capable of displaying images, such as a display screen. After calibrating the intrinsic parameters of the two cameras according to a preset calibration method, this embodiment of the invention proceeds as follows: Figure 2 The diagram shows the placement of a display device, camera, auxiliary camera, and calibration board. The preset calibration method can be the Zhang Zhengyou calibration method; however, other calibration methods are also applicable to this embodiment, such as the Tsai two-step calibration method and camera calibration methods based on active vision. Adjustments can be made according to actual needs in practical applications, and this embodiment does not impose any limitations on this. The display screen is fixed to the infrared camera, which captures a first image. The first image includes the auxiliary camera and a calibration object. In one embodiment, the calibration object is used to display the image calibrated by the camera, such as a checkerboard pattern, Aruco code, or a solid dot table. The auxiliary camera is fixed to the calibration object. In one embodiment, for example, the auxiliary camera is embedded in the calibration object; the auxiliary camera is bound above the calibration object. The auxiliary camera captures a second image, which includes the display device and a reference object within the display device. The reference object can be any image used for camera calibration, such as a dot table, checkerboard pattern, or Aruco code. To ensure that both cameras can capture clear and complete images, and that operators can easily position them without needing to carefully select angles, the coordinate transformation of the display screen and the calibration object is kept parallel. In one embodiment of the invention, for example: Figure 3AAs shown, the display screen and the calibration object are arranged on the same plane like opposite sides of a parallelogram, so that the tilt angle of the calibration object is small when it is imaged in the image. In one embodiment of the invention, for example: Figure 3B As shown, the display screen and the calibration object are placed on the same plane like two opposite sides of a square, so that the calibration object is not tilted in the image.

[0035] In general technologies, such as Figure 4 As shown, although both a main camera and an auxiliary camera are used, the auxiliary camera needs to simultaneously capture images of the display device and the calibration object, such as... Figure 5 As shown, this can easily lead to excessively tilted images captured by the camera and auxiliary camera, with the images positioned close to the image edges. This results in inaccurate calibration of the camera's extrinsic parameters. Furthermore, to enable the auxiliary camera to simultaneously capture images of both the display device and the calibration object, precise angles must be manually adjusted during placement; otherwise, the auxiliary camera will be unable to capture a complete image. In this embodiment of the invention, as long as the display device and the calibration object are placed parallel to each other, the camera can capture a complete and clear image. It should be noted that... Figures 2-5 In this diagram, the area formed by the straight line emanating from the camera represents the camera's shooting range. This invention can be applied to any location requiring camera calibration, including the cockpit of a vehicle, computers, etc. For example, in a cockpit, the camera can be a webcam capturing the driver's image, and the display device can be the dashboard; in a computer, the camera can be the computer's front-facing webcam, and the display device can be the computer screen.

[0036] Step 102: Determine the first coordinate transformation relationship between the camera and the calibration object, and determine the second coordinate transformation relationship between the auxiliary camera and the display device.

[0037] Specifically, the calibration system acquires the first image, thus obtaining the coordinates of any point on the calibration object in the first camera coordinate system. Since the calibration object is specially manufactured, the coordinates of any point on the calibration object in its coordinate system are already known. After obtaining these two coordinates, the calibration system uses a first preset algorithm to calculate the coordinates of this point in the first camera coordinate system and the calibration object coordinate system, thus obtaining the coordinate system transformation relationship between the camera and the calibration object. This first coordinate transformation relationship is called the first coordinate transformation relationship. Similarly, the reference object on the display screen is also specially manufactured, so the coordinates of each point on the reference object are already known. The calibration system uses the first preset algorithm to calculate the coordinates of this point in the reference object and the coordinates in the second camera coordinate system, thus obtaining the coordinate system transformation relationship between the display device and the auxiliary camera. This second coordinate transformation relationship is called the second coordinate transformation relationship. The transformation relationship between the coordinates is represented by the rotation matrix R and translation matrix T used for calibration from the device to the camera. In this embodiment of the invention, the first preset algorithm can be the PNP (Perspective-n-Point) algorithm. Of course, other algorithms for calculating the conversion from a 3D coordinate system to a 2D coordinate system are also applicable to the embodiments of the present invention, such as the P3P algorithm, the 1P3P algorithm, and optimization iteration methods derived from the PNP algorithm. In practical applications, these algorithms can be adjusted according to actual needs, and the embodiments of the present invention do not impose any restrictions on them.

[0038] Furthermore, in this embodiment of the invention, the camera has a first camera coordinate system;

[0039] Determining the first coordinate transformation relationship between the camera and the calibration object includes:

[0040] The first coordinate transformation relationship between the camera and the calibration object is determined based on the calibration object and the first camera coordinate system.

[0041] Specifically, in this embodiment of the invention, the camera has a first camera coordinate system. The calibration system calculates the coordinate transformation relationship between the camera and the calibration object by taking the coordinates of any point on the calibration object in its own coordinate system and the coordinates of that point in the first camera coordinate system. This coordinate transformation relationship is the first coordinate transformation relationship.

[0042] Furthermore, in this embodiment of the invention, the calibration object includes a checkerboard pattern;

[0043] Determining the first coordinate transformation relationship between the camera and the calibration object based on the first camera coordinate system of the calibration object and the camera includes:

[0044] Obtain the first coordinates of any corner point in the chessboard grid;

[0045] Determine the second coordinate point corresponding to the first coordinate point in the first camera coordinate system;

[0046] The first preset algorithm is used to calculate the first coordinate point and the second coordinate point to obtain the first coordinate transformation relationship between the auxiliary camera and the camera; the first coordinate transformation relationship includes the first rotation matrix and the first translation matrix of the coordinate transformation between the auxiliary camera and the camera.

[0047] Specifically, in this embodiment of the invention, the first image includes an auxiliary camera and a calibration object, the calibration object being used to display the image calibrated by the camera. In this embodiment, the image in the calibration object is a checkerboard pattern. The coordinates of any corner point P in the checkerboard pattern are obtained in the checkerboard coordinate system, that is, the coordinates of point P in the world coordinate system. Since the checkerboard pattern is planar, the Z coordinate of all points is 0; therefore, in this embodiment, the coordinates of the corner point P are set to X. Aux Where, coordinate X Aux Let X be the first coordinate point in the checkerboard grid. Since the auxiliary camera and the calibration object are fixed, the coordinate system of the auxiliary camera and the coordinate system of the checkerboard grid only have a translational relationship. Therefore, the coordinate X can be directly expressed as X. Aux Convert to coordinates X Aux The calibration system determines the X coordinates of P based on the first image. Aux The coordinates X in the first camera coordinate system IR And the first preset algorithm is used to determine the X coordinates of point P in the auxiliary camera. Aux and the coordinates X in the first camera coordinate system IR Calculations can be performed to obtain the first coordinate system transformation relationship from the auxiliary camera coordinate system to the infrared camera coordinate system. Wherein, coordinate X... IR Let be the second coordinate point. This transformation relationship includes the rotation matrix R from the auxiliary camera to the infrared camera. IA And the translation matrix T from the auxiliary camera to the infrared camera IA The calculation formula (1) is X IR =R IA X Aux +T IA In this embodiment of the invention, the first preset algorithm may be the PNP algorithm.

[0048] In one embodiment of the present invention, the calibration system acquires a first image and selects point P of the checkerboard pattern in the first image. Since the checkerboard pattern is specially made, the coordinates of point P have already been determined to be X. Aux Since the auxiliary camera is embedded in the calibration object, the coordinates (X, y) of point P in the second camera coordinate system can be obtained by coordinate system translation. Aux The PNP algorithm is used to calculate the coordinates X. Aux and coordinate X IRBy performing calculations, the rotation matrix R from the auxiliary camera to the infrared camera can be obtained. IA And the translation matrix T from the auxiliary camera to the infrared camera IA .

[0049] The conventional auxiliary camera method, while requiring two cameras to acquire images, involves calculating multiple transformation relationships between the calibration object and the coordinate system because the cameras are not bound to the calibration object. This increases the computational workload and increases the risk of calculation errors leading to inaccurate camera extrinsic parameter calibration. This invention fixes the auxiliary camera to the calibration object, allowing for coordinate system transformations to be performed only by translation, eliminating the need to calculate the transformation relationship between the calibration plate and the coordinate system. This significantly reduces the workload and increases the accuracy of camera extrinsic parameter calibration.

[0050] Furthermore, in this embodiment of the invention, the auxiliary camera has a second camera coordinate system; the display device displays a preset reference object;

[0051] Determining the second coordinate transformation relationship between the auxiliary camera and the display device includes:

[0052] The second coordinate transformation relationship between the auxiliary camera and the display device is determined based on the reference object and the second camera coordinate system.

[0053] Specifically, in this embodiment of the invention, the auxiliary camera has a second camera coordinate system. The display device is used to display a preset reference object, and the second image acquired by the auxiliary camera includes both the display device and the preset reference object. The calibration system calculates the coordinates of any point on the reference object in its own coordinate system and the coordinates of that point in the second camera coordinate system to determine the coordinate transformation relationship between the display device and the auxiliary camera. This coordinate transformation relationship is the second coordinate transformation relationship.

[0054] Furthermore, in this embodiment of the invention, determining the second coordinate transformation relationship between the auxiliary camera and the display device based on the reference object and the second camera coordinate system of the auxiliary camera includes:

[0055] Obtain the third coordinate point of any marker point in the reference object in the reference object coordinate system;

[0056] Determine the fourth coordinate point corresponding to the third coordinate point in the second camera coordinate system;

[0057] The first preset algorithm is used to calculate the fourth coordinate point and the third coordinate point to obtain the second coordinate transformation relationship between the display device and the auxiliary camera; the second coordinate transformation relationship includes the second rotation matrix and the second translation matrix of the coordinate transformation between the display device and the auxiliary camera.

[0058] Specifically, in this embodiment of the invention, although the camera and display device are fixed, the camera is not embedded in the calibration object like an auxiliary camera, but is separate from the display device. Therefore, the auxiliary camera only needs to capture images of the display device and the reference object within it. In one embodiment of the invention, for example, the reference object can be a dotted board. Since the reference object in the display device is specially made, the coordinates of each point in the reference object have been determined. The calibration system selects any point Q of the dotted board in the second image and determines the coordinates of point Q in the reference object coordinate system, that is, the coordinates of point Q in the world coordinate system. The third coordinate point is the coordinate of point Q in the reference object coordinate system. Similarly, the image displayed by the reference object is a planar image, therefore the Z coordinate of point Q is 0. In this embodiment of the invention, the coordinates of point Q are set to X... Screen Where, coordinate X Screen This is the third coordinate point. After determining point Q, the calibration system obtains the X coordinates of point Q in the second camera coordinate system based on the second image. Aux The fourth coordinate point is the X coordinate of point Q in the second camera coordinate system. Aux The calibration system uses the first preset algorithm to determine the X coordinates of point Q in the reference coordinate system. Screen and the coordinates X in the second camera coordinate system Aux Calculations can be performed to obtain the transformation relationship between the display device coordinate system and the second camera coordinate system. This transformation relationship includes the rotation matrix R between the display device and the auxiliary camera. AS Translation matrix T of display device and auxiliary camera AS The calculation formula (2) is X Aux =R AS X Screen +T AS In this embodiment of the invention, the first preset algorithm may be the PNP algorithm.

[0059] Step 103: Determine the final coordinate transformation relationship between the camera and the display device based on the first coordinate transformation relationship and the second coordinate transformation relationship, and determine the camera extrinsic parameters based on the final coordinate transformation relationship.

[0060] Specifically, in this embodiment of the invention, based on the obtained first and second coordinate transformation relationships, a first preset algorithm is used to calculate and indirectly obtain the transformation relationship between the display device coordinate system and the camera coordinate system. This transformation relationship is the final coordinate system transformation relationship, including the rotation matrix R from the display device coordinate system to the camera coordinate system and the translation matrix T from the display device coordinate system to the camera coordinate system. In this embodiment of the invention, the first preset algorithm can be the PNP (Perspective-n-Point) algorithm.

[0061] In general technologies, because the auxiliary camera needs to capture too many images, multiple calculations are required to determine the camera's extrinsic parameters when the auxiliary camera acts as an intermediary. This application only requires a single capture to obtain the first coordinate transformation relationship from the display screen coordinate system to the auxiliary camera coordinate system and the second coordinate transformation relationship from the auxiliary camera coordinate system to the camera coordinate system, thus determining the final coordinate transformation relationship from the display screen to the auxiliary camera coordinate system. This requires only one transformation using the auxiliary camera as an intermediate medium.

[0062] Furthermore, in this embodiment of the invention, determining the final coordinate transformation relationship between the camera and the display device based on the first coordinate transformation relationship and the second coordinate transformation relationship includes:

[0063] The first preset algorithm is used to calculate the third coordinate point of any marker point in the reference object, the first coordinate transformation relationship, and the second coordinate transformation relationship to obtain the final coordinate transformation relationship between the display screen and the camera; the final coordinate transformation relationship includes the third rotation matrix and the third translation matrix of the coordinate transformation between the display device and the camera.

[0064] Specifically, in this embodiment of the invention, the first preset algorithm is used to determine the coordinates of point Q as X. Screen Rotation matrix R from auxiliary camera to infrared camera IA Translation matrix T from auxiliary camera to infrared camera IA Rotation matrix R from display screen to auxiliary camera AS Translation matrix T from display screen to auxiliary camera AS By performing calculations, we can obtain the rotation matrix R and translation matrix T from the display device to the camera. The calculation formula (3) is X IR =R IA R AS X Screen +R IA T As +T IA That is, R = R IA R AS T = R IA T AS +T IA .

[0065] Furthermore, in this embodiment of the invention, determining the first coordinate transformation relationship between the camera and the calibration object based on the calibration object and the first camera coordinate system, and determining the second coordinate transformation relationship between the auxiliary camera and the display device based on the reference object and the second camera coordinate system, further includes:

[0066] Obtain at least one historical third coordinate transformation relationship;

[0067] The at least one third coordinate transformation relationship is calculated using a second preset algorithm to obtain the average value of the third coordinate transformation relationship;

[0068] The final coordinate transformation relationship between the camera and the display device is calculated using the average value of the third coordinate transformation relationship.

[0069] Specifically, to make the calibration of external parameters more accurate, in this embodiment of the invention, the external parameters of the camera can be determined by a single shot or by multiple shots. If multiple shots are used to determine the final transformation relationship, the calibration system needs to obtain the external parameters that have already been determined by the shots for calculation. Among them, the external parameters that have already been determined by the shots are the historical third coordinate transformation relationship, including: the rotation matrix and translation matrix of the coordinate transformation between the display device and the camera determined in the first shot; the rotation matrix and translation matrix of the coordinate transformation between the display device and the camera determined in the second shot; ... the rotation matrix and translation matrix of the coordinate transformation between the display device and the camera determined in the Nth shot. Finally, the average value of the historical third coordinate transformation relationship obtained multiple times is used as the final coordinate transformation relationship from the display device to the camera. The calculation formula (4) can be: In this process, the same pose is used for each shot, and the position and rotation angle of the camera, auxiliary camera, display device, and calibration object do not change. However, the coordinates of the selected point can be different or the same each time.

[0070] In one embodiment of the present invention, the calibration system acquires external parameters determined by N shooting sessions, for example: acquiring external parameters r1 and t1 determined by the first shooting session; acquiring external parameters r2 and t2 determined by the second shooting session; acquiring external parameters r3 and t3 determined by the third shooting session; ... acquiring external parameters r1 and t2 determined by the Nth shooting session. n t n After obtaining the parameters mentioned above, the calibration system uses formula (4) to calculate the average value of the external parameters, and uses the average value of the external parameters as the final coordinate transformation relationship from the display device to the camera.

[0071] The technical solution of this application embodiment acquires a first image captured by the camera and a second image captured by an auxiliary camera. The first image includes a calibration object and an auxiliary camera, and the second image includes a display device. The auxiliary camera is located within the field of view of the camera. A first coordinate transformation relationship between the camera and the calibration object is determined, and a second coordinate transformation relationship between the auxiliary camera and the display device is determined. Based on the first and second coordinate transformation relationships, a final coordinate transformation relationship between the camera and the display device is determined, and the camera extrinsic parameters are determined based on the final coordinate transformation relationship. In this embodiment, the camera acquires images of the auxiliary camera and the calibration object, and the auxiliary camera acquires images of the display device. By calculating the calibration object and the coordinate system, the transformation relationship between different coordinate systems can be obtained, thereby indirectly achieving the calibration of the camera extrinsic parameters. This application uses two cameras to independently capture images. With simple operation, complete and clear images of the display screen and the calibration object can be acquired. By performing simple calculations, the transformation relationship between different coordinate systems and the calibration object can be obtained, thus completing the calibration of the camera extrinsic parameters. This method solves the problems of the specular reflection method, which suffers from calculation failure due to its complex calculation approach, and the auxiliary camera method, which suffers from incomplete image acquisition due to operational difficulties.

[0072] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0073] Reference Figure 6 The diagram illustrates a structural block diagram of a camera extrinsic parameter calibration device according to an embodiment of the present invention, which may specifically include the following modules: an acquisition module 601, a first determination module 602, and a second determination module 603, wherein:

[0074] The acquisition module 601 is used to acquire a first image captured by the camera and a second image captured by an auxiliary camera; the first image includes a calibration object and an auxiliary camera, the second image includes a display device, and the auxiliary camera is located within the field of view of the camera;

[0075] The first determining module 602 is used to determine a first coordinate transformation relationship between the camera and the calibration object, and to determine a second coordinate transformation relationship between the auxiliary camera and the display device;

[0076] The second determining module 603 is used to determine the final coordinate transformation relationship between the camera and the display device based on the first coordinate transformation relationship and the second coordinate transformation relationship, and to determine the camera extrinsic parameters based on the final coordinate transformation relationship.

[0077] In this embodiment of the invention, the first determining module includes:

[0078] The first determining submodule is used to determine the first coordinate transformation relationship between the camera and the calibration object based on the calibration object and the first camera coordinate system.

[0079] In this embodiment of the invention, the first determining module further includes:

[0080] The second determining submodule is used to determine the second coordinate transformation relationship between the auxiliary camera and the display device based on the reference object and the second camera coordinate system.

[0081] In this embodiment of the invention, the first determining submodule further includes:

[0082] The first acquisition unit is used to acquire the first coordinate point of any corner point in the chessboard grid;

[0083] The first determining unit is used to determine the second coordinate point corresponding to the first coordinate point in the second camera coordinate system;

[0084] The first calculation unit is used to calculate the first coordinate point and the second coordinate point using a first preset algorithm to obtain a first coordinate transformation relationship between the auxiliary camera and the camera; the first coordinate transformation relationship includes a first rotation matrix and a first translation matrix for the coordinate transformation between the auxiliary camera and the camera.

[0085] In this embodiment of the invention, the second determining submodule includes:

[0086] The second acquisition unit is used to acquire the third coordinate point of any marker point in the reference object in the reference object coordinate system;

[0087] The second determining unit is used to determine the fourth coordinate point corresponding to the third coordinate point in the second camera coordinate system;

[0088] The second calculation unit is used to calculate the fourth coordinate point and the third coordinate point using the first preset algorithm to obtain the second coordinate transformation relationship between the display device and the auxiliary camera; the second coordinate transformation relationship includes the second rotation matrix and the second translation matrix of the coordinate transformation between the display device and the auxiliary camera.

[0089] In this embodiment of the invention, the second determining module includes:

[0090] The first calculation submodule is used to calculate the third coordinate point, the first coordinate transformation relationship, and the second coordinate transformation relationship of any marker point in the reference object using the first preset algorithm, so as to obtain the final coordinate transformation relationship between the display screen and the camera; the final coordinate transformation relationship includes the third rotation matrix and the third translation matrix of the coordinate transformation between the display device and the camera.

[0091] In this embodiment of the invention, the second determining module further includes:

[0092] The first acquisition submodule is used to acquire at least one historical third coordinate transformation relationship;

[0093] The second calculation submodule is used to calculate the at least one third coordinate transformation relationship using a second preset algorithm to obtain the average value of the third coordinate transformation relationship;

[0094] The third calculation submodule is used to calculate the final coordinate transformation relationship between the camera and the display device by using the average value of the third coordinate transformation relationship.

[0095] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0096] This invention also provides a vehicle for calibrating the extrinsic parameters of a driver's portrait camera acquired in the vehicle's cockpit based on the above-described calibration method, including:

[0097] It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, it implements the various processes of the above-described camera extrinsic parameter calibration method embodiment and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0098] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described camera extrinsic parameter calibration method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0099] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0100] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0101] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable information processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable information processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0102] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable information processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0103] These computer program instructions can also be loaded onto a computer or other programmable information processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0104] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0105] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0106] The present invention has been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for calibrating camera extrinsic parameters, characterized in that, The method comprises: acquiring a first image captured by a camera and a second image captured by an auxiliary camera; the first image comprises a calibration object and the auxiliary camera, the second image comprises a display device, the auxiliary camera is in a field of view of the camera, the camera and the display device are located on the same side; a preset reference object is displayed in the display device; the auxiliary camera is fixed with the calibration object, and the display device is parallel to the calibration object; determining a first coordinate conversion relationship between the camera and the calibration object, and determining a second coordinate conversion relationship between the auxiliary camera and the display device; determining a final coordinate conversion relationship between the camera and the display device based on the first coordinate conversion relationship and the second coordinate conversion relationship, and determining an extrinsic parameter of the camera based on the final coordinate conversion relationship; the determination of the final coordinate conversion relationship between the camera and the display device based on the first coordinate conversion relationship and the second coordinate conversion relationship comprises: acquiring a third coordinate point of any mark point in the reference object in a reference object coordinate system; calculating the third coordinate point of any mark point in the reference object, the first coordinate conversion relationship and the second coordinate conversion relationship by using a first preset algorithm to obtain the final coordinate conversion relationship between the display device and the camera; the final coordinate conversion relationship comprises a third rotation matrix and a third translation matrix of the coordinate conversion between the display device and the camera.

2. The camera extrinsic parameter calibration method of claim 1, wherein, the camera has a first camera coordinate system; the determination of the first coordinate conversion relationship between the camera and the calibration object comprises: determining the first coordinate conversion relationship between the camera and the calibration object based on the calibration object and the first camera coordinate system.

3. The method of claim 1, wherein, the auxiliary camera has a second camera coordinate system; the determination of the second coordinate conversion relationship between the auxiliary camera and the display device comprises: determining the second coordinate conversion relationship between the auxiliary camera and the display device based on the reference object and the second camera coordinate system.

4. The method of claim 2, wherein, the calibration object comprises a checkerboard; the determination of the first coordinate conversion relationship between the camera and the calibration object based on the calibration object and the first camera coordinate system of the camera comprises: acquiring a first coordinate point of any corner point in the checkerboard; determining a second coordinate point corresponding to the first coordinate point in the first camera coordinate system; calculating the first coordinate point and the second coordinate point by using a first preset algorithm to obtain a first coordinate conversion relationship between the auxiliary camera and the camera; the first coordinate conversion relationship comprises a first rotation matrix and a first translation matrix of the coordinate conversion between the auxiliary camera and the camera.

5. The method of calibrating camera extrinsics according to claim 3, wherein, the determination of the second coordinate conversion relationship between the auxiliary camera and the display device based on the reference object and the second camera coordinate system of the auxiliary camera comprises: determining a fourth coordinate point corresponding to the third coordinate point in the second camera coordinate system; The first preset algorithm is used to calculate the fourth coordinate point and the third coordinate point, to obtain a second coordinate conversion relationship between the display device and the auxiliary camera; the second coordinate conversion relationship includes a second rotation matrix and a second translation matrix of the coordinate conversion between the display device and the auxiliary camera.

6. The method of calibrating camera extrinsics according to claim 1, wherein, The method further includes: obtaining at least one historical third coordinate conversion relationship; using a second preset algorithm to calculate the at least one historical third coordinate conversion relationship, to obtain an average value of the historical third coordinate conversion relationship; using the average value of the historical third coordinate conversion relationship to calculate a final coordinate conversion relationship between the camera and the display device.

7. An apparatus for calibrating camera extrinsic parameters, characterized in that, The apparatus includes: an obtaining module, configured to obtain a first image captured by a camera and a second image captured by an auxiliary camera; the first image includes a calibration object and the auxiliary camera, and the second image includes a display device; the auxiliary camera is in a field of view of the camera, and the camera and the display device are located on the same side; the display device displays a preset reference object; the auxiliary camera is fixed with the calibration object, and the display device is parallel to the calibration object; a first determining module, configured to determine a first coordinate conversion relationship between the camera and the calibration object, and determine a second coordinate conversion relationship between the auxiliary camera and the display device; a second determining module, configured to determine a final coordinate conversion relationship between the camera and the display device based on the first coordinate conversion relationship and the second coordinate conversion relationship, and determine an external parameter of the camera based on the final coordinate conversion relationship; the second determining module includes: a second obtaining unit, configured to obtain a third coordinate point of any mark point in the reference object in a reference object coordinate system; a first calculating sub-module, configured to use a first preset algorithm to calculate the third coordinate point of any mark point in the reference object, the first coordinate conversion relationship and the second coordinate conversion relationship, to obtain a final coordinate conversion relationship between the display device and the camera; the final coordinate conversion relationship includes a third rotation matrix and a third translation matrix of the coordinate conversion between the display device and the camera.

8. A vehicle characterized by comprising: The apparatus includes: a processor, a memory, and a computer program stored in the memory and capable of running on the processor; when the computer program is executed by the processor, the steps of the camera external parameter calibration method according to any one of claims 1-6 are implemented.

9. A computer-readable storage medium, characterized in that, A computer program is stored in the computer readable storage medium; when the computer program is executed by the processor, the steps of the camera external parameter calibration method according to any one of claims 1-6 are implemented.

Images