Camera internal and external parameter calibration method, device, equipment, storage medium and program product
By acquiring the camera's captured images and target coordinates, combined with internal and external parameter optimization, the problem of large errors in camera calibration in the existing technology is solved, and higher precision camera calibration is achieved, and AVM circumferential image system and 3D reconstruction are improved. accuracy and user experience.
Patent Information
- Application Number
- CN202410469809.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-18
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-04-18
AI Technical Summary
The existing camera internal and external parameter calibration methods have large errors, which affects the user experience and accuracy of AVM surround view imaging systems and 3D reconstruction products. Especially in the context of the development of ADAS autonomous driving technology, higher accuracy is required.
By obtaining the camera's captured image, determining the actual coordinates of the target in the pixel coordinate system and the world coordinate system, combining the internal parameter set, calculating and optimizing the camera's external parameter matrix, selecting the optimal internal parameter and external parameter matrix to reduce scene restoration errors. Achieve more accurate positioning and object detection.
It improves the accuracy of internal and external parameters calibration of the camera, reduces the camera's restoration error for the captured scene, makes positioning and target detection more accurate, and is suitable for application scenarios such as surround view images, autonomous driving and map construction.
Smart Images

Figure CN118247360B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a calibration method, device, equipment, storage medium and program product for camera internal and external parameters. Background Art
[0002] The AVM (Around View Monitor) surround image system consists of four external fisheye cameras installed on the front bumper, trunk, left and right rearview mirrors. The system includes the following processes in order: calibration point detection and distortion correction, four-way fisheye camera joint calibration, projection transformation, splicing and fusion of bird's-eye views, 3D model texture mapping, etc. The image information captured by the four-way fisheye camera is processed through the above process to generate a 2D and 3D panoramic view.
[0003] Camera calibration is the process of solving the camera intrinsic parameter matrix, distortion coefficients and camera extrinsic parameter matrix. Since there are inevitable deviations in the production process of the camera, it is necessary to calibrate the camera intrinsic parameter matrix. In addition, when reconstructing a three-dimensional scene using images taken by the camera, it is necessary to know the position of the camera in three-dimensional space. The position of the camera in three-dimensional space is the camera extrinsic parameter matrix.
[0004] At present, the camera intrinsic parameter matrix is usually written by the camera manufacturer during the production process when the camera is produced (there will be a certain error compared with the actual camera intrinsic parameter matrix, generally within 15 pixels), or the Zhang Zhengyou calibration method is used to shoot the chessboard with the camera from multiple angles, and use a sufficient number of sets of mapping relationships between world coordinate points and image coordinate points to fit a set of optimal parameters. This set of parameters includes the camera intrinsic parameter set and distortion coefficients.
[0005] For the camera extrinsic parameter matrix, the default intrinsic parameter matrix or the intrinsic parameter matrix written when the camera is produced is generally used to perform a separate extrinsic parameter matrix calibration. However, at this time, under the premise that the intrinsic intrinsic parameter matrix itself has certain errors, the calibrated extrinsic parameter matrix cannot guarantee accuracy, but will amplify the error.
[0006] In the AWM surround view imaging system, the error in the calibration of the internal and external parameters of the fisheye camera will directly affect the user experience of AVM and other camera 3D reconstruction products and the accuracy of restoring the three-dimensional space. With the development of ADAS autonomous driving technology, higher accuracy requirements are put forward for the internal and external parameter calibration system of the AVM camera. Summary of the invention
[0007] In view of this, the present invention provides a method, device, equipment, storage medium and program product for calibrating the intrinsic and extrinsic parameters of a camera to improve the accuracy of the calibration of the intrinsic and extrinsic parameters of the camera.
[0008] In a first aspect, the present invention provides a method for calibrating internal and external parameters of a camera, the method comprising the following steps: acquiring a captured image of a camera; determining a first actual coordinate of a target in a pixel coordinate system and a second actual coordinate of the target in a world coordinate system based on the captured image; acquiring an internal parameter set of the camera, wherein the internal parameter set includes multiple groups of internal parameter matrices; determining an external parameter set of the camera based on the first actual coordinate of the target in the pixel coordinate system, the second actual coordinate of the target in the world coordinate system, and the internal parameter set, wherein the external parameter set includes multiple groups of external parameter matrices, and each group of internal parameter matrices corresponds to a group of external parameter matrices; determining multiple calculated coordinates of the target in the world coordinate system based on the external parameter set of the camera, wherein each group of external parameter matrices corresponds to one calculated coordinate; selecting the optimal calculated coordinate from the multiple calculated coordinates, and using a group of internal parameter matrices corresponding to the optimal calculated coordinates as the target internal parameters of the camera, and using a group of external parameter matrices corresponding to the optimal calculated coordinates as the target external parameters of the camera.
[0009] The present invention provides a method for calibrating camera internal and external parameters, which obtains a captured image of a camera, determines a first actual coordinate of a target in a pixel coordinate system and a second actual coordinate of the target in a world coordinate system according to the captured image, determines an external parameter set of the camera according to the first actual coordinate of the target in the pixel coordinate system, the second actual coordinate of the target in the world coordinate system and an internal parameter set of the camera, and determines a plurality of calculated coordinates of the target in the world coordinate according to the external parameter set of the camera; selects an optimal calculated coordinate from the plurality of calculated coordinates, and uses a set of internal parameter matrices corresponding to the optimal calculated coordinate as The target intrinsic parameters of the camera take a set of extrinsic parameter matrices corresponding to the optimal calculation coordinates as the target extrinsic parameters of the camera; it can be seen that in the present invention, the intrinsic parameter matrix written by the camera manufacturer or the intrinsic parameter matrix determined by the Zhang Zhengyou calibration method is not directly used as the target internal parameter, and the determination of the target external parameter does not only rely on the camera intrinsic parameter matrix, but selects the optimal calculation coordinates from multiple calculation coordinates, where the multiple calculation coordinates are obtained according to the camera intrinsic parameter matrix and the camera extrinsic parameter matrix, that is, the camera's restoration error of the captured scene is reduced by jointly optimizing the camera intrinsic parameter matrix and the camera extrinsic parameter matrix, so that positioning and target detection are more accurate.
[0010] In an optional embodiment, determining the external parameter set of the camera according to the first actual coordinates of the target in the pixel coordinate system, the second actual coordinates of the target in the world coordinate system, and the internal parameter set includes: obtaining the external parameter set of the camera using a PNP algorithm according to the first actual coordinates of the target in the pixel coordinate system, the second actual coordinates of the target in the world coordinate system, and the internal parameter set.
[0011] In this way, the camera's external parameter set can be accurately obtained.
[0012] In an optional embodiment, determining multiple calculated coordinates of the target in world coordinates based on the camera's extrinsic parameter set includes: obtaining a first distance from the camera to the ground; determining a third actual coordinate of the target in the camera coordinate system based on the first actual coordinate of the target in the pixel coordinate system; determining a depth coefficient of the target based on the third actual coordinate of the target in the camera coordinate system and the first distance; determining multiple calculated coordinates of the target in world coordinates based on the third actual coordinate of the target in the camera coordinate system, the depth coefficient of the target and the camera's extrinsic parameter set.
[0013] In this way, multiple calculated coordinates of the target point in the world coordinate system can be accurately obtained.
[0014] In an optional embodiment, determining the depth coefficient of the target point based on the third actual coordinates of the target point in the camera coordinate system and the first distance includes: calculating the first vector between the coordinate point of the camera and the third actual coordinates of the target point in the camera coordinate system; calculating the dot product of the unit normal vector perpendicular to the ground and the first vector to obtain the first distance; dividing the second distance by the first distance to obtain the depth coefficient of the target point.
[0015] This allows the depth factor of the target to be accurately determined.
[0016] In an optional implementation, selecting the optimal calculation coordinates from a plurality of calculation coordinates includes: acquiring an operation mode of the camera; determining an optimization target according to the operation mode; and selecting the optimal calculation coordinates from a plurality of calculation coordinates according to the optimization target.
[0017] Therefore, different optimization goals can be set according to different camera application scenarios.
[0018] In an optional embodiment, determining the optimization target according to the operating mode includes: when the operating mode of the camera is a surround view mode, taking the projection error of the three-dimensional world coordinate point in the common view area of multiple cameras as the optimization target; when the operating mode of the camera is an automatic driving mode or a map building mode, taking the error between the calculated coordinates and the second actual coordinates as the optimization target.
[0019] This allows for more accurate positioning and target detection when using surround view imaging, autonomous driving, or map building.
[0020] In a second aspect, the present invention also provides a calibration device for the internal and external parameters of a camera, the device comprising a captured image acquisition module, a captured image processing module, an internal parameter set acquisition module, an external parameter set calculation module, a world coordinate calculation module and a preferred module; the captured image acquisition module is used to acquire the captured image of the camera; the captured image processing module is used to determine the first actual coordinates of the target in the pixel coordinate system and the second actual coordinates of the target in the world coordinate system according to the captured image; the internal parameter set acquisition module is used to acquire the internal parameter set of the camera, wherein the internal parameter set includes multiple groups of internal parameter matrices; the external parameter set calculation module is used to calculate the external parameter set according to the pixel coordinate system of the target. The first actual coordinate in the reference system, the second actual coordinate in the world coordinate system of the target point, and the internal parameter set determine the external parameter set of the camera, wherein the external parameter set includes multiple groups of external parameter matrices, and each group of internal parameter matrices corresponds to a group of external parameter matrices; a world coordinate calculation module is used to determine multiple calculation coordinates of the target point in the world coordinate according to the external parameter set of the camera, wherein each group of external parameter matrices corresponds to one calculation coordinate; a selection module is used to select the optimal calculation coordinate from the multiple calculation coordinates, and use a group of internal parameter matrices corresponding to the optimal calculation coordinates as the target internal parameters of the camera, and use a group of external parameter matrices corresponding to the optimal calculation coordinates as the target external parameters of the camera.
[0021] In a third aspect, the present invention further provides a computer device, comprising a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the camera internal and external parameter calibration method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.
[0022] In a fourth aspect, the present invention further provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the camera internal and external parameter calibration method of the first aspect or any corresponding embodiment thereof.
[0023] In a fifth aspect, the present invention further provides a computer program product, comprising computer instructions, wherein the computer instructions are used to enable a computer to execute the camera internal and external parameter calibration method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0025] Figure 1 is a flow chart of a camera internal and external parameter calibration method according to an embodiment of the present invention;
[0026] Figure 2 is a flow chart of another camera internal and external parameter calibration method according to an embodiment of the present invention;
[0027] Figure 3 is a schematic diagram of a method for determining the calculated coordinates of a target point according to an embodiment of the present invention;
[0028] Figure 4 is a flow chart of yet another camera internal and external parameter calibration method according to an embodiment of the present invention;
[0029] Figure 5 is a schematic diagram of a first optimization target according to an embodiment of the present invention;
[0030] Figure 6 is a schematic diagram of a second optimization objective according to an embodiment of the present invention;
[0031] Figure 7 is a structural block diagram of a camera internal and external parameter calibration device according to an embodiment of the present invention;
[0032] Figure 8 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0033] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0034] According to an embodiment of the present invention, an embodiment of a method for calibrating intrinsic and extrinsic parameters of a camera is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0035] In this embodiment, a method for calibrating internal and external parameters of a camera is provided, which can be used in computer equipment. Figure 1 is a flow chart of a camera internal and external parameter calibration method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:
[0036] Step S101: Acquire an image captured by a camera.
[0037] Step S102: determining the first actual coordinates of the target point in the pixel coordinate system and the second actual coordinates of the target point in the world coordinate system according to the captured image.
[0038] In the embodiment of the present invention, a plurality of target points are included in the captured image.
[0039] Step S103: Acquire the camera's internal parameter set, where the internal parameter set includes multiple groups of internal parameter matrices.
[0040] Specifically, the camera's internal parameter set can be generated according to the coefficients of the camera's initial internal parameter matrix (including the camera's internal parameter set and the distortion coefficient) and the camera's internal parameter matrix error allowable range. The internal parameter set includes multiple groups of internal parameter matrices. A group of internal parameter matrices can be represented by K, and a group of internal parameter matrices includes multiple coefficients. In other words, the internal parameter set includes multiple Ks, for example, the internal parameter set includes K1, K2, ... K n .
[0041] Step S104: Determine the camera's external parameter set according to the target's first actual coordinates in the pixel coordinate system, the target's second actual coordinates in the world coordinate system, and the internal parameter set, wherein the external parameter set includes multiple groups of external parameter matrices, and each group of internal parameter matrices corresponds to a group of external parameter matrices.
[0042] Specifically, the external parameter set includes multiple groups of external parameter matrices, one group of external parameter matrices includes R and t, where R is the rotation matrix R from the 3D world coordinate system to the 3D camera coordinate system, and t is the representation of the vector from the origin of the 3D camera coordinate system to the origin of the 3D world coordinate system in the camera coordinate system. In other words, the external parameter set includes multiple groups (R, t), for example, the external parameter set includes (R1, t1), (R2, t2)...(R n , t n ).
[0043] Step S105: Determine multiple calculated coordinates of the target point in the world coordinate system according to the camera's extrinsic parameter set, wherein each set of extrinsic parameter matrices corresponds to one calculated coordinate.
[0044] Specifically, when the external parameter set includes n groups (R, t), for example (R1, t1), (R2, t2) ... (Rn, tn), n calculated coordinates can be obtained according to step S105.
[0045] Step S106: Select the optimal calculation coordinates from the multiple calculation coordinates, use a set of internal parameter matrices corresponding to the optimal calculation coordinates as the target internal parameters of the camera, and use a set of external parameter matrices corresponding to the optimal calculation coordinates as the target external parameters of the camera.
[0046] The present invention provides a method for calibrating camera internal and external parameters, which obtains a captured image of a camera, determines a first actual coordinate of a target in a pixel coordinate system and a second actual coordinate of the target in a world coordinate system according to the captured image, determines an external parameter set of the camera according to the first actual coordinate of the target in the pixel coordinate system, the second actual coordinate of the target in the world coordinate system and an internal parameter set of the camera, and determines a plurality of calculated coordinates of the target in the world coordinate according to the external parameter set of the camera; selects an optimal calculated coordinate from the plurality of calculated coordinates, and uses a set of internal parameter matrices corresponding to the optimal calculated coordinate as The target intrinsic parameters of the camera take a set of extrinsic parameter matrices corresponding to the optimal calculation coordinates as the target extrinsic parameters of the camera; it can be seen that in the present invention, the intrinsic parameter matrix written by the camera manufacturer or the intrinsic parameter matrix determined by the Zhang Zhengyou calibration method is not directly used as the target internal parameter, and the determination of the target external parameter does not only rely on the camera intrinsic parameter matrix, but selects the optimal calculation coordinates from multiple calculation coordinates, where the multiple calculation coordinates are obtained according to the camera intrinsic parameter matrix and the camera extrinsic parameter matrix, that is, the camera's restoration error of the captured scene is reduced by jointly optimizing the camera intrinsic parameter matrix and the camera extrinsic parameter matrix, so that positioning and target detection are more accurate.
[0047] In this embodiment, a method for calibrating internal and external parameters of a camera is provided, which can be used in computer equipment. Figure 2 FIG. 4 is a flow chart of another camera internal and external parameter calibration method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:
[0048] Step S201: Acquire an image captured by a camera.
[0049] Step S202: determining the first actual coordinates of the target point in the pixel coordinate system and the second actual coordinates of the target point in the world coordinate system according to the captured image.
[0050] Step S203: Acquire the camera's internal parameter set, where the internal parameter set includes multiple groups of internal parameter matrices.
[0051] Step S204: Determine the camera's external parameter set according to the target's first actual coordinates in the pixel coordinate system, the target's second actual coordinates in the world coordinate system, and the internal parameter set, wherein the external parameter set includes multiple groups of external parameter matrices, and each group of internal parameter matrices corresponds to a group of external parameter matrices.
[0052] In an optional implementation, the external parameter set of the camera is obtained by using a PNP algorithm according to the first actual coordinates of the target point in the pixel coordinate system, the second actual coordinates of the target point in the world coordinate system, and the internal parameter set.
[0053] Specifically, the PNP algorithm is based on the following formula (1):
[0054]
[0055] In formula (1), p represents the coordinate of the point in the pixel coordinate system, P C Represents the coordinates of the point in the camera coordinate system, P W represents the coordinates of the point in the world coordinate system, w represents the depth of the point, K represents the intrinsic parameter set of the camera, R CW Represents the rotation matrix R from the 3D world coordinate system to the 3D camera coordinate system, Represents the vector from the origin of the 3D camera coordinate system to the origin of the 3D world coordinate system in the camera coordinate system.
[0056] Step S205: Determine multiple calculated coordinates of the target point in the world coordinate system according to the camera's extrinsic parameter set, wherein each set of extrinsic parameter matrices corresponds to one calculated coordinate.
[0057] Figure 3 is a schematic diagram of a method for determining the calculated coordinates of a target point according to an embodiment of the present invention; Figure 3 As shown, according to the camera's external parameter set, determining multiple calculated coordinates of the target in the world coordinates includes the following steps:
[0058] Step S2051: Obtain a first distance from the camera to the ground.
[0059] For example, Figure 3 As shown, the point where the camera is located is point O, and the first distance from the camera to the ground is OH.
[0060] Step S2052: Determine the third actual coordinates of the target point in the camera coordinate system according to the first actual coordinates of the target point in the pixel coordinate system.
[0061] For example, Figure 3 As shown in the figure, the actual coordinates of the target in the pixel coordinate system are (u, v, 1). According to the camera's intrinsic parameter matrix K and the first actual coordinates of the target in the pixel coordinate system (u, v, 1), the actual coordinates of the target in the camera coordinate system (X A , Y A , 1), that is, point A.
[0062]
[0063] Step S2053: determining a depth coefficient of the target point according to the third actual coordinate of the target point in the camera coordinate system and the first distance.
[0064] For example, Figure 3 As shown, point B is a corner point on the calibration site, which is projected to the camera imaging plane through the camera imaging model, and point A is its projection position on the normalized plane, so:
[0065]
[0066] Based on the above formula (3), determining the depth coefficient of the target point according to the third actual coordinate of the target point in the camera coordinate system and the first distance includes: calculating a first vector between the coordinate point of the camera and the third actual coordinate of the target point in the camera coordinate system; calculating the dot product of the unit normal vector perpendicular to the ground and the first vector to obtain a first distance; and dividing the second distance by the first distance to obtain the depth coefficient of the target point.
[0067] For example, Figure 3 As shown, the coordinate point of the camera is calculated with the point A(X A , Y A , 1) between the first vector Calculate the unit normal vector perpendicular to the ground and the first vector The dot product of OA′ gives the first distance, OA′. Divide OA′ by OH to get w.
[0068] Step S2054: Determine multiple calculated coordinates of the target in the world coordinate system according to the third actual coordinate in the target camera coordinate system, the depth coefficient of the target and the external parameter set of the camera.
[0069] For example, Figure 3 As shown, according to the actual coordinates of the target in the camera coordinate system, point B (X BW , Y BW , Z BW ), the depth coefficient w of the target and the external parameter matrix (R, t) of the camera are used to calculate the calculated coordinates (X) of the target in the camera coordinate system using the following formula (4): B , Y B , Z B ); then, according to the calculated coordinates of the target point in the camera coordinate system, the calculated coordinates of the target point in the world coordinate system P are obtained using the following formula (5): word .
[0070]
[0071] P word =R -1 [X B Y B Z B ] T -R -1 t Formula (5)
[0072] Step S206: Select the optimal calculation coordinates from the multiple calculation coordinates, use a set of internal parameter matrices corresponding to the optimal calculation coordinates as the target internal parameters of the camera, and use a set of external parameter matrices corresponding to the optimal calculation coordinates as the target external parameters of the camera.
[0073] The calibration method of camera internal and external parameters provided by the present invention does not directly use the internal parameter matrix written by the camera manufacturer or the internal parameter matrix determined by Zhang Zhengyou's calibration method as the target internal parameter. The determination of the target external parameter does not only rely on the camera internal parameter matrix, but selects the optimal calculation coordinates from multiple calculation coordinates, wherein the multiple calculation coordinates are obtained according to the camera internal parameter matrix and the camera extrinsic parameter matrix, that is, the restoration error of the camera to the captured scene is reduced by the joint optimization of the camera internal parameter matrix and the camera extrinsic parameter matrix, so that positioning and target detection are more accurate.
[0074] In this embodiment, a method for calibrating internal and external parameters of a camera is provided, which can be used in computer equipment. Figure 4 FIG. 1 is a flow chart of another camera internal and external parameter calibration method according to an embodiment of the present invention. Figure 4 As shown, the process includes the following steps:
[0075] Step S401: Acquire an image captured by a camera.
[0076] Step S402: Determine the first actual coordinates of the target point in the pixel coordinate system and the second actual coordinates of the target point in the world coordinate system according to the captured image.
[0077] Step S403: Acquire the camera's internal parameter set, where the internal parameter set includes multiple groups of internal parameter matrices.
[0078] Step S404: Determine the camera's external parameter set according to the target's first actual coordinates in the pixel coordinate system, the target's second actual coordinates in the world coordinate system, and the internal parameter set, wherein the external parameter set includes multiple groups of external parameter matrices, and each group of internal parameter matrices corresponds to a group of external parameter matrices.
[0079] Step S405: Determine multiple calculated coordinates of the target point in the world coordinate system according to the camera's external parameter set, wherein each set of external parameter matrices corresponds to one calculated coordinate.
[0080] Step S406: Select the optimal calculation coordinates from the multiple calculation coordinates, use a set of internal parameter matrices corresponding to the optimal calculation coordinates as the target internal parameters of the camera, and use a set of external parameter matrices corresponding to the optimal calculation coordinates as the target external parameters of the camera.
[0081] In an optional implementation, selecting the optimal calculation coordinate from multiple calculation coordinates includes the following steps:
[0082] Step S4061: Get the operating mode of the camera.
[0083] Specifically, the operating modes of the camera include a surround view mode, an automatic driving mode, and a map building mode. It should be noted that when there are multiple cameras, the operating modes of the camera include a surround view mode.
[0084] Step S4062: Determine the optimization target according to the operation mode.
[0085] In an optional embodiment, determining the optimization target according to the operating mode includes: when the operating mode of the camera is a surround view mode, taking the projection error of the three-dimensional world coordinate point of the common view area of each camera as the optimization target; when the operating mode of the camera is an automatic driving mode or a map building mode, taking the error between the calculated coordinates and the second actual coordinates as the optimization target.
[0086] Step S4063: Select the optimal calculation coordinate from multiple calculation coordinates according to the optimization target.
[0087] Specifically, in the 360-degree surround image of AVM, the stitching effect of the four fisheye images is quantified, and the projection error of the three-dimensional world coordinate points in the common view area that can be observed by adjacent fisheye cameras is taken as the optimization target. A set of solutions that minimize the projection error of the four common view areas are obtained as the final optimized target internal parameters and target external parameters.
[0088] like Figure 5 As shown, the cross in the upper left corner is the point of the front camera (that is, the position of the point obtained by inverse perspective transformation according to the external parameter matrix of the front camera), and the circle is the point of the left camera (that is, the position of the point obtained by inverse perspective transformation according to the external parameter matrix of the left camera). The front camera and the left camera form a common viewing area 1, and the other common viewing areas are similar. Figure 5 It can be seen that the optimization goal is to obtain a set of parameters that maximizes the overlap of points in all common view areas.
[0089] For any combination of extrinsic parameter matrices (an extrinsic parameter matrix combination includes the extrinsic parameter matrix of the front camera, the extrinsic parameter matrix of the rear camera, the extrinsic parameter matrix of the left camera and the extrinsic parameter matrix of the right camera), the maximum error values of common view area 1, common view area 2, common view area 3 and common view area 4 are calculated respectively.
[0090] The maximum error value of the common vision area 1 can be obtained by the following method: respectively calculate the distance difference corresponding to each target point in the common vision area 1, such as Figure 5 As shown, a total of 8 distance differences are obtained, and the maximum value is selected from the 8 distance differences to obtain the maximum error of the common view area 1.
[0091] For example, for the same point in the common viewing area 1, the extrinsic matrix of the front camera is used to perform an inverse perspective transformation to obtain point 1, P1 = (X1, Y1, Z1); the extrinsic matrix of the left camera is used to perform an inverse perspective transformation to obtain point 2, P1 = (X2, Y2, Z2), and the distance difference corresponding to the point can be calculated using the following formula (6);
[0092]
[0093] After respectively obtaining the maximum error value of common viewing area 1, the maximum error value of common viewing area 2, the maximum error value of common viewing area 3 and the maximum error value of common viewing area 4, the maximum value is selected among the maximum errors of common viewing area 1, common viewing area 2, common viewing area 3 and common viewing area 4, that is, the maximum value corresponding to any set of external parameter matrices is obtained.
[0094] Traverse all combinations of external parameter matrices to obtain N maximum values; select the minimum value among the N maximum values, and obtain the target internal parameters and target external parameters corresponding to the minimum value.
[0095] It should be noted that the common view area here is only an example, and the common view area selection of this method can be specially designed according to different types of cameras. As can be seen from the figure, the projection points of the common view area after optimization by this solution are almost coincident, and after quantification, the reprojection error can be reduced to less than 1 cm.
[0096] In addition, when driving autonomously or using cameras to build high-precision maps, the average error between the three-dimensional world coordinates corresponding to all two-dimensional points captured by a single camera and the measured real-world coordinates can be used as the optimization target, and the solution with the minimum error can be optimized as the final internal and external parameter matrix.
[0097] For example, Figure 6 As shown in the figure, the circle is the actual physical space point position, the cross is the point position obtained by inverse perspective transformation according to the camera's extrinsic parameter matrix, and the error between the physical space point position and the point position obtained by inverse perspective transformation according to the camera's extrinsic parameter matrix is the optimization target.
[0098] Among them, for Figure 6 The same point in the physical space is P3 = (X3, Y3, Z3); the inverse perspective transformation is performed using the camera's extrinsic matrix to obtain point 4, P4 = (X4, Y4, Z4). The error between the position of the physical space point and the position of the point obtained by the inverse perspective transformation based on the camera's extrinsic matrix can be calculated using formula (7).
[0099]
[0100] The camera internal and external parameter calibration method provided in this embodiment can set different optimization targets according to different camera application scenarios, and jointly optimize the camera internal and external parameters to reduce the camera's restoration error of the captured scene, making positioning and target detection more accurate.
[0101] In this embodiment, a calibration device for camera internal and external parameters is also provided, which is used to implement the above embodiments and preferred implementation modes, and will not be repeated here. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0102] This embodiment provides a camera internal and external parameter calibration device. Figure 7 As shown, including:
[0103] The captured image acquisition module 701 is used to acquire the captured image of the camera;
[0104] A captured image processing module 702 is used to determine a first actual coordinate of the target point in a pixel coordinate system and a second actual coordinate of the target point in a world coordinate system according to the captured image;
[0105] An internal parameter set acquisition module 703 is used to acquire an internal parameter set of a camera, wherein the internal parameter set includes multiple groups of internal parameter matrices;
[0106] The external parameter set calculation module 704 is used to determine the external parameter set of the camera according to the first actual coordinate of the target point in the pixel coordinate system, the second actual coordinate of the target point in the world coordinate system, and the internal parameter set, wherein the external parameter set includes multiple groups of external parameter matrices, and each group of internal parameter matrices corresponds to a group of external parameter matrices;
[0107] A world coordinate calculation module 705 is used to determine a plurality of calculated coordinates of the target point in the world coordinate according to the camera's external parameter set, wherein each set of external parameter matrices corresponds to one calculated coordinate;
[0108] The optimal selection module 706 is used to select the optimal calculation coordinates from multiple calculation coordinates, and use a set of internal parameter matrices corresponding to the optimal calculation coordinates as the target internal parameters of the camera, and use a set of external parameter matrices corresponding to the optimal calculation coordinates as the target external parameters of the camera.
[0109] In some optional implementations, the external parameter set calculation module 704 is specifically used to obtain the camera's external parameter set using the PNP algorithm based on the first actual coordinates of the target in the pixel coordinate system, the second actual coordinates of the target in the world coordinate system, and the internal parameter set.
[0110] In some optional implementations, the world coordinate calculation module 705 is specifically used to: obtain a first distance from the camera to the ground; determine a third actual coordinate of the target in the camera coordinate system based on the first actual coordinate of the target in the pixel coordinate system; determine a depth coefficient of the target based on the third actual coordinate of the target in the camera coordinate system and the first distance; determine multiple calculated coordinates of the target in the world coordinate system based on the third actual coordinate of the target in the camera coordinate system, the depth coefficient of the target and the external parameter set of the camera.
[0111] In some optional implementations, the world coordinate calculation module 705 includes a distance acquisition unit, a coordinate conversion unit, a depth coefficient determination unit, and a calculated coordinate determination unit. The depth coefficient determination unit is specifically used to: calculate a first vector between the coordinate point of the camera and the third actual coordinate of the target point in the camera coordinate system; calculate the dot product of the unit normal vector perpendicular to the ground and the first vector to obtain a first distance; divide the second distance by the first distance to obtain a depth coefficient of the target point.
[0112] In an optional implementation, the optimization module 706 is specifically used to: obtain the operation mode of the camera; determine the optimization target according to the operation mode; and select the optimal calculation coordinate from multiple calculation coordinates according to the optimization target.
[0113] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.
[0114] The calibration device of the camera internal and external parameters in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.
[0115] The embodiment of the present invention also provides a computer device having the above Figure 8 The calibration device for the internal and external parameters of the camera shown.
[0116] See also Figure 8 , Figure 8 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 8As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 A processor 10 is taken as an example.
[0117] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.
[0118] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.
[0119] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0120] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.
[0121] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.
[0122] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.
[0123] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.
[0124] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.
[0125] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.
Claims
1. A method for calibrating camera internal and external parameters, characterized in that: The method comprises: Get the image captured by the camera; Determine a first actual coordinate of a target point in a pixel coordinate system and a second actual coordinate of the target point in a world coordinate system according to the captured image; Acquire an internal parameter set of the camera, wherein the internal parameter set includes multiple groups of internal parameter matrices; Determine an external parameter set of the camera according to a first actual coordinate of the target point in the pixel coordinate system, a second actual coordinate of the target point in the world coordinate system, and the internal parameter set, wherein the external parameter set includes multiple groups of external parameter matrices, and each group of internal parameter matrices corresponds to a group of external parameter matrices; Determine a plurality of calculated coordinates of the target point in the world coordinates according to the extrinsic parameter set of the camera, wherein each set of extrinsic parameter matrices corresponds to one calculated coordinate; Selecting an optimal calculation coordinate from the multiple calculation coordinates, using a set of internal parameter matrices corresponding to the optimal calculation coordinates as target internal parameters of the camera, and using a set of external parameter matrices corresponding to the optimal calculation coordinates as target external parameters of the camera; Determining a plurality of calculated coordinates of the target point in the world coordinate according to the external parameter set of the camera includes: Obtaining a first distance from the camera to the ground; Determine a third actual coordinate of the target point in the camera coordinate system according to the first actual coordinate of the target point in the pixel coordinate system; Determine a depth coefficient of the target point according to a third actual coordinate of the target point in the camera coordinate system and the first distance; Determine a plurality of calculated coordinates of the target point in the world coordinate system according to the third actual coordinate in the target camera coordinate system, the depth coefficient of the target point, and the external parameter set of the camera; The determining the depth coefficient of the target point according to the third actual coordinate of the target point in the camera coordinate system and the first distance comprises: Calculating a first vector between the coordinate point of the camera and a third actual coordinate of the target point in the camera coordinate system; Calculate the dot product of a unit normal vector perpendicular to the ground and the first vector to obtain a second distance; Dividing the second distance by the first distance to obtain a depth coefficient of the target; The selecting the optimal calculation coordinate from the multiple calculation coordinates comprises: Obtaining the operating mode of the camera; Determining an optimization target according to the operation mode; An optimal calculation coordinate is selected from the plurality of calculation coordinates according to the optimization target.
2. The method according to claim 1, characterized in that Determining the external parameter set of the camera according to the first actual coordinate of the target point in the pixel coordinate system, the second actual coordinate of the target point in the world coordinate system, and the internal parameter set includes: The external parameter set of the camera is obtained by using a PNP algorithm according to the first actual coordinates of the target point in the pixel coordinate system, the second actual coordinates of the target point in the world coordinate system, and the internal parameter set.
3. The method according to claim 1, characterized in that Determining the optimization target according to the operation mode includes: When the operation mode of the camera is a surround view mode, the projection errors of the three-dimensional world coordinate points in the common view area of the multiple cameras are used as the optimization target; When the operation mode of the camera is the automatic driving mode or the map building mode, the error between the calculated coordinates and the second actual coordinates is used as the optimization target.
4. A camera internal and external parameter calibration device, characterized in that: The device comprises: A captured image acquisition module is used to acquire the image captured by the camera; A captured image processing module, used to determine a first actual coordinate of a target point in a pixel coordinate system and a second actual coordinate of the target point in a world coordinate system according to the captured image; An internal parameter set acquisition module, used to acquire an internal parameter set of the camera, wherein the internal parameter set includes multiple groups of internal parameter matrices; an external parameter set calculation module, used to determine the external parameter set of the camera according to the first actual coordinate of the target point in the pixel coordinate system, the second actual coordinate of the target point in the world coordinate system, and the internal parameter set, wherein the external parameter set includes multiple groups of external parameter matrices, and each group of internal parameter matrices corresponds to a group of external parameter matrices; A world coordinate calculation module, used to determine a plurality of calculated coordinates of the target point in the world coordinates according to the extrinsic parameter set of the camera, wherein each set of extrinsic parameter matrices corresponds to one calculated coordinate; A selection module, used for selecting the best calculation coordinates from the multiple calculation coordinates, using a set of internal parameter matrices corresponding to the best calculation coordinates as the target internal parameters of the camera, and using a set of external parameter matrices corresponding to the best calculation coordinates as the target external parameters of the camera; The world coordinate calculation module is specifically used to: obtain a first distance from the camera to the ground; determine a third actual coordinate of the target in the camera coordinate system according to the first actual coordinate of the target in the pixel coordinate system; determine a depth coefficient of the target according to the third actual coordinate of the target in the camera coordinate system and the first distance; determine a plurality of calculated coordinates of the target in the world coordinate system according to the third actual coordinate of the target in the camera coordinate system, the depth coefficient of the target and the external parameter set of the camera; The world coordinate calculation module includes a distance acquisition unit, a coordinate conversion unit, a depth coefficient determination unit and a calculated coordinate determination unit; wherein the depth coefficient determination unit is specifically used to: calculate a first vector between the coordinate point of the camera and the third actual coordinate of the target point in the camera coordinate system; calculate a dot product of a unit normal vector perpendicular to the ground and the first vector to obtain a second distance; divide the second distance by the first distance to obtain a depth coefficient of the target point; The optimization module is specifically used to: obtain the operation mode of the camera; determine the optimization target according to the operation mode; and select the optimal calculation coordinate from the multiple calculation coordinates according to the optimization target.
5. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the camera internal and external parameter calibration method according to any one of claims 1 to 3 by executing the computer instructions.
6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the camera internal and external parameter calibration method according to any one of claims 1 to 3.
7. A computer program product, characterized in that The method comprises computer instructions, wherein the computer instructions are used to enable a computer to execute the camera internal and external parameter calibration method according to any one of claims 1 to 3.
