Camera external parameter calibration method and related device

Abstract

The invention discloses a camera external parameter calibration method and a related device, and the method comprises the steps: obtaining a target image which is shot by a to-be-calibrated target camera of a target vehicle and comprises a target reference object, and further determining the pixel point coordinates of the target reference object in the target image under a pixel coordinate system, the actual point coordinates of the target reference object in the world coordinate system are obtained, then the external parameters of the target camera are determined according to the internal parameters of the target camera, the pixel point coordinates and the actual point coordinates, and the external parameters of the target camera are further calibrated. According to the method, actual point coordinates in a world coordinate system and pixel point coordinates of a target reference object in a target image shot by a target camera are obtained, external parameters can be determined according to existing internal parameters, the pixel point coordinates and the actual point coordinates, and the external parameters obtained through calculation are further calibrated; the external parameters of the target camera do not need to be manually calculated and calibrated, the calibration efficiency is improved, and the precision of the calibration result is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular, to a method and related device for calibrating the external parameters of a camera. Background Art

[0002] Currently, in order to make the data collected by the cameras installed on vehicles more accurate, it is generally necessary to calibrate the external parameters of the cameras on the vehicles.

[0003] In traditional calibration methods, when calibrating the external parameters of a vehicle's camera, professional personnel are generally required to collect and calibrate data on-site, which increases the labor cost and time cost and reduces the efficiency of external parameter calibration. At the same time, traditional calibration methods usually involve complex calibration data collection processes and calibration algorithms, and are affected by environmental factors, resulting in low accuracy of external parameter calibration. Summary of the Invention

[0004] Based on the above problems, this application provides a method and related device for calibrating the external parameters of a camera, aiming to solve the problems of low calibration efficiency and low accuracy caused by traditional external parameter calibration.

[0005] The embodiments of this application disclose the following technical solutions:

[0006] In a first aspect, the embodiments of this application provide a method for calibrating the external parameters of a camera, the method including:

[0007] Obtain a target image captured by a target camera to be calibrated of a target vehicle; the target image includes a target reference object; the target reference object indicates a reference object used for calibrating the target camera;

[0008] Obtain the actual point coordinates of the target reference object in the world coordinate system; the origin of the world coordinate system is the projection point of the target camera on the ground, the X-axis of the world coordinate system is parallel to the head direction of the target vehicle, the Y-axis of the world coordinate system is perpendicular to the head direction of the target vehicle in the horizontal direction, and the Z-axis of the world coordinate system is perpendicular to the ground;

[0009] Determine the pixel point coordinates of the target reference object in the pixel coordinate system in the target image; the origin of the pixel coordinate system is the vertex at the upper left corner of the target image, the X-axis of the pixel coordinate system is parallel to the width direction of the target image, and the Y-axis of the pixel coordinate system is parallel to the height direction of the target image;

[0010] Determine the external parameters of the target camera according to the internal parameters, pixel point coordinates, and actual point coordinates of the target camera;

[0011] Calibrate the external parameters of the target camera.

[0012] Optionally, before obtaining the target image captured by the target camera to be calibrated of the target vehicle, the method further includes:

[0013] Determine the visible area of the target camera; the visible area is the area captured by the target camera;

[0014] Set at least one set of reference groups within the visible area; each reference group includes two target reference objects; the two target reference objects are respectively arranged on both sides of the central axis of the target vehicle, the connection line between the two target reference objects is perpendicular to the central axis, and the perpendicular distances from the two target reference objects to the central axis are equal; the distances between each reference group and the target vehicle are different;

[0015] Obtaining the target image captured by the target camera to be calibrated of the target vehicle includes:

[0016] Obtain a target image captured by the target camera including at least one set of reference objects.

[0017] Optionally, if the target image includes at least two sets of reference groups, after determining the pixel coordinates of the target reference objects in the pixel coordinate system in the target image, the method further includes:

[0018] Determine the target reference group among at least two sets of reference groups; the pixel distance between the two target reference objects in the target reference group is the largest; the pixel distance is determined according to the pixel coordinates of the two target reference objects;

[0019] Use the pixel coordinates of the two target reference objects in the target reference group as the verification point coordinates, and use the pixel coordinates of the two target reference objects in each reference group other than the target reference group as the calibration point coordinates;

[0020] Sort the two calibration point coordinates corresponding to each reference group based on the two verification point coordinates corresponding to the target reference group and a preset sorting rule; where the preset sorting rule is associated with the position order of at least two sets of reference groups in the world coordinate system.

[0021] Optionally, determining the pixel coordinates of the target reference objects in the pixel coordinate system in the target image includes:

[0022] Perform regional cropping on the target image to obtain a target image region; the target image region includes the target image;

[0023] Perform preprocessing and binarization processing on the target image region to obtain a binarized image;

[0024] Extract the contour of the target reference object from the binarized image;

[0025] Determine the center point of the target reference object according to the contour;

[0026] Determine the pixel coordinates of the target reference object in the pixel coordinate system according to the center point.

[0027] Optionally, determine the external parameters of the target camera according to the internal parameters of the target camera, the pixel coordinates and the actual point coordinates, including:

[0028] Determine the internal parameter matrix according to the internal parameters;

[0029] Normalize the pixel coordinates based on the internal parameter matrix to obtain the normalized point coordinates;

[0030] Determine the external parameters of the target camera through a preset camera pose algorithm according to the normalized point coordinates and the actual point coordinates.

[0031] Optionally, calibrate the external parameters of the target camera, including:

[0032] Determine the projection point coordinates of the actual point coordinates in the pixel coordinate system according to the internal parameters and the external parameters;

[0033] Calibrate the external parameters of the target camera according to the projection point coordinates and the pixel coordinates.

[0034] Optionally, calibrate the external parameters of the target camera according to the projection point coordinates and the pixel coordinates, including:

[0035] Determine the error between the projection point coordinates and the pixel coordinates;

[0036] Judge whether the error is greater than the preset error; if the error is greater than the preset error, recalibrate the external parameters of the target camera; if the error is less than or equal to the preset error, the calibration is successful, and the external parameters are used as the target external parameters of the target camera.

[0037] In a second aspect, an embodiment of the present application provides a device for calibrating the external parameters of a camera. The device includes:

[0038] An image acquisition module, configured to acquire a target image captured by a target camera to be calibrated of a target vehicle; the target image includes a target reference object; the target reference object indicates a reference object used for calibrating the target camera;

[0039] An actual point coordinate acquisition module, configured to acquire the actual point coordinates of the target reference object in the world coordinate system; the origin of the world coordinate system is the projection point of the target camera on the ground as the origin, the X axis of the world coordinate system is parallel to the front direction of the target vehicle, the Y axis of the world coordinate system is perpendicular to the front direction of the target vehicle in the horizontal direction, and the Z axis of the world coordinate system is perpendicular to the ground;

[0040] A pixel coordinate determination module, configured to determine the pixel coordinates of a target reference object in a target image in a pixel coordinate system; the origin of the pixel coordinate system is the vertex at the upper left corner of the target image, the X-axis of the pixel coordinate system is parallel to the width direction of the target image, and the Y-axis of the pixel coordinate system is parallel to the height direction of the target image;

[0041] An external parameter determination module, configured to determine the external parameters of the target camera according to the internal parameters of the target camera, the pixel coordinates, and the actual point coordinates;

[0042] A calibration module, configured to calibrate the external parameters of the target camera.

[0043] In a third aspect, an embodiment of the present application provides a computer device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, where when the processor executes the computer program, the camera external parameter calibration method described in the first aspect is implemented.

[0044] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, and when the instructions are run on a terminal device, the terminal device is caused to execute the camera external parameter calibration method described in the first aspect.

[0045] Compared with the prior art, the present application has the following beneficial effects:

[0046] The camera external parameter calibration method provided by the embodiment of the present application obtains a target image including a target reference object captured by a target camera to be calibrated of a target vehicle, further determines the pixel coordinates of the target reference object in the target image in a pixel coordinate system, and obtains the actual point coordinates of the target reference object in a world coordinate system, and then determines the external parameters of the target camera according to the internal parameters of the target camera, the pixel coordinates, and the actual point coordinates, and further calibrates the external parameters of the target camera. Among them, by obtaining the actual point coordinates in the world coordinate system and the pixel coordinates of the target reference object in the target image captured by the target camera, the external parameters can be determined according to the existing internal parameters, pixel coordinates, and actual point coordinates, and further by calibrating the calculated external parameters, there is no need for manual calculation and calibration of the external parameters of the target camera, improving the calibration efficiency and the accuracy of the calibration result. Description of the Drawings

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0048] Figure 1 A flowchart of a method for calibrating the external parameters of a camera provided by an embodiment of the present application;

[0049] Figure 2 A schematic diagram of setting a target reference object provided by an embodiment of the present application;

[0050] Figure 3 A schematic diagram of a process for calibrating the external parameters of a camera provided by an embodiment of the present application;

[0051] Figure 4 A schematic structural diagram of a device for calibrating the external parameters of a camera provided by an embodiment of the present application. Specific embodiments

[0052] Currently, in order to make the data collected by the cameras installed on vehicles more accurate, it is generally necessary to calibrate the external parameters of the cameras on the vehicles.

[0053] In traditional calibration methods, when calibrating the external parameters of vehicle cameras, it generally requires professional personnel to collect and calibrate data on-site, increasing the labor cost and time cost and reducing the efficiency of external parameter calibration. At the same time, traditional calibration methods usually involve complex calibration data collection processes and calibration algorithms. For non-professional technical personnel, it takes a certain amount of training and experience accumulation to learn and master the method for calibrating the external parameters of cameras, and the operation is relatively complex. Moreover, the calibration of the external parameters of cameras is easily affected by human factors and environmental factors, which may result in low accuracy of external parameter calibration.

[0054] To solve the problems existing in traditional calibration methods, related technologies also provide a deep learning calibration method, which requires a large amount of calibration data to train the model or perform calibration calculations, and at the same time, it is necessary to ensure the quality of the calibration data. However, since the deep learning calibration method needs to establish a good calibration model and train and optimize the model to adapt to different vehicles, cameras, and environmental conditions, it may involve professional knowledge in fields such as deep learning and image processing. For non-professional technical personnel, a large amount of training and guidance are required to perform the operation, increasing the time cost, and operational errors may result in low accuracy of external parameter calibration.

[0055] To solve the above technical problems, an embodiment of the present application provides a method and related device for calibrating the external parameters of a camera. The method includes: obtaining a target image including a target reference object captured by a target camera to be calibrated of a target vehicle, further determining the pixel coordinates of the target reference object in the pixel coordinate system in the target image, and obtaining the actual point coordinates of the target reference object in the world coordinate system, and then determining the external parameters of the target camera according to the internal parameters, pixel coordinates, and actual point coordinates of the target camera, and further calibrating the external parameters of the target camera.

[0056] In this way, by obtaining the actual point coordinates in the world coordinate system and the pixel coordinates of the target reference object in the target image captured by the target camera, the external parameters can be determined based on the existing internal parameters, pixel coordinates, and actual point coordinates. Furthermore, by calibrating the calculated external parameters, it is not necessary for humans to calculate and calibrate the external parameters of the target camera, which improves the calibration efficiency and the accuracy of the calibration results.

[0057] To enable those skilled in the art to better understand the solution of this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of this application.

[0058] See Figure 1 , which is a schematic flowchart of a method for external parameters of a camera provided by an embodiment of this application.

[0059] Combined with Figure 1 shown, the method for calibrating the external parameters of a camera provided by an embodiment of this application may include:

[0060] S101: Obtain a target image captured by a target camera to be calibrated on a target vehicle.

[0061] The target vehicle refers to the vehicle to be calibrated for the external parameters of the camera. The target camera refers to the camera to be calibrated on the target vehicle. The target image refers to the image captured by the target camera. The target image includes a target reference object, and the target reference object can be one or more, which is not specifically limited herein. The target reference object indicates the reference object used for calibrating the target camera, such as a black disc, a black square disc, a colored flag, etc., which is not specifically limited herein.

[0062] The external parameters of the camera refer to the position and attitude parameters of the camera in the world coordinate system, and these parameters determine the relative position relationship between the camera coordinate system and the world coordinate system. Specifically, the external parameters of the camera may include a translation vector and a rotation matrix. The translation vector represents the position of the camera in the world coordinate system. The rotation matrix describes the direction of the coordinate axes of the world coordinate system relative to the coordinate axes of the camera.

[0063] In order to improve the accuracy of calibrating the external parameters of the target camera, in some possible implementation manners, before performing step S101, the target reference object may also be set, and the specific method is as follows:

[0064] A1: Determine the visible area of the target camera.

[0065] Among them, the visible area is the area captured by the target camera. It should be understood that the target reference object needs to be set within the visible area of the target camera, so as to facilitate obtaining the target image including the target reference object captured by the target camera subsequently, in order to realize the subsequent calibration of the external parameters of the target camera.

[0066] A2: Set at least one set of reference groups within the visible area.

[0067] Among them, each reference group includes two target reference objects, and the two target reference objects are respectively set on both sides of the central axis of the target vehicle. The line connecting the two target reference objects is perpendicular to the central axis, and the perpendicular distances from the two target reference objects to the central axis are equal. The distances of each reference group from the target vehicle are different.

[0068] The central axis of the target vehicle refers to the distance between two perpendicular lines that pass through the midpoints of two adjacent wheels on the same side of the target vehicle and are perpendicular to the longitudinal symmetry plane of the target vehicle. Specifically, the central axis is the line formed by the midpoints of two adjacent wheels (such as the front wheel and the rear wheel) on the same side (such as the left side or the right side) of the target vehicle. The central axis is perpendicular to the longitudinal symmetry plane of the vehicle, that is, the plane that divides the vehicle into two halves from front to back.

[0069] Among them, if at least one set of reference object groups is set, step S101 includes: obtaining a target image including at least one set of reference objects captured by the target camera. It should be understood that the target image should include all the set target reference objects for subsequent external parameter calibration.

[0070] To illustrate the setting method of the reference objects, as an example, assume that the reference object is a black disc (abbreviated as black circle), the central axis of the vehicle coincides with the center line of the lane, the lane line on the left side of the center line of the lane is the left lane line, and the lane line on the right side of the center line of the lane is the right lane line. Set 3 control groups and one verification group. The specific setting method is as follows: Select 6 obvious positions on the inner edge between the left lane line and the right lane line within the visible area of the target camera, such as at the cement floor seam or at the intersection of the lane line and the floor seam, and mark them as A - F respectively. Among them, the lines connecting AF, BE, and CD need to be perpendicular to the center line of the lane, and A and F, B and E, C and D should be as close as possible on the same horizontal line to ensure the accuracy and stability of the external parameter calibration. In addition, 2 points to the left of the left lane line or to the right of the right lane line can also be selected as verification points, and marked as G and H respectively. Stick black circles on the eight points A - H, and the centers of the circles need to be aligned with the calibration points and the verification points.

[0071] It should be noted that since the target camera itself is not a ranging tool, during the external parameter calibration process, it is still necessary to use a measuring tool to measure the target reference object in the world coordinate system. In the embodiments of the present application, the setting method of the target reference object can be standardized to ensure that non-professionals can set the target reference object and measure the actual point coordinates of the target reference object in the world coordinate system, without the need for a large amount of professional knowledge accumulation, saving labor costs and time costs without affecting the calibration accuracy.

[0072] S102: Obtain the actual point coordinates of the target reference object in the world coordinate system.

[0073] The world coordinate system, also known as the coordinate space, is a coordinate system used to describe and locate the position of an object. The world coordinate system is the absolute coordinate system of the system and is used to determine the positions of all points on the screen before establishing the user coordinate system. In three-dimensional space, the world coordinate system uses three coordinate axes (X, Y, Z) to define the position of a point. Among them, the origin of the world coordinate system is the projection point of the target camera on the ground, the X-axis of the world coordinate system is parallel to the front direction of the target vehicle, the Y-axis of the world coordinate system is perpendicular to the front direction of the target vehicle in the horizontal direction, and the Z-axis of the world coordinate system is perpendicular to the ground.

[0074] The actual point coordinates refer to the point coordinates of the target reference object in the world coordinate system.

[0075] Combined with Figure 2 As shown, assuming that the target reference object includes A - H, the parameters of the target reference object can be measured first. Measure the distances from points A - H to the front of the vehicle, that is, the perpendicular distances from the projection point of the camera center on the ground to the lane lines (a total of 8 parameters), denoted as dist_A to dist_H; measure the straight-line distances of the three straight lines of A and F, B and E, and C and D, denoted as dist_AF, dist_BE, and dist_CD respectively; measure the distances from points G and H to the inner edge of the left lane line, denoted as dist_GL and dist_HL; measure the distances from the two front wheels of the vehicle to the inner side of their respective nearest lane lines, denoted as dist_Left and dist_Right; measure the offset distance of the target camera from the center axis of the target vehicle, denoted as Δy.

[0076] Then, the vertical distances (X coordinates in the world coordinate system) between the target reference objects A - H and the target camera can be further determined: X A = dist_A; X B = dist_B; X C = dist_C; X D = dist_D; X E = dist_E; X F= dist_F; X G = dist_G; X H = dist_H. Further, the width of the lane line can be determined as follows in Equation (1):

[0077]

[0078] Then, the corresponding Y coordinates of the target reference objects A - H in the world coordinate system are determined through the following Equation (2):

[0079]

[0080] Y G = Y D + dist_GL;

[0081] Y H = Y D + dist_HL; (2)

[0082] where dist_GL represents the vertical distance from the target reference object G to the lane line, and dist_HL represents the vertical distance from the target reference object to the lane line.

[0083] It should be noted that in the embodiments of the present application, non - professional personnel only need to input the measured data (such as dist_Lane, dist_Right, etc.), and then the actual point coordinates of the target reference object in the world coordinate system can be automatically generated through a software program. There is no need for the user to calculate manually, avoiding manual calculation errors and improving the accuracy of the external parameter calibration.

[0084] In a possible implementation, after successfully obtaining the X and Y coordinates of the 8 points of the target reference objects A - H in the world coordinate system, the 8 actual point coordinates can also be arranged according to a preset rule and output in the form of a hexadecimal byte stream.

[0085] S103: Determine the pixel coordinates of the target reference object in the pixel coordinate system in the target image.

[0086] The pixel coordinate system is a coordinate system used to describe the positions of pixel points in an image. The pixel coordinate system is a two - dimensional rectangular coordinate system that reflects the arrangement of pixels in the chip of the target camera. The pixel coordinate system is used to describe the position of each pixel point in the target image. The origin of the pixel coordinate system is the vertex at the upper left corner of the target image. The X - axis of the pixel coordinate system is parallel to the width direction of the target image, and the Y - axis of the pixel coordinate system is parallel to the height direction of the target image.

[0087] In a possible implementation manner, step S103 may include:

[0088] B1: Crop the target image to obtain the target image region.

[0089] Among them, the target image region includes the target image. Specifically, cropping the target image is to select the part of the target image that only contains the target reference object and its surrounding area, which can reduce the computational complexity of subsequent processing and improve efficiency. As an example, assume that the target image captured by the target camera includes the target reference object and other backgrounds. A rectangular frame can be set to select the area that only contains the target reference object. The coordinates of this rectangular frame can be determined in advance according to the actual situation or dynamically calculated through other image processing algorithms.

[0090] B2: Preprocess and binarize the target image region to obtain a binary image.

[0091] Preprocessing refers to preprocessing the target image region, such as grayscale processing, Gaussian blur processing, etc. Grayscale processing is the process of converting a color image into a grayscale image, which can simplify image information, reduce computational complexity, and highlight the features of the target. As an example, a color image with three RGB channels can be converted into a single-channel grayscale image. Common grayscale methods include the average method, the maximum method, the weighted average method, etc. Gaussian blur is an image smoothing technique used to reduce image noise and detail levels, making the image blurred. Performing Gaussian blur on the target image region before extracting the contour of the target reference object can improve the robustness of subsequent operations.

[0092] Binarization is the process of converting the target image region into black and white, which can emphasize the edge information of the target object and facilitate subsequent contour extraction. As an example, a threshold is set, and pixels in the grayscale image less than the threshold are set to 0 (black), and pixels greater than or equal to the threshold are set to 255 (white).

[0093] B3: Extract the contour of the target reference object from the binary image.

[0094] Contour extraction is the process of detecting and extracting the edge contour of the target reference object from the binary image. After extracting the contour of the target reference object, the pixel coordinates of the center of the target reference object can be calculated.

[0095] B4: Determine the center point of the target reference object according to the contour.

[0096] The center point means the center of the target reference object. If the target reference object is circular, its centroid or the center of the minimum circumscribed circle is calculated as the center point.

[0097] B5: Determine the pixel coordinates of the target reference object in the pixel coordinate system according to the center point.

[0098] It should be understood that since the target reference object is a relatively obvious reference object, in order to improve the accuracy of calibration, it is necessary to determine the center point of the target reference object, and then determine the pixel coordinates of the center point as the pixel coordinates of the target reference object in the pixel coordinate system.

[0099] Since the center point coordinates extracted from the image are disordered, in order to achieve the same order as the world coordinate system, in a possible implementation, if the target image includes at least two reference groups, after step S103, the method may further include:

[0100] C1: Determine the target reference group among at least two reference groups.

[0101] Among them, the pixel distance between the two target reference objects in the target reference group is the largest. The pixel distance is determined according to the pixel coordinates of the two target reference objects. It should be understood that if the pixel distance between two target reference objects is the largest, it proves that this reference group is the farthest from the target camera. Therefore, this reference group can be used as the target reference group for subsequent sorting.

[0102] C2: Use the pixel coordinates corresponding to the two target reference objects in the target reference group as the verification point coordinates, and use the pixel coordinates corresponding to the two target reference objects in each reference group other than the target reference group as the calibration point coordinates.

[0103] As an example, assume that according to half of the width of the target image, the pixel coordinates corresponding to the two target reference objects in each reference group are divided into two parts (since the target reference objects in the same reference group are symmetric along the central axis, they can be divided into two parts), denoted as Point_Left and Point_Right respectively. Then, by calculating the distance between Point_Left and Point_Right, find the two target reference objects with the largest distance between Point_Left and Point_Right, and use the pixel coordinates corresponding to these two target reference objects as the verification point coordinates. All the remaining pixel coordinates in Point_Left and all the pixel coordinates in Point_Right are used as the calibration point coordinates.

[0104] C3: Sort the two calibration point coordinates corresponding to each reference group based on the two verification point coordinates corresponding to the target reference group and the preset sorting rule.

[0105] Among them, the preset sorting rule is associated with the position order of at least two reference groups in the world coordinate system. In order to ensure the order of the pixel coordinates, the calibration point coordinates and the verification point coordinates can be sorted according to their coordinate sizes respectively, to ensure the correspondence between the pixel coordinates on the target image and the actual point coordinates in the world coordinate system, and provide a reliable basis for the subsequent calibration process.

[0106] S104: Determine the extrinsic parameters of the target camera based on the intrinsic parameters, pixel coordinates, and actual coordinates of the target camera.

[0107] The intrinsic parameters are parameters that describe the internal properties of the target camera, and these parameters usually determine how the target camera maps coordinate points in the three-dimensional world onto the two-dimensional image plane.

[0108] The intrinsic parameters of the target camera may include focal length, principal point coordinates, distortion coefficients, etc. The focal length represents the focal lengths of the camera in the X-axis and Y-axis directions. The physical meaning of the focal length is how many pixels are represented by each f length on the imaging plane with a focal length of f. The principal point coordinates are the number of horizontal and vertical pixel offsets from the origin of the image physical coordinate system (image center) to the origin of the image pixel coordinate system (upper left corner of the image). The distortion coefficients include radial distortion coefficients and tangential distortion coefficients. These coefficients are used to describe the image distortion generated during the lens manufacturing and installation processes for correction in image processing and computer vision tasks.

[0109] As an example, step S104 can be implemented in the following way:

[0110] D1: Determine the intrinsic matrix based on the intrinsic parameters.

[0111] The intrinsic matrix is a matrix commonly used in computer vision and image processing, mainly used to describe the internal attribute parameters of the camera. The intrinsic matrix is a bridge for transforming 3D camera coordinates to 2D homogeneous image coordinates. It contains key information such as the focal length and principal point position of the target camera.

[0112] D2: Normalize the pixel coordinates based on the intrinsic matrix to obtain the normalized point coordinates.

[0113] It should be understood that for the convenience of subsequent extrinsic parameter calculation, it is necessary to linearly transform the pixel coordinates and actual coordinates through the intrinsic matrix of the target camera so that the form of the intrinsic matrix of the target camera is the identity matrix. Specifically, the pixel coordinates are normalized through the intrinsic matrix to obtain the actual coordinates in the world coordinate system, that is, the normalized point coordinates.

[0114] D3: Determine the extrinsic parameters of the target camera based on the normalized point coordinates and actual coordinates through a preset camera pose algorithm.

[0115] The preset camera pose algorithm refers to an algorithm for calculating the external parameters of the target camera, such as the PnP (Perspective-n-Point) algorithm. The PnP algorithm can calculate the rotation matrix and the translation vector. The PnP algorithm is a method for solving the correspondence between 3D and 2D points. The PnP algorithm describes how to estimate the pose of the camera when the positions of n 3D space points are known. Specifically, the PnP algorithm can be used to solve for the rotation matrix (R) and the translation vector (T) by combining the normalized coordinates and the actual point coordinates, which are the external parameters of the target camera.

[0116] S105: Calibrate the external parameters of the target camera.

[0117] It should be understood that by obtaining the actual point coordinates in the world coordinate system and the pixel coordinates of the target reference object in the target image captured by the target camera, the external parameters can be determined based on the existing internal parameters, pixel coordinates, and actual point coordinates, and further, by calibrating the calculated external parameters, there is no need for manual calculation and calibration of the external parameters of the target camera, which improves the calibration efficiency and the accuracy of the calibration result.

[0118] As a possible implementation, step S105 may include:

[0119] E1: Determine the projected point coordinates of the actual point coordinates in the pixel coordinate system according to the internal parameters and the external parameters.

[0120] It should be understood that after determining the external parameters of the target camera, the external parameter matrix and the internal parameter matrix obtained from the solution of the external parameters can be used to re-project the actual point coordinates back to the pixel coordinate system to obtain the projected point coordinates after projection.

[0121] E2: Calibrate the external parameters of the target camera according to the projected point coordinates and the pixel coordinates.

[0122] It should be understood that after determining the external parameters, the actual point coordinates are projected into the pixel coordinate system through the internal parameters and the external parameters to obtain the projected point coordinates. The projected point coordinates are used to reflect the pixel coordinates under this external parameter, and then the external parameter can be calibrated by comparing the projected point coordinates and the pixel coordinates.

[0123] As a possible implementation, step E2 may include:

[0124] F1: Determine the error between the projected point coordinates and the pixel coordinates.

[0125] Specifically, calculate the error between the projected point coordinates and the pixel coordinates, that is, the reprojection error (referring to the error of this application).

[0126] F2: Determine whether the error is greater than a preset error; if the error is greater than the preset error, recalibrate the extrinsic parameters of the target camera; if the error is less than or equal to the preset error, the calibration is successful, and the extrinsic parameters are used as the target extrinsic parameters of the target camera.

[0127] It should be understood that if the error does not exceed the preset error, it can be considered that the extrinsic parameter calculation is accurate and the calibration is successful, and then the extrinsic parameter coordinates of the target camera can be used as the extrinsic parameters.

[0128] Based on the camera extrinsic parameter calibration method provided in the above embodiments, combined with Figure 3 As shown, an embodiment of the present application also provides a camera extrinsic parameter calibration process. Among them, in the communication system of the target vehicle in this embodiment, DoIP (Diagnostic over Internet Protocol) can be integrated. A remote user can trigger the calibration task by sending a DoIP command to the communication system of the target vehicle, and at the same time carry the world coordinate byte stream generated in the previous step as an input parameter.

[0129] It should be understood that the communication system of the autonomous vehicle has integrated the DoIP protocol, enabling a remote user to trigger the calibration task by sending a DoIP command, which allows the extrinsic parameter calibration process to be carried out under remote control, providing a more convenient way for maintenance and management, thus realizing remote extrinsic parameter calibration.

[0130] Further based on Figure 2 As shown, the target reference object is 8 disks, and the camera extrinsic parameter calibration process may include:

[0131] Step 1: Obtain the center point coordinates of 8 disks in the world coordinate system (referring to actual point coordinates).

[0132] Among them, after the calibration task is triggered, the calibration algorithm can first parse the byte stream of the actual point coordinates into the center point coordinates of points A - H in the world coordinate system (i.e., actual point coordinates). And include A - F in the calibration point set, and list points G and H in the verification point set.

[0133] It should be understood that by automatically parsing the actual point coordinates, the operation difficulty of the user can be reduced, the error caused by manual parsing can be avoided, and the calibration accuracy can be improved.

[0134] Step 2: Extract the center point coordinates of 8 disks in the pixel coordinate system in the target image (referring to pixel point coordinates).

[0135] Specifically, after the camera calibration task is triggered, first the calibration algorithm obtains the image collected by the camera, and then uses traditional image processing algorithms to extract the coordinates of the disk center in the pixel coordinate system.

[0136] It should be understood that by extracting the pixel coordinates in the target image, compared with processing the target image using deep learning, the extraction process of this embodiment is easier to understand and master, and while ensuring the calculation accuracy, it can reduce the calculation complexity and computing power, and improve the real-time performance and stability of the algorithm.

[0137] Step 3: Process the pixel coordinates.

[0138] Since the pixel coordinate points of the center of the disc extracted from the target image are disordered, in order to achieve the same order as the world coordinate system, through system processing, the automatically ordered arrangement of the extracted pixel coordinates can be realized, improving the reliability of the calibration result and providing a reliable basis for the subsequent calibration process.

[0139] Step 4: Obtain the internal parameters of the target camera.

[0140] It should be understood that since the internal parameters have been embedded in the E 2 PROM of the target camera during its design. Therefore, when the target camera is started, the camera driver can automatically detect and identify the camera model, and read the corresponding internal parameters from the E 2 PROM, simplifying the external parameter calibration process.

[0141] Step 5: Calculate the external parameters of the target camera.

[0142] Specifically, given the center point coordinates of the disc in the pixel coordinate system and the world coordinate system and the internal parameters of the target camera, the external parameters can be calculated through a preset camera algorithm, and the rotation matrix and translation vector of the world coordinate system relative to the camera coordinate system can be obtained.

[0143] Step 6: Verify the external parameters of the target camera.

[0144] After the external parameter calculation is completed, the verification error is carried out in two steps:

[0145] First, verify the reprojection error of the calibration points. The six coordinate points A - F participating in the external parameter calculation are reprojected back to the pixel coordinate system through the internal and external parameters, and the error between the projected point coordinates after projection and the pixel point coordinates participating in the calculation is calculated. If the error exceeds the preset error, it is considered that the external parameter calibration of the target camera fails.

[0146] Then, verify the projection errors at two verification points G and H. By projecting the pixel coordinates of points G and H into the world coordinate system through the internal and external parameters, the point coordinates of G and H in the world coordinate system can be obtained. Calculate the error between the obtained point coordinates and the measured point coordinates, and determine whether the error exceeds the threshold. If it exceeds the threshold, it is determined that the calibration fails; otherwise, it is considered that the calibration is successful. After verifying the successful calibration, the calibration result will be automatically saved to the autonomous driving system for use by the perception system.

[0147] Step 7: Transmit the external parameter calibration status and result.

[0148] It should be understood that during the execution of the calibration, a calibration status flag can be set to monitor the status of the calibration process in real time. By monitoring the execution status of the calibration algorithm, it can be detected whether the external parameter calibration is in progress, as well as whether the calibration is successful or failed.

[0149] After successful calibration, the target external parameters of the target camera calculated through the external parameters will be automatically transmitted to the system. The external parameters will be encoded in hexadecimal byte stream or other formats to meet the needs of data transmission, and data transmission will be carried out through a secure communication protocol to ensure the integrity and security of the data.

[0150] Based on the camera external parameter calibration method provided in the above embodiments, refer to Figure 4 This figure is a schematic structural diagram of a camera external parameter calibration device provided in an embodiment of the present application. Combining Figure 4 As shown, the camera external parameter calibration device 400 provided in an embodiment of the present application may include:

[0151] An image acquisition module 401, configured to acquire a target image captured by a target camera to be calibrated of a target vehicle; the target image includes a target reference object; the target reference object indicates a reference object used for calibrating the target camera;

[0152] An actual point coordinate acquisition module 402, configured to acquire the actual point coordinates of the target reference object in the world coordinate system; the origin of the world coordinate system is the projection point of the target camera on the ground, the X-axis of the world coordinate system is parallel to the front direction of the target vehicle, the Y-axis of the world coordinate system is perpendicular to the front direction of the target vehicle in the horizontal direction, and the Z-axis of the world coordinate system is perpendicular to the ground;

[0153] A pixel point coordinate determination module 403, configured to determine the pixel point coordinates of the target reference object in the pixel coordinate system in the target image; the origin of the pixel coordinate system is the top left vertex of the target image, the X-axis of the pixel coordinate system is parallel to the width direction of the target image, and the Y-axis of the pixel coordinate system is parallel to the height direction of the target image;

[0154] An external parameter determination module 404, configured to determine the external parameters of the target camera according to the internal parameters, pixel coordinates, and actual coordinates of the target camera;

[0155] A calibration module 405, configured to calibrate the external parameters of the target camera.

[0156] As an example, before the image acquisition module 401, the apparatus 400 further includes:

[0157] A first determination unit, configured to determine the visible area of the target camera; the visible area is the area captured by the target camera;

[0158] A first setting unit, configured to set at least one set of reference groups within the visible area; each reference group includes two target reference objects; the two target reference objects are respectively arranged on both sides of the central axis of the target vehicle, the connection line between the two target reference objects is perpendicular to the central axis, and the perpendicular distances from the two target reference objects to the central axis are equal; the distances between each reference group and the target vehicle are different;

[0159] The image acquisition module 401 is configured to: acquire a target image captured by the target camera and including at least one set of reference objects.

[0160] As an example, if the target image includes at least two sets of reference groups, after the pixel coordinate determination module 403, the apparatus 400 further includes:

[0161] A second determination unit, configured to determine a target reference group among at least two sets of reference groups; the pixel distance between the two target reference objects in the target reference group is the largest; the pixel distance is determined according to the pixel coordinates of the two target reference objects;

[0162] A second setting unit, configured to use the pixel coordinates corresponding to the two target reference objects in the target reference group as verification point coordinates, and use the pixel coordinates corresponding to the two target reference objects in each reference group other than the target reference group as calibration point coordinates;

[0163] A sorting unit, configured to sort the two calibration point coordinates corresponding to each reference group based on the two verification point coordinates corresponding to the target reference group and a preset sorting rule; wherein the preset sorting rule is associated with the position order of at least two sets of reference groups in the world coordinate system.

[0164] As an example, the pixel coordinate determination module 403 includes:

[0165] A cropping unit, configured to perform regional cropping on the target image to obtain a target image area; the target image area includes the target image;

[0166] A processing unit, configured to perform preprocessing and binarization processing on the target image area to obtain a binarized image;

[0167] An extraction unit, configured to extract the contour of the target reference object from the binary image;

[0168] A center point determination unit, configured to determine the center point of the target reference object according to the contour;

[0169] A pixel point coordinate determination unit, configured to determine the pixel point coordinates of the target reference object in the pixel coordinate system according to the center point.

[0170] As an example, the external parameter determination module 404 is configured to:

[0171] Determine the internal parameter matrix according to the internal parameters;

[0172] Perform normalization processing on the pixel point coordinates based on the internal parameter matrix to obtain the normalized point coordinates;

[0173] Determine the external parameters of the target camera according to the normalized point coordinates and the actual point coordinates through a preset camera pose algorithm.

[0174] As an example, the calibration module 405 includes:

[0175] A projection unit, configured to determine the projection point coordinates of the actual point coordinates in the pixel coordinate system according to the internal parameters and the external parameters;

[0176] A calibration unit, configured to calibrate the external parameters of the target camera according to the projection point coordinates and the pixel point coordinates.

[0177] As an example, the calibration unit is configured to:

[0178] Determine the error between the projection point coordinates and the pixel point coordinates;

[0179] Judge whether the error is greater than a preset error; if the error is greater than the preset error, re-calibrate the external parameters of the target camera; if the error is less than or equal to the preset error, the calibration is successful, and the external parameters are used as the target external parameters of the target camera.

[0180] The camera external parameter calibration device provided by the embodiments of the present application has the same beneficial effects as the camera external parameter calibration method provided by the above embodiments, and thus will not be elaborated herein.

[0181] The embodiments of the present application further provide corresponding devices and computer storage media for implementing the solutions provided by the embodiments of the present application.

[0182] Wherein, the device includes a memory and a processor, the memory is used to store instructions or codes, and the processor is used to execute the instructions or codes so that the device executes the camera external parameter calibration method described in any embodiment of the present application.

[0183] The computer storage medium stores code, and when the code is run, the device running the code implements the camera external parameter calibration method according to any embodiment of the present application.

[0184] It should be noted that the embodiments in this specification are all described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the device and equipment embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For the relevant parts, reference can be made to the partial description of the method embodiments. The device and equipment embodiments described above are only illustrative. The units described as separate components may or may not be physically separated, and the components referred to as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative efforts.

[0185] The "first", "second" (if any) in the names such as "first" and "second" mentioned in the embodiments of the present application are only used as name identifiers and do not represent the first and second in sequence.

[0186] From the description of the above embodiments, it can be clearly understood by those skilled in the art that all or part of the steps in the above embodiment methods can be implemented by means of software plus a general hardware platform. Based on such an understanding, the technical solution of the present application can be embodied in the form of a software product. The computer software product can be stored in a storage medium, such as a read-only memory (ROM) / RAM, magnetic disk, optical disc, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network communication device such as a router) to execute the methods described in each embodiment or some parts of the embodiments of the present application.

[0187] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A camera extrinsic calibration method, characterized in that: The method comprises: Acquire a target image captured by a target camera to be calibrated of a target vehicle; the target image includes a target reference object; the target reference object indicates a reference object used to calibrate the target camera; Obtain the actual point coordinates of the target reference object in a world coordinate system; the origin of the world coordinate system is the projection point of the target camera on the ground as the origin, the X-axis of the world coordinate system is parallel to the front direction of the target vehicle, the Y-axis of the world coordinate system is perpendicular to the front direction of the target vehicle in the horizontal direction, and the Z-axis of the world coordinate system is perpendicular to the ground; Determine the pixel coordinates of the target reference object in the target image in a pixel coordinate system; the origin of the pixel coordinate system is the vertex of the upper left corner of the target image, the X axis of the pixel coordinate system is parallel to the width direction of the target image, and the Y axis of the pixel coordinate system is parallel to the height direction of the target image; Determine the external parameters of the target camera according to the internal parameters of the target camera, the pixel point coordinates and the actual point coordinates; The external parameters of the target camera are calibrated.

2. The method according to claim 1, characterized in that Before acquiring the target image captured by the target camera to be calibrated of the target vehicle, the method further includes: Determine the visible area of ​​the target camera; the visible area is the area photographed by the target camera; At least one reference group is set in the visible area; each of the reference groups includes two target reference objects; the two target reference objects are respectively set on both sides of the central axis of the target vehicle, the line between the two target reference objects is perpendicular to the central axis, and the vertical distances from the two target reference objects to the central axis are equal; the distances between the reference groups and the target vehicle are different; The step of obtaining a target image captured by a target camera to be calibrated of the target vehicle includes: A target image including at least one set of reference objects captured by the target camera is obtained.

3. The method according to claim 2, characterized in that If the target image includes at least two reference groups, after determining the pixel coordinates of the target reference object in the target image in a pixel coordinate system, the method further includes: Determine a target reference group among the at least two reference groups; the pixel distance between two target reference objects in the target reference group is the largest; the pixel distance is determined according to the pixel point coordinates of the two target reference objects; Using pixel point coordinates corresponding to two target reference objects in the target reference group as verification point coordinates, and using pixel point coordinates corresponding to two target reference objects in each reference group other than the target reference group as calibration point coordinates; Based on the two verification point coordinates corresponding to the target reference group and a preset sorting rule, the two calibration point coordinates corresponding to each reference group are sorted; wherein the preset sorting rule is associated with the position order of at least two reference groups in the world coordinate system.

4. The method according to claim 1, characterized in that: The determining the pixel coordinates of the target reference object in the target image in a pixel coordinate system includes: Performing regional cropping on the target image to obtain a target image region; the target image region includes the target image; Preprocessing and binarizing the target image region to obtain a binarized image; Extracting the outline of the target reference object from the binary image; Determine the center point of the target reference object according to the outline; The pixel coordinates of the target reference object in a pixel coordinate system are determined according to the center point.

5. The method according to claim 1, characterized in that The step of determining the external parameters of the target camera according to the internal parameters of the target camera, the pixel point coordinates and the actual point coordinates includes: Determine an internal parameter matrix according to the internal parameters; Normalizing the pixel point coordinates based on the intrinsic parameter matrix to obtain normalized point coordinates; According to the normalized point coordinates and the actual point coordinates, the external parameters of the target camera are determined by a preset camera posture algorithm.

6. The method according to claim 1, characterized in that The calibrating the external parameters of the target camera includes: Determine the projection point coordinates of the actual point coordinates in the pixel coordinate system according to the internal parameters and the external parameters; The external parameters of the target camera are calibrated according to the projection point coordinates and the pixel point coordinates.

7. The method according to claim 6, characterized in that The calibrating the external parameters of the target camera according to the projection point coordinates and the pixel point coordinates includes: Determining the error between the projection point coordinates and the pixel point coordinates; Determine whether the error is greater than a preset error; if the error is greater than the preset error, recalibrate the external parameters of the target camera; if the error is less than or equal to the preset error, the calibration is successful, and the external parameters are used as the target external parameters of the target camera.

8. A camera external parameter calibration device, characterized in that: The device comprises: An image acquisition module, used to acquire a target image captured by a target camera to be calibrated of a target vehicle; the target image includes a target reference object; the target reference object indicates a reference object used to calibrate the target camera; An actual point coordinate acquisition module is used to acquire the actual point coordinates of the target reference object in a world coordinate system; the origin of the world coordinate system is the projection point of the target camera on the ground as the origin, the X-axis of the world coordinate system is parallel to the front direction of the target vehicle, the Y-axis of the world coordinate system is perpendicular to the front direction of the target vehicle in the horizontal direction, and the Z-axis of the world coordinate system is perpendicular to the ground; a pixel coordinate determination module, used to determine the pixel coordinates of the target reference object in the target image in a pixel coordinate system; the origin of the pixel coordinate system is the vertex of the upper left corner of the target image, the X axis of the pixel coordinate system is parallel to the width direction of the target image, and the Y axis of the pixel coordinate system is parallel to the height direction of the target image; An external parameter determination module, used to determine the external parameters of the target camera according to the internal parameters of the target camera, the pixel point coordinates and the actual point coordinates; The calibration module is used to calibrate the external parameters of the target camera.

9. A computer device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the camera extrinsic parameter calibration method as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the camera extrinsic parameter calibration method as described in any one of claims 1 to 7.

Images