Camera calibration method and device and storage medium

Abstract

The invention provides a camera calibration method and device and a storage medium. The scheme comprises the following steps: acquiring at least one frame of image about a calibration board acquired by a camera; obtaining a mark coordinate system coordinate of each first feature point according to the attribute information of each mark on the calibration board; for at least one frame of image, executing the following processing frame by frame: obtaining a pixel coordinate system coordinate of a first feature point in the current frame of image, and obtaining a transformation relation from a mark coordinate system of each mark on the calibration plate to a camera coordinate system according to the mark coordinate system coordinate and the pixel coordinate system coordinate of the first feature point; obtaining an initial value of the transformation relation from the calibration plate coordinate system to the camera coordinate system of the calibration plate according to the transformation relation from the mark coordinate system to the camera coordinate system; and on the basis of the transformation relation initial value, obtaining a transformation relation optimization value from the calibration plate coordinate system to the camera coordinate system corresponding to the current frame image. The robustness, the accuracy and the universality of the camera calibration method can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to a camera calibration method, device, and storage medium. Background Art

[0002] Sensor calibration in the field of autonomous driving involves the calibration of internal and external parameters of sensors such as cameras, Lidar (Light detection and ranging), millimeter-wave radar, inertial navigation, and integrated navigation, so as to achieve the unification of the coordinates of each sensor and provide reliable input for perception algorithms. The calibration methods of cameras and other sensors are mainly divided into target-based calibration and non-target-based calibration methods. The non-target-based calibration methods can be further divided into environmental scene-based calibration methods and motion-based calibration methods. For target-based calibration methods and environmental scene-based calibration methods, effective feature extraction and association of the same features of the sensor to be calibrated (such as Lidar) are the basis for optimizing the solution of internal and external parameters. Therefore, robust, accurate, effectively spatially distributed feature points with identification IDs and including two-dimensional and three-dimensional information are the ideal features required for camera calibration.

[0003] At present, the extraction of effective features is still a difficult point in the target-based camera calibration method. The extraction of effective features still has problems such as low extraction efficiency, high requirements for the spatial distribution of the calibration plate, insufficient number of feature points, and poor feature consistency. As a result, the robustness and versatility of the target-based camera calibration method are low. At the same time, the success rate and accuracy of camera calibration are difficult to guarantee, making it unsuitable for mass production calibration in the field of autonomous driving. Summary of the Invention

[0004] In view of this, the present disclosure provides a camera calibration method, apparatus, and storage medium to improve the robustness, accuracy, and versatility of a target-based camera calibration method.

[0005] According to a first aspect of the present disclosure, a camera calibration method is provided, the method comprising: acquiring at least one frame of image of a calibration plate captured by a camera; obtaining the marker coordinate system coordinates of each first feature point based on the attribute information of each marker on the calibration plate; performing the following processing frame by frame for at least one frame of image: obtaining the pixel coordinate system coordinates of the first feature point in the current frame image, obtaining the transformation relationship from the marker coordinate system to the camera coordinate system of each marker on the calibration plate based on the marker coordinate system coordinates and the pixel coordinate system coordinates of the first feature point; obtaining an initial value of the transformation relationship from the calibration plate coordinate system to the camera coordinate system of the calibration plate based on the transformation relationship from the marker coordinate system to the camera coordinate system; and obtaining an optimized value of the transformation relationship from the calibration plate coordinate system to the camera coordinate system corresponding to the current frame image based on the initial value of the transformation relationship.

[0006] According to a second aspect of the present disclosure, a camera calibration device is provided, comprising: one or more processors, and a memory storing a program, wherein the program comprises instructions that, when executed by the processor, cause the processor to execute the method described in the first aspect.

[0007] According to a third aspect of the present disclosure, a computer-readable storage medium storing a program is provided, wherein the program includes instructions, which, when executed by one or more processors of a computing device, cause the computing device to execute the method described in the first aspect above.

[0008] As can be seen from the above technical solution, the present disclosure first solves the transformation relationship from the marker coordinate system of a single marker to the camera coordinate system, and then uses the transformation relationship from the marker coordinate system of each marker to the camera coordinate system to solve the initial value of the transformation relationship from the calibration plate coordinate system to the camera coordinate system. Finally, based on the initial value of the transformation relationship, the optimized value of the transformation relationship from the calibration plate coordinate system to the camera coordinate system is obtained through optimization solution. The three-dimensional coordinates of the first feature point of each marker on the calibration plate will participate in the optimization solution of the transformation relationship, and the detection accuracy will not be reduced due to the omission of individual markers. The solution is solved in a coarse-to-fine manner, which improves the accuracy and robustness of the optimization solution of the transformation relationship, and the optimization solution method also has high versatility. The optimized value of the transformation relationship obtained in this way can be directly used for camera calibration, which can significantly improve the robustness, accuracy and versatility of the target-based camera calibration method. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0010] Figure 1 A system architecture diagram applicable to the embodiments of the present disclosure;

[0011] Figure 2 A schematic diagram of a process flow of a camera calibration method provided by an embodiment of the present disclosure;

[0012] Figure 3 This is an example diagram of a calibration plate involved in an embodiment of the present disclosure;

[0013] Figure 4 A schematic diagram of a marking coordinate system and a calibration plate coordinate system involved in an embodiment of the present disclosure;

[0014] Figure 5 An example diagram of a camera coordinate system and a pixel coordinate system involved in the embodiments of the present disclosure;

[0015] Figure 6 This is an example diagram of the associated storage of the calibration plate identifier, the marking identifier, and the coordinates of the first feature point involved in the embodiments of the present disclosure;

[0016] Figure 7 is another flowchart of the camera calibration method according to an embodiment of the present disclosure;

[0017] Figure 8 A schematic diagram of a process for calibrating a camera and a lidar according to an embodiment of the present disclosure;

[0018] Figure 9 Schematic diagram of an exemplary implementation process for calibrating a camera and a lidar based on the initial values ​​of the camera's intrinsic parameters and extrinsic parameters in an embodiment of the present disclosure;

[0019] Figure 10 Another schematic diagram of a process for calibrating a camera and a lidar according to an embodiment of the present disclosure; Figure 11 A modular structural block diagram of the calibration device provided in an embodiment of the present disclosure;

[0020] Figure 12 A schematic block diagram of a calibration device in the form of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] The following will clearly and completely describe the technical solutions in the embodiments of the present disclosure in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0022] The terms used in the embodiments of the present disclosure are only for the purpose of describing specific embodiments and are not intended to limit the present disclosure. The singular forms "a", "the" and "the" used in the embodiments of the present disclosure and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should be understood that the term "and / or" used in this article is merely a kind of association relationship describing associated objects, indicating that there can be three relationships, for example, A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are an "or" relationship. Depending on the context, the word "if" as used herein can be interpreted as "at the time of..." or "when..." or "in response to determining..." or "in response to detecting." Similarly, depending on the context, the phrase "if it is determined" or "if detecting (stated condition or event)" can be interpreted as "when determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)".

[0023] Explanation of terms:

[0024] Calibration room: The calibration room is more conducive to forming a standardized calibration operation process. By arranging calibration plates (also called targets) suitable for feature extraction of various types of sensors in the calibration room, the robustness and accuracy of sensor parameter calibration can be improved. It can also be connected to the vehicle production process to realize sensor calibration before the vehicle leaves the factory, ensuring the safety of autonomous driving.

[0025] Marker Coordinate System: Each marker has a corresponding marker coordinate system. This is a three-dimensional rectangular coordinate system with a specific point in the marker (either the marker's center or a vertex) as its origin. In some embodiments, the plane formed by the x-axis and the y-axis of this three-dimensional rectangular coordinate system is the plane where the marker resides, and the z-axis of the three-dimensional rectangular coordinate system is perpendicular to the plane where the marker resides.

[0026] Calibration plate coordinate system: A three-dimensional rectangular coordinate system with the geometric center of the calibration plate or another selected point on the calibration plate as its origin. In some embodiments, the plane formed by the abscissa and ordinate axes of this three-dimensional rectangular coordinate system is the calibration plate plane, and the z-axis of the three-dimensional rectangular coordinate system is perpendicular to the calibration plate plane.

[0027] Camera coordinate system: a three-dimensional rectangular coordinate system with the camera as its origin.

[0028] Pixel coordinate system: Usually defined in such a way that the origin is located in the upper left corner of the image, the horizontal axis (u) is parallel to the horizontal axis of the camera coordinate system to the right, and the vertical axis (v) is parallel to the vertical axis of the camera coordinate system downward.

[0029] Camera extrinsics: including the rotation matrix and translation variables between the camera coordinate system and other coordinate systems. The rotation matrix and translation vector determine the position of the camera in other coordinate systems.

[0030] Iterative Closest Point (ICP): A point cloud matching algorithm that uses ICP to solve the transformation relationship between two known 3D coordinate systems, given the coordinates of a 3D point in two 3D coordinate systems and their corresponding relationship. In the disclosed embodiment, the coordinates of the first feature point and / or the second feature point in different coordinate systems are sequentially read according to the plate identification and marking identification to directly obtain the point correspondence relationship between the different coordinate systems. This eliminates the need to execute the point correspondence determination process, greatly simplifying the ICP problem construction process, reducing computational complexity and computational effort, and reducing the use of computing resources. This reduces hardware costs while also improving processing efficiency.

[0031] PnP (Perspective-n-Point): is a method for solving the motion of three-dimensional to two-dimensional point pairs. When the three-dimensional coordinates of n (n is an integer greater than 1) feature points in the world coordinate system and the pixel coordinates of these points are known, the camera's pose in the world coordinate system can be estimated through PnP. If the world coordinate system is selected as the lidar coordinate system, PnP can be used to solve the camera's pose in the lidar coordinate system, that is, the pose of the camera coordinate system relative to the lidar coordinate system. This pose is the transformation relationship between the camera coordinate system and the lidar coordinate system. In the embodiment of the present disclosure, the coordinates of the first feature point or the second feature point in different coordinate systems are read in sequence according to information such as plate identification and mark identification, so that the point correspondence in different coordinate systems can be directly obtained. There is no need to perform the point correspondence determination process, which can greatly simplify the construction process of the PnP problem, reduce computational complexity and amount of calculation, reduce the use of computing resources, reduce hardware costs, and improve processing efficiency.

[0032] High reflectivity points: Feature points with high laser intensity values ​​in the point cloud data. High reflectivity points on the calibration plate can be feature points with higher reflectivity than surrounding points on the calibration plate. High reflectivity points can be formed by placing high-reflectivity materials at specific locations on the calibration plate.

[0033] In order to facilitate the understanding of the embodiments of the present disclosure, first Figure 1 The system architecture applicable to the embodiments of the present disclosure is described. Figure 1 An exemplary system architecture diagram to which the present disclosure can be applied is shown in FIG. Figure 1The system primarily includes sensors such as cameras, lidar, millimeter-wave radar, inertial navigation, and integrated navigation, as well as a calibration device. Each sensor communicates with the calibration device. The camera is used to capture images of the environment within its field of view. The calibration device can be used to extract feature points based on the camera images and the point cloud data provided by the lidar, and then calibrate the sensor based on the feature points. The camera can be, but is not limited to, a short-focus / telephoto camera using a pinhole model or a fisheye camera using a Scaramuzza model.

[0034] The system can be deployed on different platforms to perform camera calibration, depending on the needs of different scenarios. For example, the system can be installed on a mobile platform such as a vehicle or robot, and the mobile platform can be stopped and aligned at a designated location within a calibration scenario, such as a calibration room (e.g., the center of the calibration room).

[0035] As a possible implementation method, the calibration device can be set on a computer terminal with strong computing power deployed on the sensor-carrying platform, or it can be set on a server side deployed independently of the sensor-carrying platform. The calibration device can communicate with each sensor on the carrying platform through various wireless communication methods, or it can be interconnected with each sensor through a physical interface.

[0036] It should be understood that Figure 1 The number of calibration devices and sensors in the figure is only for illustration. Any number and type of sensors and calibration devices may be used as required.

[0037] Figure 2 This is a flow chart of the camera calibration method provided by the embodiment of the present disclosure. The method can be performed by Figure 1 The calibration device in the system shown is performed. Figure 2 , the method may include the following steps:

[0038] Step 201: Acquire at least one frame of image of a calibration plate captured by a camera, wherein a plurality of marks are provided on the calibration plate, and the calibration plate includes a first feature point of each mark.

[0039] Step 203: Obtain the marker coordinate system coordinates of each first feature point according to the attribute information of each marker on the calibration plate.

[0040] Step 205: Perform the following processing on at least one frame of image to obtain the optimized value of the transformation relationship from the calibration plate coordinate system to the camera coordinate system corresponding to at least one frame of image: first, determine the pixel coordinate system coordinates of the first feature point from the current frame image; second, obtain the transformation relationship from the marker coordinate system to the camera coordinate system of each marker on the calibration plate according to the marker coordinate system coordinates and the pixel coordinate system coordinates of each first feature point on the calibration plate; third, obtain the initial value of the transformation relationship from the calibration plate coordinate system to the camera coordinate system of the calibration plate according to the transformation relationship from the marker coordinate system to the camera coordinate system; finally, based on the initial value of the transformation relationship, obtain the optimized value of the transformation relationship from the calibration plate coordinate system to the camera coordinate system corresponding to the current frame image through optimization solution.

[0041] As described above, the present disclosure first solves the transformation relationship from the marker coordinate system of a single marker to the camera coordinate system, and then uses the transformation relationship from the marker coordinate system of each marker to the camera coordinate system to solve the initial value of the transformation relationship from the calibration plate coordinate system to the camera coordinate system. Finally, based on the initial value of the transformation relationship, the optimized value of the transformation relationship from the calibration plate coordinate system to the camera coordinate system is obtained through optimization solution. The three-dimensional coordinates of the first feature point of each marker on the calibration plate will participate in the optimization solution of the transformation relationship, and the detection accuracy will not be reduced due to the omission of individual markers. The solution is solved in a coarse-to-fine manner, which improves the accuracy and robustness of the optimization solution of the transformation relationship, and the optimization solution method also has high versatility. The optimized value of the transformation relationship obtained in this way can be directly used for camera calibration, which can significantly improve the robustness, accuracy and versatility of the target-based camera calibration method.

[0042] The calibration plate, mark, calibration hole, various feature points and coordinate system involved in the embodiments of the present disclosure are first described in detail below.

[0043] The calibration plate, also known as a target, is primarily used for sensor calibration. In some embodiments of the present disclosure, one or more marks may be provided on the calibration plate, each with a corresponding marking identifier. In some other embodiments of the present disclosure, the calibration plate may further include one or more calibration holes.

[0044] In some embodiments, to better capture effective features during calibration, the marks on the calibration plate may be arranged symmetrically, and the calibration holes on the calibration plate may also be arranged symmetrically.

[0045] The mark can be printed, drawn, or pasted on the calibration plate. The mark can be a regular shape such as a quadrilateral, triangle, circle, or other. The mark can also be an irregular shape. In some embodiments of the present disclosure, the mark can be, but is not limited to, a QR code, a picture, etc., and each mark has a unique mark identifier. For example, the mark can be, but is not limited to, an ArUco marker (arucomarker), i.e., an ArUco QR code.

[0046] The calibration hole is preferably a regular shape. For example, the calibration hole can be, but is not limited to, a circular through hole (referred to as a circular hole), a square through hole, a rectangular through hole, or a through hole of any other shape.

[0047] Figure 3 An example diagram of a calibration plate according to an embodiment of the present disclosure is shown. Figure 3 The calibration plate 300 is provided with four symmetrically distributed marks and four symmetrically distributed circular holes. The four marks can be QR codes or pictures. Each mark can have the same size, and each circular hole can have the same size.

[0048] Different types of calibration plates can be designed as needed to calibrate different sensors in different scenarios. To facilitate feature capture and camera calibration calculations, different calibration plates, different markings on the plates, and different calibration holes on the plates can be of the same or different sizes, and this disclosure does not impose any restrictions on this.

[0049] The first feature point can be pre-selected as needed. In order to more accurately capture the features of the first feature point, the first feature point can be a feature point on the outline of the mark, for example, see Figure 4 For example, the first feature point can be Figure 4 In the example, the four corner points p5 to p8 of the quadrilateral are marked.

[0050] The second feature point of the calibration plate can also be pre-selected according to actual needs. Similarly, in order to facilitate accurate capture of feature points, the second feature point of the calibration plate can preferably be a second feature point about the structural contour of the calibration plate. Specifically, the second feature point of the calibration plate can include but is not limited to: feature points on the contour of the calibration plate, feature points on the contour of the calibration hole in the calibration plate, the center point of the calibration hole, and the high reflection point of the calibration plate. The feature points on the contour of the calibration plate can include but are not limited to the edge corner points of the calibration plate. The feature points on the contour of the calibration hole in the calibration plate can be the corner points of a polygonal calibration hole.

[0051] In one example, if the calibration plate is not provided with a calibration hole, the second characteristic point of the calibration plate may include characteristic points on the contour of the calibration plate. For example, the characteristic points on the contour of the calibration plate may include the following: Figure 4 The four edge corner points p1 to p4 of the quadrilateral calibration plate in the example.

[0052] In one example, if a calibration hole is provided on the calibration plate, the second characteristic points of the calibration plate may include characteristic points on the contour of the calibration plate, characteristic points on the contour of the calibration hole in the calibration plate, and the center point of the calibration hole. Figure 4 For example, the second characteristic points of the calibration plate may include four edge corner points p1 to p4 of the calibration plate and a center point p9 of a circular calibration hole on the calibration plate.

[0053] The attribute information of the calibration plate may include but is not limited to the plate identification of the calibration plate, the outline size of the calibration plate, the characteristic points on the outline of the calibration plate, the number of marks on the calibration plate, the sequence number of each mark, the mark identification, the number of calibration holes on the calibration plate, the sequence number of each calibration hole, the identification of the calibration hole and other information.

[0054] The attribute information of the marking plate also includes attribute information of the markers. The attribute information of the markers may include, but is not limited to, the plate identification of the calibration plate to which it belongs, the position information of the markers, the marker identification, the outline size, the number of first feature points, and the relative positional relationship of each first feature point relative to a specific corner point of the calibration plate. The positional information of the markers may include the relative positional relationship of the center point or multiple corner points of different markers relative to a specific corner point of the calibration plate, as well as the relative positional relationship between different markers.

[0055] The calibration plate's attribute information also includes the attribute information of the calibration holes. This attribute information may include, but is not limited to, the plate identifier of the calibration plate, the location information of the calibration hole, the outline dimensions of the calibration hole, and the number of second characteristic points of the calibration hole. The location information of the calibration holes includes the relative positional relationship of the center point or multiple corner points of the calibration hole relative to a specific corner point of the calibration plate, as well as the relative positional relationship between different calibration holes.

[0056] In one example, the position information of the mark can also be represented by the adjacent edge distance between the mark and the calibration plate, and the adjacent edge distance between the mark and the mark. Similarly, the position information of the calibration hole can also be represented by the adjacent edge distance between the calibration hole and the calibration plate, the adjacent edge distance between the calibration hole and the mark, and the adjacent edge distance between the calibration hole and the calibration hole. Among them, the adjacent edge distance between the mark and the calibration plate includes the distance between the edge of the mark and the adjacent edge of the calibration plate, and the adjacent edge distance between the mark and the mark includes the distance between the edge of the mark and the adjacent edges of other marks. The adjacent edge distance between the calibration hole and the calibration plate includes the distance between the center point or other point of the calibration hole and the adjacent edge of the calibration plate.

[0057] Outline dimensions can be expressed by side lengths. For example, the outline dimensions of a calibration plate can include the length and width of the calibration plate, the outline dimensions of each mark can include the length and width of the mark, and the outline dimensions of a calibration hole can include the diameter or radius of a circular hole, the length and width of a quadrilateral hole, etc.

[0058] The information of the second characteristic point of the calibration plate may include the position and identification of the characteristic point on the contour of the calibration plate, the position and identification of the center point of the calibration hole, such as the position and identification of multiple corner points and / or center point of the calibration hole.

[0059] The information of the first feature point may include the position and identification of the first feature point. Figure 3For example, if the first feature points are selected as the four corner points of the marker, the corresponding feature point information may include the identifiers of the four corner points on the marker and their distances to the edge of the calibration plate. The attribute information of the calibration plate, the attribute information of the marker, and the attribute information of the calibration hole can be associated and stored using the plate identifier, marker identifier, etc.

[0060] The following is a detailed description of an exemplary implementation of step 201 in the camera calibration method according to an embodiment of the present disclosure.

[0061] In step 201, obtaining at least one frame of image of the calibration plate captured by the camera may include: respectively obtaining at least one frame of image of multiple calibration plates captured by the camera, different calibration plates may be provided with different marks, each calibration plate has a corresponding plate identifier, and each mark has a corresponding mark identifier.

[0062] The camera calibration accuracy can be improved by arranging various types of calibration plates within the field of view (FOV) of the camera. When multiple calibration plates are set, different calibration plates can be distributed at different spatial positions, different heights, different distances, and different angles within the field of view of the camera. Different calibration plates can be distinguished by plate identifiers, which may be, but are not limited to, the number, position, name, or other information that can uniquely represent the calibration plate. For example, the plate identifier of each calibration plate can be numbered according to the type of calibration plate and the mark on the calibration plate. Different calibration plates may be provided with different marks and / or different calibration holes, and different calibration plates or different marks on the same calibration plate are distinguished by mark identifiers, and different calibration plates or different calibration holes on the same calibration plate are distinguished by setting calibration hole identifiers.

[0063] In step 201, multiple frames of images can be acquired so that feature point extraction and camera calibration can be performed by integrating the processing results of the multiple frames of images. For example, the camera can perform continuous frame imaging of various types of calibration plates within its field of view while being fixed and stationary, select n frames (n is an integer greater than 1) containing images of the calibration plates, and transmit these n frames of images to the calibration device for subsequent feature point extraction and camera calibration processes. In this way, camera calibration can be completed directly by extracting two-dimensional and three-dimensional feature points through n consecutive frames of images at a time, avoiding the need to continuously move the calibration plate during the calibration process and achieving the efficiency required for production line calibration.

[0064] In step 201, acquiring at least one frame of an image of a calibration plate captured by a camera may include acquiring a sequence of images captured by the camera, performing image detection on each frame of the sequence, and determining at least one frame of the image that contains the calibration plate and has a number of markers on the calibration plate that is greater than or equal to a first predetermined threshold. The first predetermined threshold may be set based on the detection requirements of a specific scene. For example, the first predetermined threshold may be set to the number of markers required for the minimum detection requirement of the current scene. Thus, if a frame of image does not contain the calibration plate or the number of markers detected on the calibration plate in the frame of image is less than the minimum detection requirement, the frame of image may be discarded. If a frame of image contains the calibration plate and the number of markers detected on the calibration plate in the frame of image is greater than or equal to the minimum detection requirement, the calibration plate in the frame of image may be included in the feature point extraction and calibration process. Thus, a robust verification mechanism for detecting a number of markers on a calibration plate that is greater than or equal to the minimum detection requirement may be utilized to discard poorly observed images, further improving feature point extraction precision, extraction efficiency, robustness, and accuracy.

[0065] The coordinate system involved in the embodiment of the present disclosure and an exemplary implementation of step 203 in the camera calibration method are described in detail below.

[0066] In step 203, for each marker, a marker coordinate system of the marker may be constructed first, and then the marker coordinate system coordinates of the first feature point are calculated based on the attribute information of the marker. The marker coordinate system coordinates of the first feature point are the three-dimensional coordinates of the first feature point on the marker in the marker coordinate system of the marker.

[0067] Furthermore, step 203 may also include: for each calibration plate, the calibration plate coordinate system coordinates of various feature points such as the first feature point and / or the second feature point on the calibration plate may be pre-stored. Specifically, for each calibration plate, the calibration plate coordinate system of the calibration plate may be first constructed, and then the calibration plate coordinate system coordinates of various feature points such as the first feature point and / or the second feature point on the calibration plate may be calculated based on the attribute information of the calibration plate, the attribute information of each mark on the calibration plate, and the attribute information of each calibration hole on the calibration plate. Taking the first feature point as an example, the calibration plate coordinate system coordinates of the first feature point are the three-dimensional coordinates of the first feature point on the mark in the calibration plate coordinate system of the calibration plate to which the mark belongs.

[0068] The marker coordinate system and calibration plate coordinate system can be flexibly constructed according to actual needs. For example, the calibration plate coordinate system of each calibration plate can be constructed with the calibration plate center as the origin, the calibration plate plane as the xy axis plane, and the z axis perpendicular to the calibration plate plane. The marker coordinate system of each marker can be constructed with the marker center as the origin, the marker plane as the xy axis plane, and the z axis perpendicular to the marker plane. Figure 4An example diagram showing the calibration plate coordinate system and the marker coordinate system of the present disclosure is shown in FIG. Figure 4 The calibration plate coordinate system, the marker coordinate system, and the camera coordinate system are all Cartesian three-dimensional rectangular coordinate systems. The origin of the marker coordinate system x1y1z1 is selected as the geometric center O1 of the marker, and the origin of the calibration plate coordinate system x2y2z2 is selected as the geometric center O2 of the calibration plate. It should be understood that Figure 4 These are all exemplary implementations, and the specific implementations of the coordinate systems in the embodiments of the present disclosure are not limited thereto.

[0069] Figure 4 The calibration plate shown includes a mark and a center calibration hole (i.e., a circular hole). The second feature points of the calibration plate can include the edge corner points p1~p4 of the calibration plate and the center point of the calibration hole (i.e., the center of the circular hole) p9. The first feature points of the mark can include the four corner points p5~p8 of the mark.

[0070] It should be noted that Figure 4 For ease of calibration, the calibration plate typically includes two or more symmetrically arranged marks and / or two or more calibration holes.

[0071] The camera coordinate system is a three-dimensional Cartesian coordinate system, while the pixel coordinate system is a two-dimensional Cartesian coordinate system. For example, the camera coordinate system can be constructed with the camera center as the origin, the xy-axis plane parallel to the camera image plane, and the z-axis perpendicular to the xy-axis plane. The pixel coordinate system can be constructed with the upper left corner of the camera image plane as the origin, the horizontal axis x facing right, and the vertical axis y facing forward. Figure 5 An example diagram of a camera coordinate system and a pixel coordinate system in an embodiment of the present disclosure is shown. Figure 5 In the example, the origin of the camera coordinate system x3y3z3 is selected as the camera center O3, and the origin O of the pixel coordinate system uOv is selected as the upper left corner O of the image plane.

[0072] The following is a detailed description of an exemplary implementation of step 205 in the camera calibration method according to an embodiment of the present disclosure.

[0073] In step 205, after determining the pixel coordinate system coordinates of each first feature point, the current frame image can also be checked through the reprojection error of the calibration plate, so that the frame image and the pixel coordinate system coordinates of the corresponding first feature points can be discarded when the frame image is a poor observation image.

[0074] Specifically, if the reprojection error of the calibration plate is large (for example, greater than the second predetermined threshold), it means that the feature point features detected based on the image have a large error compared with the actual features of the feature points on the calibration plate, that is, the quality of the current frame image does not meet the calibration requirements of the calibration plate, and the current frame image is regarded as a poor observation image for the calibration plate, and is discarded and does not participate in subsequent processing. If the reprojection error of the calibration plate is within a reasonable range (for example, less than or equal to the second predetermined threshold), it means that the feature point features detected based on the image are basically consistent with the actual features of the feature points, that is, the quality of the current frame image meets the calibration requirements of the calibration plate and can continue to participate in subsequent processing.

[0075] In specific applications, the second predetermined threshold can be an empirical value and can be flexibly adjusted in different application scenarios. It should be understood that a frame image may not meet the calibration requirements of calibration plate 1, but may meet the calibration requirements of calibration plate 2. Therefore, the embodiment of the present disclosure determines whether the current frame image is suitable for calibration of each calibration plate based on the reprojection error of each calibration plate.

[0076] The reprojection error of the calibration plate can be obtained in various applicable ways. In some embodiments of the present disclosure, in step 205, the pixel coordinate system coordinates of the first feature point of each mark in the current frame image can be obtained by performing feature point detection on the current frame image, and the pixel coordinate system coordinates are the detection coordinates in the pixel coordinate system. After the feature point detection is performed on the current frame image, the projection coordinates of each first feature point on the calibration plate in the pixel coordinate system can be obtained based on the previously obtained transformation relationship optimization value and the attribute information of each mark on the calibration plate, and then the reprojection error of the calibration plate can be determined based on the detection coordinates and projection coordinates of each first feature point on the calibration plate in the pixel coordinate system.

[0077] The projection coordinates of each first feature point on the calibration plate in the pixel coordinate system can be obtained by various applicable methods. In some embodiments of the present disclosure, obtaining the projection coordinates of each first feature point in the pixel coordinate system based on the transformation relationship optimization value and the attribute information of each mark on the calibration plate can include: first, determining the camera projection model according to the type information of the camera; second, calculating the first camera coordinate system coordinates of each first feature point on the calibration plate based on the transformation relationship optimization value from the calibration plate coordinate system to the camera coordinate system corresponding to the previous frame image and the calibration plate coordinate system coordinates of each first feature point; finally, calculating the projection coordinates of each first feature point on the calibration plate in the pixel coordinate system based on the camera projection model and the first camera coordinate system coordinates of each first feature point on the calibration plate. The first camera coordinate system coordinates are the first coordinates in the camera coordinate system.

[0078] First, the camera projection model is determined based on the camera type information, and then the projection coordinates are calculated. This allows for the calculation of reprojection errors in a manner appropriate to the camera type (e.g., short-focus cameras, long-focus cameras, fisheye cameras, etc.) during the calibration process. This further improves the accuracy of the reprojection error calculation and, in turn, the efficiency and accuracy of subsequent feature point extraction. For example, if the current camera is a short-focus / telephoto camera, its camera projection model is determined to be a pinhole model; if the current camera is a fisheye camera, its camera projection model is determined to be a Scaramuzza model.

[0079] Specifically, determining the reprojection error of the calibration plate according to the projection coordinates and detection coordinates of each first feature point on the calibration plate in the pixel coordinate system may include: determining the reprojection error of each first feature point on the calibration plate according to the projection coordinates and detection coordinates of each first feature point on the calibration plate in the pixel coordinate system, and calculating the average value of the reprojection errors of all first feature points of the calibration plate in the current frame image as the reprojection error of the calibration plate in the current frame image.

[0080] In one implementation, the reprojection error of each first feature point may be, but is not limited to, the distance between the detection coordinates and the projection coordinates of the first feature point in the pixel coordinate system. Figure 3 Taking any corner point of any mark in the calibration plate shown as an example, the three-dimensional coordinates of the marked corner point in the camera coordinate system can be obtained first according to the optimization value of the transformation relationship from the calibration plate coordinate system corresponding to the previous frame image or other previous frame images to the camera coordinate system and the calibration plate coordinate system coordinates of each first feature point. With the help of the camera projection model used by the current camera (for example, the pinhole model used by the short / telephoto camera, the Scaramuzza model used by the fisheye camera, etc.), the projection coordinates of the marked corner point in the pixel coordinate system are obtained from the camera coordinate system, and the projection coordinates are compared with the detected coordinates of the marked corner point in the pixel coordinate system obtained by performing feature point detection on the current frame image in step 205. The distance between the detected coordinates of the marked corner point in the pixel coordinate system and the projection coordinates is calculated as the reprojection error of the marked corner point.

[0081] The reprojection error of the calibration plate may be, but is not limited to, the average or weighted average of the reprojection errors of all first feature points on the calibration plate in the current frame image. It should be noted that the present disclosure does not limit the specific calculation method of the first feature point reprojection error and the calibration plate reprojection error.

[0082] As can be seen from the above, the reprojection error method can be used to eliminate observation image frames that do not meet the calibration requirements in advance, avoiding poor observation image frames from entering the subsequent optimization solution. The reprojection error is a robust verification mechanism that can further improve the accuracy of calibration.

[0083] Furthermore, step 205 may also include: pre-storing the calibration plate coordinates of the first feature point and the second feature point of each calibration plate; associating and storing the plate identifier of each calibration plate with the mark identifier of each mark in the calibration plate; and associating and storing the pixel coordinates of each first feature point with the corresponding plate identifier of the calibration plate according to the mark identifier of each mark for subsequent processing. Figure 3 Taking the calibration plate shown as an example, the marked corner points of each frame image are detected to obtain the pixel coordinate system coordinates of each marked corner point (that is, the detected coordinates of the pixel coordinate system in the previous text) and the identifier of each marker. According to the correspondence between the identifier of each marker and the pre-stored calibration plate identifier and the marker identifier, the detected coordinates of the marked corner point in the pixel coordinate system are associated with the plate identifier of the calibration plate to which the marker belongs and stored, so as to be used for the calculation of the camera coordinate system coordinates of the first feature point in subsequent frames of images and / or each calibration plate, and then the transformation relationship from the marker coordinate system to the camera coordinate system of each marker on the calibration plate is obtained.

[0084] In one implementation, the coordinates of the first feature point and the second feature point on the calibration plate (the coordinates include but are not limited to one or more of the coordinates of the calibration coordinate system, the calibration plate coordinate system, the marker coordinate system, the pixel coordinate system, the camera coordinate system, and the lidar coordinate system) can be stored in a pre-agreed fixed order so as to directly read one or more coordinates of the required feature point or points. At the same time, the point correspondence between different coordinate systems can be directly determined through sequential storage, so that the coordinates of the feature points can be directly applied to the construction process of problems such as ICP and PnP, without having to find the point correspondence between different coordinate systems, effectively reducing the amount of calculation and the computational complexity, improving processing efficiency and saving computing resources.

[0085] In one example, the coordinates of each first feature point in the marker (e.g., one or more of the coordinates of the calibration space coordinate system, the coordinates of the calibration plate coordinate system, the coordinates of the marker coordinate system, the coordinates of the pixel coordinate system, the coordinates of the camera coordinate system, and the coordinates of the lidar coordinate system) can be stored in association with the corresponding marker identifier and plate identifier according to the position order of each marker in the calibration plate. For each marker, the coordinates of each first feature point on the marker can be stored in association with the corresponding marker identifier, marker sequence number, and plate identifier according to the position order of the first feature point on the marker.

[0086] In one example, the coordinates of each second characteristic point of each calibration hole in the calibration plate (e.g., one or more of the coordinates of the calibration space coordinate system, the coordinates of the calibration plate coordinate system, the coordinates of the camera coordinate system, and the coordinates of the lidar coordinate system) can be associated and stored with the corresponding plate identifier in sequence according to the position sequence of each calibration hole in the calibration plate. Specifically, the coordinates of the second characteristic point of each calibration hole in the calibration plate and the sequence number of each calibration hole are associated and stored with the corresponding plate identifier in sequence, and the coordinates of the second characteristic points of the calibration hole contour are stored and associated with the corresponding plate identifier according to the position sequence of each second characteristic point on the calibration plate contour.

[0087] by Figure 3 For example, the four corner points of the marker are used as first feature points, and the four corner points of the calibration plate 300 and the centers of the circular calibration holes on the calibration plate 300 are used as second feature points. The coordinates of the first feature points and the second feature points can be stored in a clockwise order in association with the plate identifier, the marker identifier, etc. The two calibration plates can be stored in sequence or separately, that is, the first feature points are stored in association with the marker identifier, the plate identifier, and the marker sequence number, and the second feature points are also stored in association with the calibration hole identifier, the plate identifier, and the calibration hole sequence number.

[0088] For each mark on the calibration plate, the coordinates of the corner points of each mark can be stored in the clockwise order of "left, top, right, bottom". At the same time, for each mark, the coordinates of the four corner points on the mark are associated with the corresponding plate identifier, mark identifier, and mark sequence number in the clockwise order of "top left corner, top right corner, bottom right corner, bottom left corner".

[0089] For the multiple corner points on the calibration plate, the coordinates of the multiple corner points on the calibration plate, the sequence numbers of the multiple corner points, and the corresponding plate identifiers can also be stored in a clockwise order of "upper left corner, upper right corner, lower right corner, lower left corner". For the various calibration holes on the calibration plate, the coordinates of the center points of the calibration holes, the sequence numbers of the center points, and the corresponding plate identifiers can also be stored in a clockwise order of "left, top, right, bottom".

[0090] Assume that the current image frame contains n calibration plates (n is an integer greater than 1), and calibration plate 1 among the n calibration plates contains m marks (m is an integer greater than 1). The associated storage method can be: Figure 6 The exemplary manner shown, Figure 6 The curve with arrows in the middle represents the correlation line. Figure 6It can be seen that by associating and storing the coordinates of each corner point (for example, the pixel coordinate system coordinates of the corner point) in the order of the marker identifier, the calibration plate identifier, and the corner point position, it is convenient to query the coordinates of a first feature point of a certain marker, the coordinates of all first feature points of a certain marker, the coordinates of all first feature points on a certain calibration plate, and even the coordinates of all first feature points in a frame image and its calibration plate and marker information. It should be noted that Figure 6 This is just an example, and the storage method in a specific application is not limited to this.

[0091] In some possible implementations of the present disclosure, in step 205, based on the marker coordinate system coordinates and pixel coordinate system coordinates of each first feature point on the calibration plate, the transformation relationship from the marker coordinate system of each marker on the calibration plate to the camera coordinate system is obtained, which may include: for a certain marker in a certain frame image, the marker coordinate system coordinates and pixel coordinate system coordinates of the first feature point of the marker can be queried and read according to the marker identifier, a PnP problem can be constructed, and the transformation relationship from the marker coordinate system of the marker to the camera coordinate system is solved. In this embodiment, since the marker coordinate system coordinates and pixel coordinate system coordinates of the first feature point are stored in the order of the plate identifier, the marker identifier and the position of the first feature point, the marker coordinate system coordinates and the pixel coordinate system coordinates obtained by reading naturally satisfy the point correspondence relationship required by the PnP problem. Therefore, there is no need to perform the process of determining the point correspondence relationship, which simplifies the construction process of the PnP problem.

[0092] In some possible implementations of the present disclosure, the initial value of the transformation relationship from the calibration plate coordinate system to the camera coordinate system of the calibration plate in step 205 may be, but is not limited to, the average value of the transformation relationship from the marker coordinate system to the camera coordinate system of all markers in the calibration plate, the weighted average value of the transformation relationship from the marker coordinate system to the camera coordinate system of all markers in the calibration plate, the average value from the marker coordinate system to the camera coordinate system of some markers in the calibration plate whose reprojection error is lower than a set threshold, the weighted average value, etc.

[0093] For example, for multiple detected marks on the same calibration plate, the average value of each calculated transformation relationship is used as the initial value of the transformation relationship from the calibration plate coordinate system to the camera coordinate system. For example, for multiple detected marks on a certain calibration plate, the transformation relationships from the mark coordinate system to the camera coordinate system of these marks can be directly averaged to obtain the initial value of the transformation relationship from the calibration plate coordinate system to the camera coordinate system of the calibration plate. For another example, for a certain calibration plate, the initial value of the transformation relationship from the calibration plate coordinate system to the camera coordinate system of the calibration plate can be calculated by weighted averaging, where the weight of the transformation relationship with a smaller reprojection error is relatively larger, and the weight of the transformation error with a larger reprojection error is relatively smaller.

[0094] By calculating the initial value of the transformation relationship from the calibration plate coordinate system to the camera coordinate system based on the transformation relationship of all markers or some markers with smaller reprojection errors, a more accurate transformation relationship from the calibration plate coordinate system to the camera coordinate system can be obtained.

[0095] In some possible implementations of the present disclosure, in step 205, obtaining the optimized value of the transformation relationship from the calibration plate coordinate system corresponding to the current frame image to the camera coordinate system through optimization based on the initial value of the transformation relationship may include: constructing a PnP optimization problem, optimizing and solving the PnP optimization problem based on the initial value of the transformation relationship, and obtaining the optimized value of the transformation relationship from the calibration plate coordinate system corresponding to the current frame image to the camera coordinate system. Thus, by calculating each calibration plate in a certain frame image, the transformation relationship from the calibration plate coordinate system to the camera coordinate system of all calibration plates that meet the conditions in the frame image can be obtained, and for the calibration plate, all the first feature points on the calibration plate coordinate system are used to participate in solving the transformation relationship from the calibration plate coordinate system to the camera coordinate system, so that a more accurate transformation relationship from the calibration plate coordinate system to the camera coordinate system can be obtained. Similarly, the PnP optimization problem here does not need to execute the point correspondence determination process, and the computational complexity is low and the amount of calculation is small.

[0096] In step 205, for each calibration plate, the initial transformation relationship between the calibration plate coordinate system and the camera coordinate system is calculated using the transformation relationship between the marker coordinate system and the camera coordinate system. An optimization solution is then performed based on this initial transformation relationship between the calibration plate coordinate system and the camera coordinate system. This improves the robustness and accuracy of the calculation, further enhancing the accuracy of the calibration. Similarly, this optimization solution does not require the point correspondence determination process, resulting in low computational complexity and small amount of computation.

[0097] By performing the processing of step 205 on each frame of the at least one image obtained in step 201, the optimized transformation relationship value from the calibration plate coordinate system to the camera coordinate system corresponding to the multiple frames of images can be obtained. Furthermore, for the multiple frames of images, the optimized transformation relationship value from the calibration plate coordinate system to the camera coordinate system of each calibration plate and the plate identifier of the calibration plate can be stored in correspondence, and then matched based on the correspondence between the calibration plate identifier and the image frame number. This can achieve the associated storage of the image frame number, the calibration plate identifier, and the optimized transformation relationship value from the calibration plate coordinate system to the camera coordinate system of the calibration plate. In subsequent calibration calculations, the corresponding optimized transformation relationship value can be queried based on the image frame number and the calibration plate identifier.

[0098] In the present disclosure, the optimized value of the transformation relationship from the calibration plate coordinate system to the camera coordinate system can be used to extract the features of various feature points on the calibration plate (for example, determine the camera coordinate system coordinates of various feature points on the calibration plate), thereby realizing camera calibration through the features of various feature points on the calibration plate (for example, the camera coordinate system coordinates, pixel coordinate system coordinates, etc. of various feature points). The camera calibration may include but is not limited to the calibration of camera intrinsic parameters, the calibration of camera extrinsic parameters, and the joint calibration of the camera and other sensors such as lidar. Thus, at least one of the following parameters can be obtained: camera intrinsic parameters, camera extrinsic parameters, and the transformation relationship between the camera coordinate system and the coordinate system of other sensors (for example, the lidar coordinate system). The camera's data perception of the surrounding environment and the data fusion of the camera and other sensors can be realized through the camera intrinsic parameters, camera extrinsic parameters, and the transformation relationship between the camera coordinate system and the coordinate system of other sensors.

[0099] In one example, in scenarios such as autonomous driving, assisted driving, and mobile robots, the camera intrinsics and extrinsic parameters of multiple cameras, along with the environmental images captured by these cameras, can be used to estimate the position of obstacles within the camera's field of view, thereby enabling obstacle detection based on visual sensors. In another example, in applications such as simultaneous localization and mapping (SLAM), the camera intrinsics and extrinsic parameters of a full-view camera, along with the environmental images captured by these cameras, can be used to estimate high-definition maps and real-time positioning based on visual sensors. In another example, in a multi-sensor fusion scenario, the camera intrinsics and extrinsic parameters, as well as the transformation relationship between the camera coordinate system and the lidar coordinate system, can be used to achieve camera and lidar registration and data fusion, thereby obtaining more accurate and complete external environment information, and then achieving high-precision obstacle detection, path planning, and driving behavior decision-making, thereby improving driving safety under advanced driver assistance systems (ADAS).

[0100] It should be understood that the camera calibration of the embodiments of the present disclosure is not limited to the above three cases, and the application of calibration parameters is not limited to the above three examples.

[0101] Other specific implementations of the camera calibration method according to the embodiment of the present disclosure are described in detail below.

[0102] See also Figure 7 As shown, after obtaining the optimized transformation relationship values ​​for the multiple image frames in step 205, the camera calibration method of the present disclosure may further include: step 207, determining the camera coordinate system coordinates of the first and second feature points of the calibration plate. In this way, the feature points of the calibration plate can be extracted based on the optimized transformation relationship values ​​for the multiple image frames.

[0103] In one implementation, step 207 may include: calculating the first camera coordinate system coordinates of each first feature point on the calibration plate based on the optimized value of the transformation relationship from the calibration plate coordinate system to the camera coordinate system corresponding to the current frame image and the calibration plate coordinate system coordinates of each first feature point in the current frame image.

[0104] Furthermore, in step 207, multiple target images having a reprojection error of the calibration plate less than or equal to a second predetermined threshold value can be determined from at least one frame of image acquired in step 201, and the second camera coordinate system coordinates (i.e., the second coordinates in the camera coordinate system) of each first feature point on the calibration plate can be obtained based on the first camera coordinate system coordinates of each first feature point in the multiple target images. The multiple target images are determined by the method of "detecting the mirror feature points of the current frame image and then verifying the frame image using the reprojection error of the calibration plate" in step 205. Thus, a more accurate and robust calibration feature point feature can be obtained by integrating the calculation results of the multiple frames of image.

[0105] In one implementation, step 207 may further include: in response to the reprojection error being less than or equal to a second predetermined threshold, obtaining the camera coordinate system coordinates of the second feature point by using the transformation relationship optimization value and the attribute information of the calibration plate.

[0106] Specifically, based on the optimized value of the transformation relationship from the calibration plate coordinate system to the camera coordinate system corresponding to the current frame image and the calibration plate coordinate system coordinates of the second feature point of the calibration plate in the current frame image, the first camera coordinate system coordinates of the second feature point of the calibration plate are calculated based on the spatial transformation of the spatial point. Finally, the second camera coordinate system coordinates of the second feature point of the calibration plate can be obtained based on the first camera coordinate system coordinates of the second feature point of the calibration plate in multiple target images whose reprojection error is less than or equal to a second predetermined threshold. The calibration plate coordinate system coordinates of the second feature point of the calibration plate can be calculated and pre-stored based on the attribute information of the calibration plate. For details, please refer to the previous description and will not be repeated here. In this way, the features of each second feature point on the calibration plate with higher accuracy and better robustness can be obtained by integrating the calculation results of multiple frames of images.

[0107] In one example, for any first feature point or any second feature point on any calibration plate, an average of the first camera coordinate system coordinates of the corresponding feature point on the calibration plate in all qualified image frames (i.e., multiple target image frames with a reprojection error less than or equal to a second predetermined threshold) can be calculated, and the average value can be used as the second camera coordinate system coordinate of the feature point. Specifically, based on the first camera coordinate system coordinates of each second feature point in the multiple target image frames, obtaining the second camera coordinate system coordinates of each second feature point includes: calculating the average of the first camera coordinate system coordinates of the predetermined feature point on the calibration plate corresponding to each image frame, and using the average value as the second camera coordinate system coordinate of the predetermined feature point on the calibration plate. The predetermined feature points include, but are not limited to, the first feature point and the second feature point on the calibration plate.

[0108] It should be noted that the coordinates of the second camera coordinate system may also be obtained by other calculation methods such as weighted averaging. The present disclosure does not limit the specific calculation method of the coordinates of the second camera coordinate system.

[0109] It can be seen from the above that the optimized value of the transformation relationship from the calibration plate coordinate system to the camera coordinate system obtained by the embodiment of the present disclosure can not only extract the features of various feature points such as the first feature point and the second feature point on the calibration plate, but also can comprehensively calculate the results of multiple frames of images to obtain the camera coordinate system coordinates of various feature points on the calibration plate with higher accuracy and better robustness. Therefore, while realizing the comprehensive extraction of feature point features, the accuracy and robustness of the feature points are further improved.

[0110] Furthermore, in step 207, after obtaining the camera coordinate system coordinates of various feature points on the calibration plate (e.g., the second camera coordinate system coordinates), the camera coordinate system coordinates of the various feature points on the calibration plate can also be associated with the plate identification, marking identification, and other information of the calibration plate and stored for use in subsequent calibration processes. Thus, the disclosed embodiment can extract all features within the camera's field of view at once, and these features have identification information.

[0111] In summary, the embodiments of the present disclosure can obtain the feature point features required for camera calibration. These feature point features may include: the camera coordinate system coordinates of each second feature point on the calibration plate (for example, the center point of the calibration hole on the calibration plate, the edge corner points of the calibration plate, the high reflection point, etc.), the pixel coordinate system coordinates and the camera coordinate system coordinates of each first feature point on the calibration plate, the coordinates of the first feature point are associated with the calibration plate identifier and the marking identifier, and the coordinates of each second feature point on the calibration plate are associated with the calibration plate identifier. These feature point features can not only realize the calibration of the camera's internal and external parameters, but also realize the joint calibration of the camera and other sensors (for example, lidar, millimeter wave radar, etc.).

[0112] by Figure 3Taking the calibration plate as an example, its feature point features may include: the camera coordinate system coordinates and pixel coordinate system coordinates of the corner points on the four marks on the calibration plate, the camera coordinate system coordinates of the four corner points on the calibration plate, and the camera coordinate system coordinates of the centers of the four circular holes on the calibration plate. The coordinate information of the corner points of the marks is associated with the mark identifier of the marks and the board identifier of the marking plate. The coordinate information of the corner points on the calibration plate is associated with the board identifier of the calibration plate, and the coordinates of the center of the circular holes on the calibration plate are associated with the board identifier of the calibration plate.

[0113] See also Figure 7 As shown, after step 207, the camera calibration method may further include: step 209, performing camera calibration based on the camera coordinate system coordinates of the first feature point and / or the second feature point on the calibration plate. Preferably, the camera coordinate system coordinates of the first feature point and the second feature point on the calibration plate involved in the camera calibration may be the second camera coordinate system coordinates obtained in step 207, that is, the camera coordinate system coordinates obtained by integrating the calculation results of multiple frames of target images.

[0114] In some possible implementations, camera calibration can be performed in a calibration room. The calibration room can include multiple types of calibration plates, and these calibration plates can have different positions, heights, angles, etc. The camera calibration is performed based on the camera coordinate system coordinates of the first feature points and / or second feature points on each calibration plate in the calibration room. This can better meet the spatial distribution requirements of the feature points required for calibration, avoid the need to move the position and angle of the calibration plate during the calibration process, and further improve calibration efficiency.

[0115] For example, multiple types of calibration plates can be deployed in the calibration room, and the positions, heights, and angles of these calibration plates relative to the camera in the calibration room are different. Some calibration plates contain marks, some calibration plates contain marks and circular holes, and some calibration plates contain circular holes.

[0116] In one implementation of the present disclosure, pairwise and joint calibration of cameras and other target sensors (e.g., lidar, millimeter-wave radar, etc.) can be performed based on the camera coordinate system coordinates of the second feature point on the calibration plate (e.g., the center point of the calibration hole, the corner point of the edge of the calibration plate, etc.).

[0117] Taking the joint calibration of cameras and lidar as an example, camera calibration based on the camera coordinate system coordinates of the first feature point and / or the second feature point on the calibration plate may include: using the point cloud data about the calibration plate collected by the lidar to obtain the lidar coordinate system coordinates of the second feature point on the calibration plate; matching the camera coordinate system coordinates of each second feature point on each calibration plate with the corresponding lidar coordinate system coordinates through the calibration plate identifier and the position of the second feature point, constructing an ICP problem, and optimizing the solution to obtain the transformation relationship between the camera coordinate system and the lidar coordinate system. In this way, the second feature point features of the calibration plate can be used to realize the joint calibration of cameras and other sensors (for example, lidar, millimeter wave radar, etc.). In addition to meeting the calibration production line requirements, it also meets the calibration requirements of specific scenarios, such as only requiring pairwise calibration.

[0118] In this embodiment, since the camera coordinate system coordinates and the lidar coordinate system coordinates of the second feature point can be stored in the order of the plate identification and the position of the second feature point, the read camera coordinate system coordinates and the lidar coordinate system coordinates naturally meet the point correspondence required by the ICP problem. Therefore, there is no need to perform the point correspondence determination process, which simplifies the construction process of the ICP problem, reduces the computational complexity, reduces the amount of calculation, saves computing resources, and improves processing efficiency.

[0119] In one example, when solving the transformation relationship from the camera coordinate system to the lidar coordinate system, the lidar coordinates of the second feature point on the calibration plate are first obtained using the point cloud data collected by the lidar about the calibration plate. The lidar coordinates of the second feature point on the calibration plate are stored in order of the second feature point's position and associated with the plate identifier of the calibration plate. Next, the camera coordinates and lidar coordinates of all the second feature points of each calibration plate are sequentially read. The ICP problem is directly constructed using the camera coordinates of all the second feature points of each calibration plate as the source point cloud and the lidar coordinates of all the second feature points of each calibration plate as the target point cloud. This solves the rotation transformation relationship and translation transformation relationship from the camera coordinate system to the lidar coordinate system. The rotation transformation relationship and translation transformation relationship from the camera coordinate system to the lidar coordinate system are the transformation relationship from the camera coordinate system to the lidar coordinate system. The transformation relationship includes a transformation matrix, the rotation transformation relationship includes a rotation matrix, and the translation transformation relationship includes a translation matrix.

[0120] In an example, when solving the transformation relationship from the lidar coordinate system to the camera coordinate system, the point cloud data of the calibration plate collected by the lidar is first used to obtain the lidar coordinate system coordinates of the second feature point on the calibration plate, and the lidar coordinate system coordinates of the second feature point on the calibration plate are stored in the order of the position of the second feature point and associated with the plate identifier of the calibration plate; then, the camera coordinate system coordinates and lidar coordinate system coordinates of all the second feature points of each calibration plate are read sequentially, and the ICP problem is directly constructed with the lidar coordinate system coordinates of all the second feature points of each calibration plate as the source point cloud and the camera coordinate system coordinates of all the second feature points of each calibration plate as the target point cloud, so as to solve the rotation transformation relationship and translation transformation relationship from the lidar coordinate system to the camera coordinate system. The rotation transformation relationship and translation transformation relationship from the lidar coordinate system to the camera coordinate system are the transformation relationship from the lidar coordinate system to the camera coordinate system.

[0121] Figure 8 A flowchart of parameter calibration for a camera and a lidar according to an embodiment of the present disclosure is shown.

[0122] like Figure 8 As shown, in step 801, images and point cloud data of a calibration plate collected by a camera and a laser radar are respectively obtained, where the calibration plate is provided with a mark, and the calibration plate includes a first feature point of the mark and a second feature point of a structural contour of the calibration plate;

[0123] In step 803, the pixel coordinate system coordinates of the first feature point are determined based on the image;

[0124] In step 805, the laser radar coordinate system coordinates of the second feature point are determined based on the point cloud data;

[0125] In step 807, a first transformation relationship between the calibration plate coordinate system and the laser radar coordinate system is determined based on the calibration plate coordinate system coordinates and the laser radar coordinate system coordinates of the second feature point;

[0126] In step 809, the laser radar coordinate system coordinates of the first feature point are determined according to the calibration plate coordinate system coordinates of the first feature point and the first transformation relationship; and

[0127] In step 811 , the camera and the lidar are calibrated according to the pixel coordinate system coordinates and the lidar coordinate system coordinates of the first feature point.

[0128] In one example, step 805 includes:

[0129] Step 805A: Identify point cloud points corresponding to the same laser beam from the point cloud data. For example, the point cloud points corresponding to the same laser beam may be determined based on the laser beam identifier of each point cloud point in the point cloud data; or the angle of the corresponding laser beam may be determined based on the coordinates of each point cloud point in the point cloud data, and points whose angle difference between the corresponding laser beams is less than a predetermined threshold may be determined as point cloud points corresponding to the same laser beam.

[0130] Step 805B: identifying edge feature points from the point cloud points corresponding to the same laser beam;

[0131] Step 805C: Determine a second feature point based on the edge feature points. For example, perform line fitting on the edge feature points to obtain at least two lines; determine the intersection of the at least two lines to obtain a first reference candidate point for the outer contour of the calibration plate; and determine the first reference point from the first reference candidate points based on the attribute information of the calibration plate. For another example, perform shape fitting on the edge feature points to determine a second reference candidate point for the calibration hole; and select the second reference point from the second reference candidate points based on the attribute information of the calibration plate and the calibration hole.

[0132] In one implementation, the parameter calibration of step 811 can be performed by Figure 9 The process shown is implemented as follows, see Figure 9 , the process includes the following steps:

[0133] Step 901: Construct a Point-N-Pick (PnP) problem based on the LiDAR coordinates and the pixel coordinates of the first feature point. Optimize and solve the PnP problem to obtain the initial values ​​of the camera's intrinsic and extrinsic parameters. A corresponding camera intrinsic parameter model and the PnP cost function can also be determined based on the camera type. Fisheye cameras have a corresponding fisheye camera model, and perspective cameras have a corresponding perspective camera model.

[0134] Step 903: Calibrate the camera and lidar based on the initial values ​​of the camera's internal parameters and external parameters.

[0135] Here, if m frames of image and m frames of point cloud data are obtained in step 801, where m is an integer greater than 1, m sets of pixel coordinate system coordinates of the first feature point can be obtained in step 803, and m sets of lidar coordinate system coordinates of the first feature point can be obtained in step 809. At this time, the theoretical value of the intrinsic parameter and the manually measured value of the extrinsic parameter can be used as the initial value, and a PnP problem can be constructed based on the m sets of pixel coordinate system coordinates and the m sets of lidar coordinate system coordinates of the first feature point, respectively. By optimizing and solving the PnP problem, the m first intrinsic parameters and m first extrinsic parameters of the camera are obtained. The second intrinsic parameter of the camera is then determined based on the m first intrinsic parameters (such as averaging or weighted averaging the m first intrinsic parameters), and the second intrinsic parameter is used as the initial value of the camera intrinsic parameter, while the m first extrinsic parameters are used as the initial value of the camera extrinsic parameter. Of course, the initial value of the camera intrinsic parameter can be the intrinsic parameter value calibrated by the camera's factory parameters. The disclosed embodiment does not limit the source of the initial value of the camera intrinsic parameter.

[0136] In one example, the exemplary implementation process of calibrating the camera and the lidar based on the initial values ​​of the camera's internal parameters and external parameters in step 903 is shown in Figure 10 , which may include the following steps:

[0137] Step 1001: construct a PnP problem based on the coordinates of the calibration plate coordinate system and the pixel coordinate system of the first feature point, optimize and solve the PnP problem based on the initial value of the camera's intrinsic parameter, and obtain a second transformation relationship between the calibration plate coordinate system and the camera coordinate system;

[0138] Step 1003, determining the camera coordinate system coordinates of the second feature point according to the calibration plate coordinate system coordinates of the second feature point and the second transformation relationship;

[0139] Step 1005: construct an ICP problem based on the camera coordinate system coordinates and the lidar coordinate system coordinates of the second feature point, optimize and solve the ICP problem based on the initial values ​​of the camera's extrinsic parameters, and obtain a third transformation relationship between the camera coordinate system and the lidar coordinate system.

[0140] Additionally, step 903 may further include constructing a point-n-play (PnP) problem based on the coordinates of the calibration plate and the pixel coordinates of the first feature point, and optimizing the PnP problem based on the initial values ​​of the camera's intrinsic parameters to obtain optimized values ​​of the camera's intrinsic parameters. In practical applications, whether to perform the camera intrinsic parameter optimization step can be determined based on whether the camera intrinsic parameters need to be recalibrated.

[0141] In one implementation, the calibration of step 811 can also be performed by Figure 10 The process shown is implemented.

[0142] See also Figure 10 As shown, the process may include the following steps:

[0143] Step 1001: Obtain a second transformation relationship between the calibration plate coordinate system and the camera coordinate system according to the calibration plate coordinate system coordinates and the pixel coordinate system coordinates of the first feature point;

[0144] Step 1003: Obtain the camera coordinate system coordinates of the second feature point according to the calibration plate coordinate system coordinates and the second transformation relationship; and

[0145] Step 1005: Obtain a third transformation relationship between the camera coordinate system and the lidar coordinate system according to the camera coordinate system coordinates and the lidar coordinate system coordinates of the second feature point.

[0146] In one implementation, the transformation relationship between the inter-calibration coordinate system and the camera coordinate system can be determined based on the camera coordinate system coordinates of the second feature points (e.g., the center point of the calibration hole, the corner point of the edge of the calibration plate, etc.) on each calibration plate in the calibration room and the inter-calibration coordinate system coordinates. Specifically, performing camera calibration based on the camera coordinate system coordinates of the first feature point and / or the second feature point on the calibration plate may include: storing the inter-calibration coordinate system coordinates of the second feature points on each calibration plate in the order of the positions of the second feature points and associating them with the plate identifier of the calibration plate; then, sequentially reading the calibration coordinate system coordinates and the camera coordinate system coordinates of all the second feature points of each calibration plate, so that the camera coordinate system coordinates of the second feature points on each calibration plate can be directly corresponded to the inter-calibration coordinate system coordinates of the second feature points on each calibration plate through the calibration plate identifier and the position of the second feature point, thereby constructing an ICP problem, and optimizing the solution to obtain the initial value of the transformation relationship between the inter-calibration coordinate system and the camera coordinate system. In this way, the calibration of the camera and the inter-calibration coordinate system can be achieved, so that the external parameters of the camera and other target sensors can be obtained through the transfer of the inter-calibration coordinate system.

[0147] In this embodiment, the camera coordinate system coordinates and the calibration coordinate system coordinates of the second feature point are stored in the order of the position of the second feature point and associated with the plate identification, so that the read camera coordinate system coordinates and the calibration coordinate system coordinates naturally meet the point correspondence required by the ICP problem. Therefore, there is no need to perform the point correspondence determination process, which simplifies the construction process of the ICP problem, reduces the computational complexity, reduces the amount of calculation, saves computing resources, and improves processing efficiency.

[0148] In one example, a total station or other similar device can be used to measure and obtain the inter-calibration coordinate system coordinates of the second feature point on each calibration plate. The inter-calibration coordinate system coordinates of the second feature point on each calibration plate are associated with the plate identifier of the calibration plate and stored. The method for obtaining the inter-calibration coordinate system coordinates of the second feature point on each calibration plate is not limited to this method, and any other applicable method can be applied in the present disclosure.

[0149] In an example, the inter-calibration coordinate system coordinates of all the second feature points of each calibration plate can be used as the source point cloud, and the camera coordinate system coordinates of all the second feature points of each calibration plate can be used as the target point cloud to construct an ICP problem, so as to obtain the first rotation transformation relationship and the first translation transformation relationship from the inter-calibration coordinate system to the camera coordinate system, and use the first rotation transformation relationship and the second translation transformation relationship as the initial values ​​of the transformation relationship from the inter-calibration coordinate system to the camera coordinate system.

[0150] Furthermore, performing camera calibration based on the camera coordinate system coordinates of the first feature point and / or the second feature point on the calibration plate may also include: based on the initial value of the transformation relationship from the inter-calibration coordinate system to the camera coordinate system, the pixel coordinate system coordinates of the first feature point on each calibration plate are matched with its inter-calibration coordinate system coordinates through the marking identifier and the position of the first feature point, a PnP problem is constructed, and the optimized solution is obtained to obtain the optimized value of the transformation relationship from the inter-calibration coordinate system to the camera coordinate system.

[0151] In one example, a total station or other similar device can be used to measure and obtain the inter-calibration coordinate system coordinates of the first feature point on each calibration plate, and the inter-calibration coordinate system coordinates of the first feature point on each calibration plate are associated with the corresponding mark identifier and the calibration plate identifier and stored. It is understood that the method for obtaining the inter-calibration coordinate system coordinates of the first feature point on each calibration plate is not limited to this, and any other applicable method can be applied to the present disclosure.

[0152] In an example, the inter-calibration coordinate system coordinates and pixel coordinate system coordinates of all the first feature points of each calibration plate can be read, and the PnP problem can be directly constructed based on the initial value of the transformation relationship from the inter-calibration coordinate system to the camera coordinate system (i.e., the first rotation transformation relationship and the first translation transformation relationship mentioned above), so as to obtain the second rotation transformation relationship and the second translation transformation relationship from the inter-calibration coordinate system to the camera coordinate system, and use the second rotation transformation relationship and the second translation transformation relationship as the optimized value of the transformation relationship from the inter-calibration coordinate system to the camera coordinate system.

[0153] As can be seen from the above, the present disclosure adopts a coarse-to-fine approach, that is, the ICP problem is first constructed using the calculated three-dimensional coordinates of the second feature point of the calibration plate in the camera coordinate system and its three-dimensional coordinates in the calibration interval coordinate system, and the optimization result is used as the initial value. Then, on the basis of the initial value, the pixel coordinate system coordinates of the first feature point on the calibration plate and the coordinates of the calibration interval coordinate system are used to construct the PnP problem to solve the camera extrinsic parameters. There is no need to manually give the initial value, thereby achieving a more robust and accurate camera calibration.

[0154] In summary, the embodiments of the present disclosure can not only realize the external parameter calibration between the camera and the calibration, but also arrange the specific calibration plate of the target sensor according to other calibration requirements, such as lidar and millimeter-wave radar, solve the three-dimensional coordinates of the specific calibration plate in the camera coordinate system, and perform pairwise calibration and joint calibration of the camera and target sensor, which is highly universal.

[0155] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0156] The camera calibration method of the disclosed embodiment has the following beneficial effects: 1. It can efficiently, accurately and comprehensively extract the features of the feature points required for calibration. In addition to being directly used for external parameter calibration between the camera and the calibration, these features can also be used according to other calibration requirements. 2. It is suitable for camera calibration of various types of cameras such as short-focus cameras, telephoto cameras, fisheye cameras, etc., which can realize external parameter calibration between the camera and the calibration, and can also perform pairwise calibration and joint calibration of cameras and target sensors (for example, lidar and millimeter-wave radar), with high versatility. 3. It can meet the requirements of autonomous driving vehicle production line calibration and calibration between calibrations for robust, accurate and efficient internal and external parameter calibration. 4. It can meet the requirements of high efficiency, accuracy and robustness for feature point extraction and calibration required for camera calibration in production line calibration. After actual testing, it has more advantages than other existing calibration methods.

[0157] According to an embodiment of another aspect, a calibration device is provided. Figure 11 FIG. 1 is a schematic block diagram of a calibration device in the form of a computer program according to an embodiment. Figure 11 As shown, the apparatus 1100 includes: an image acquisition unit 1101, a marker coordinate system coordinate determination unit 1102, and an image frame processing unit 1103. It may further include a camera coordinate system coordinate determination unit 1104 and a calibration execution unit 115. The main functions of each component unit are as follows:

[0158] The image acquisition unit 101 is configured to acquire at least one frame of image of a calibration plate captured by a camera, wherein the calibration plate is provided with a plurality of marks, and the calibration plate includes a first feature point of each mark;

[0159] The marker coordinate system coordinate determining unit 1102 is configured to obtain the marker coordinate system coordinates of each first feature point according to the attribute information of each marker on the calibration plate;

[0160] The image frame processing unit 103 is configured to perform the following processing frame by frame for the at least one frame of image: determining the pixel coordinate system coordinates of the first feature point from the current frame image; obtaining the transformation relationship from the marker coordinate system of each marker on the calibration plate to the camera coordinate system according to the marker coordinate system coordinates and the pixel coordinate system coordinates of each first feature point on the calibration plate; obtaining the initial value of the transformation relationship from the calibration plate coordinate system to the camera coordinate system of the calibration plate according to the transformation relationship from the marker coordinate system of each marker on the calibration plate to the camera coordinate system; and, based on the initial value of the transformation relationship, obtaining the optimized value of the transformation relationship from the calibration plate coordinate system to the camera coordinate system corresponding to the current frame image through optimization solution.

[0161] Furthermore, the camera coordinate system coordinate determining unit 1104 is configured to determine the camera coordinate system coordinates of the first feature point and the second feature point of the calibration plate.

[0162] Furthermore, the calibration execution unit 1105 is configured to perform camera calibration based on the camera coordinate system coordinates of the first feature point and / or the second feature point on the calibration plate.

[0163] The specific details of the functions of the above-mentioned components can be found in the previous method section and will not be repeated here.

[0164] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple. For relevant parts, refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0165] In addition, an embodiment of the present disclosure further provides a computer-readable storage medium on which a computer program is stored. The program includes instructions, and when the instructions are executed by one or more processors of a computing device, the steps of any one of the method embodiments described above are executed.

[0166] The calibration device provided in the embodiment of the present disclosure may be implemented in the form of an electronic device. In this case, Figure 12As shown in , the calibration device includes one or more processors 1201, and also includes a memory 1202 for storing one or more programs, which are executed by the one or more processors 1201 to implement the method flow shown in the above embodiments of the present disclosure and / or the program units corresponding to each unit in the device. The various components are interconnected using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor 1201 can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of the user interface on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories.

[0167] The processor 1201 may include one or more single-core processors or multi-core processors. The processor 901 may include any combination of general-purpose processors or dedicated processors (such as image processors, application processors, baseband processors, etc.). The memory 1202 is a computer-readable storage medium provided by the present disclosure, which can be used to store non-transient software programs, non-transient computer executable programs and units, such as the embodiments of the present disclosure. Figure 2 、 Figure 8-10 The processor 1201 executes the program, instructions and units corresponding to the calibration method shown in the above method embodiment by running the non-transient software program, instructions and units stored in the memory 1202.

[0168] The calibration device may further include: an input device 1203 and an output device 1204. The processor 1201, the memory 1202, the input device 1203 and the output device 1204 may be connected via a bus or other means. Figure 9 The example of the connection via bus is taken. The input device 1203 can receive input digital or character information, and generate signal input related to the user settings and function control of the calibration device, such as a touch screen, a keypad, a mouse, a trackpad, a touchpad, an indicator stick, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 1204 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0169] The above-mentioned program (also referred to as software, software application, or code) includes machine instructions for a programmable processor, and these computing programs can be implemented using object-oriented programming languages, assembly or machine languages. As time goes by and technology develops, the meaning of media becomes more and more extensive, and the dissemination path of computer programs is no longer limited to tangible media, but can also be downloaded directly from the Internet. Any combination of one or more computer-readable storage media can be used. Computer-readable storage media can be used, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or components, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by an instruction execution system, device or device or used in combination with it.

[0170] A1: A camera calibration method, the method comprising: obtaining at least one frame of image captured by a camera about a calibration plate, wherein a plurality of marks are provided on the calibration plate, and the calibration plate includes a first feature point about each mark; obtaining the mark coordinate system coordinates of each first feature point according to the attribute information of each mark on the calibration plate; performing the following processing on the at least one frame of image: determining the pixel coordinate system coordinates of the first feature point from the current frame image; obtaining the transformation relationship from the mark coordinate system to the camera coordinate system of each mark on the calibration plate according to the mark coordinate system coordinates and the pixel coordinate system coordinates of each first feature point on the calibration plate; obtaining the initial value of the transformation relationship from the calibration plate coordinate system to the camera coordinate system of the calibration plate according to the transformation relationship from the mark coordinate system to the camera coordinate system of each mark on the calibration plate; and obtaining the optimized value of the transformation relationship from the calibration plate coordinate system to the camera coordinate system corresponding to the current frame image through optimization solution based on the initial value of the transformation relationship. A6. The method as described in A1, wherein the pixel coordinate system coordinates are detection coordinates in the pixel coordinate system, and the method further comprises: obtaining the projection coordinates of each first feature point on the calibration plate in the pixel coordinate system based on the optimized transformation relationship value and the attribute information of each marker on the calibration plate; and determining the reprojection error of the calibration plate based on the detection coordinates and projection coordinates of each first feature point on the calibration plate in the pixel coordinate system. A8. The method as described in A6, wherein the calibration plate further comprises second feature points related to the structural contour of the calibration plate, and the method further comprises: in response to the reprojection error being less than or equal to a second predetermined threshold, obtaining the camera coordinate system coordinates of the second feature point using the optimized transformation relationship value and the attribute information of the calibration plate. A9. The method as described in A8, wherein the camera coordinate system coordinates of the second feature point are the first camera coordinate system coordinates of the second feature point, and the method further comprises: determining, from at least one frame of image, multiple frames of target images in which the reprojection error of the calibration plate is less than or equal to the second predetermined threshold; and obtaining the second camera coordinate system coordinates of each second feature point based on the first camera coordinate system coordinates of each second feature point in the multiple frames of target image.

[0171] A10. The method described in A8, wherein the mark is a QR code, the first feature points include the corner points of the mark, and the second feature points include at least one of the following: the corner points of the outline of the calibration plate, the corner points of the calibration hole in the calibration plate, and the center point of the calibration hole. A11. The method described in A6, wherein the projection coordinates of each first feature point in the pixel coordinate system are obtained based on the transformation relationship optimization value and the attribute information of each mark on the calibration plate, including: determining a camera projection model based on the camera type information; calculating the first camera coordinate system coordinates of each first feature point on the calibration plate based on the transformation relationship optimization value and the calibration plate coordinate system coordinates of each first feature point; and calculating the projection coordinates of each first feature point on the calibration plate in the pixel coordinate system based on the camera projection model and the first camera coordinate system coordinates of each first feature point on the calibration plate. A12. The method as described in A6, wherein determining the reprojection error of the calibration plate based on the projection coordinates and detection coordinates of each first feature point on the calibration plate in the pixel coordinate system includes: determining the reprojection error of each first feature point on the calibration plate based on the projection coordinates and detection coordinates of each first feature point on the calibration plate in the pixel coordinate system; and calculating the average of the reprojection errors of all first feature points of the calibration plate in the current frame image as the reprojection error of the calibration plate in the current frame image. A13. The method as described in A9, wherein obtaining the second camera coordinate system coordinates of each second feature point in multiple frames of target images based on the first camera coordinate system coordinates of each second feature point includes: calculating the average of the first camera coordinate system coordinates of the second feature point on the calibration plate corresponding to each frame image, and using the average as the second camera coordinate system coordinate of the second feature point on the calibration plate.

[0172] A14. A method as described in A1, wherein the calibration plate further includes a second feature point about the structural contour of the calibration plate, and the method further includes: obtaining the camera coordinate system coordinates of the first feature point and the second feature point based on the optimized value of the transformation relationship and the pre-stored calibration plate coordinate system coordinates of the first feature point and the second feature point; and calibrating the camera parameters based on the camera coordinate system coordinates of at least one of the first feature point and the second feature point. A15. A method as described in A14, wherein calibrating the camera parameters based on the camera coordinate system coordinates of at least one of the first feature point and the second feature point includes: obtaining the laser radar coordinate system coordinates of each second feature point on the calibration plate using point cloud data about the calibration plate collected by the laser radar; corresponding the camera coordinate system coordinates of each second feature point on each calibration plate to the corresponding laser radar coordinate system coordinates through the calibration plate identifier, constructing an ICP problem, and optimizing the solution to obtain the transformation relationship between the camera coordinate system and the laser radar coordinate system.

[0173] A16. The method as described in A14, wherein the calibration plate is located in a calibration room, the calibration room includes multiple calibration plates, each calibration plate has a plate identifier, and calibrating camera parameters based on the camera coordinate system coordinates of at least one of the first feature point and the second feature point includes: obtaining the inter-calibration coordinate system coordinates of the second feature point on each calibration plate; corresponding the camera coordinate system coordinates of the second feature point on each calibration plate with the inter-calibration coordinate system coordinates of the predetermined feature point on each calibration plate through the plate identifier, constructing an ICP problem, and optimizing the solution to obtain an initial value of the transformation relationship from the inter-calibration coordinate system to the camera coordinate system. A17. The method as described in A16, wherein each mark on each calibration plate has a mark identifier, further includes: based on the initial value of the transformation relationship from the inter-calibration coordinate system to the camera coordinate system, corresponding the pixel coordinate system coordinates of the first feature point on each calibration plate with the inter-calibration coordinate system coordinates of the first feature point through the mark identifier, constructing a PnP problem, and optimizing the solution to obtain an optimized value of the transformation relationship from the inter-calibration coordinate system to the camera coordinate system.

[0174] The technical solutions provided by the present disclosure are introduced in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present disclosure. The description of the above embodiments is only used to help understand the method and core ideas of the present disclosure. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scopes based on the ideas of the present disclosure. In summary, the content of this specification should not be understood as a limitation on the present disclosure. The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, etc. made within the spirit and principles of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A camera calibration method, characterized in that: The method comprises: Acquire at least one frame of image of a calibration plate captured by a camera, wherein the calibration plate is provided with a plurality of marks, and the calibration plate includes a first feature point of each mark; Obtaining the marker coordinate system coordinates of each first feature point according to the attribute information of each marker on the calibration plate; The following processing is performed on the at least one frame of image: Determine the pixel coordinate system coordinates of the first feature point from the current frame image; Obtaining a transformation relationship from the marker coordinate system of each marker on the calibration plate to the camera coordinate system according to the marker coordinate system coordinates and the pixel coordinate system coordinates of each first feature point on the calibration plate; Obtaining an initial value of the transformation relationship between the calibration plate coordinate system and the camera coordinate system of the calibration plate according to the transformation relationship between the mark coordinate system of each mark on the calibration plate and the camera coordinate system; and Based on the initial value of the transformation relationship, the optimized value of the transformation relationship from the calibration plate coordinate system to the camera coordinate system corresponding to the current frame image is obtained through optimization solution.

2. The method according to claim 1, wherein: The acquiring of at least one frame of image of the calibration plate captured by the camera includes: Get the image sequence captured by the camera; Image detection is performed on each frame of the image sequence to determine at least one frame of the image that contains the calibration plate and has a number of marks on the calibration plate that is greater than or equal to a first predetermined threshold.

3. The method according to claim 1, wherein The acquiring of at least one frame of image of the calibration plate captured by the camera includes: At least one frame of image of a plurality of calibration plates captured by the camera is respectively acquired, wherein different calibration plates are provided with different marks, each calibration plate has a corresponding plate identifier, and each mark has a corresponding mark identifier.

4. The method of claim 3, further comprising: Pre-storing the coordinates of the calibration plate coordinate system of the first feature point of each calibration plate; The plate identification of each calibration plate is associated with the mark identification of each mark in the calibration plate and stored; as well as According to the mark identifier of each mark, the pixel coordinate system coordinates of each first feature point and the plate identifier of the corresponding calibration plate are associated and stored.

5. The method according to claim 1, wherein The initial value of the transformation relationship from the calibration plate coordinate system to the camera coordinate system of the calibration plate is the average value of the transformation relationship from the marker coordinate system to the camera coordinate system of all markers in the calibration plate; The method of obtaining the optimized value of the transformation relationship from the calibration plate coordinate system corresponding to the current frame image to the camera coordinate system through optimization and solution based on the initial value of the transformation relationship includes: constructing a PnP optimization problem, and optimizing and solving the PnP optimization problem based on the initial value of the transformation relationship to obtain the optimized value of the transformation relationship from the calibration plate coordinate system corresponding to the current frame image to the camera coordinate system.

6. The method of claim 1, wherein: The pixel coordinate system coordinates are detection coordinates in the pixel coordinate system, and the method further includes: Obtaining the projection coordinates of each first feature point on the calibration plate in a pixel coordinate system according to the optimized value of the transformation relationship and the attribute information of each mark on the calibration plate; The reprojection error of the calibration plate is determined according to the detection coordinates and the projection coordinates of each first feature point on the calibration plate in the pixel coordinate system.

7. The method according to claim 6, wherein: The at least one frame of image is a plurality of frames of image, and the method further includes: Determine, from the at least one frame of image, a plurality of frames of target images whose reprojection error of the calibration plate is less than or equal to a second predetermined threshold; The second camera coordinate system coordinates of each first feature point on the calibration plate are obtained according to the first camera coordinate system coordinates of each first feature point in the multi-frame target image.

8. The method of claim 6, wherein: The calibration plate further includes a second feature point about a structural contour of the calibration plate, and the method further includes: In response to the reprojection error being less than or equal to a second predetermined threshold, the camera coordinate system coordinates of the second feature point are obtained by using the transformation relationship optimization value and the attribute information of the calibration plate.

9. A calibration device, characterized in that: The device comprises: one or more processors, and A memory storing a program, the program comprising instructions which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a program, the program comprising instructions which, when executed by one or more processors of a computing device, cause the computing device to perform the method according to any one of claims 1 to 8.

Images