Joint calibration method and device, electronic equipment and computer readable storage medium

Abstract

The application provides a joint calibration method and device, electronic equipment and computer readable storage medium. The method obtains a checkerboard image of a calibration board through a shooting device, and obtains a skeleton point cloud image corresponding to the checkerboard image through a laser radar, wherein the calibration board has arrayed checkerboards, each checkerboard includes a central region and a boundary region surrounding the outer periphery of the central region, the central region and the boundary region have different reflectivities, a plurality of initial corner points and spatial coordinate values of each initial corner point are determined according to point cloud brightness of each point cloud data in the skeleton point cloud image, wherein the point cloud brightness has a positive correlation with the reflectivity, pixel coordinate values corresponding to each initial corner point are obtained from the checkerboard image, and a transformation matrix between a camera coordinate system corresponding to the shooting device and a laser radar coordinate system corresponding to the laser radar is calculated according to the spatial coordinate values of each initial corner point and the pixel coordinate values corresponding to the initial corner points, thereby improving the calibration effect.

Description

Technical Field

[0001] This invention relates to the field of sensor calibration technology, and in particular to a combined calibration method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] When an electronic device is equipped with a camera and a lidar, in order for the camera and lidar to work together, it is necessary to perform joint calibration on the camera and lidar to determine the transformation matrix between the coordinate systems of the camera and the lidar.

[0003] One common joint calibration method is to use a checkerboard calibration board. However, since the checkerboard calibration board is a flat plane without prominent shape features, it's difficult for the LiDAR to find feature points within the point cloud data after acquiring it. Users must manually select feature points from the checkerboard's point cloud data, which is not only cumbersome but also prone to significant selection errors, resulting in a low accuracy of the calibrated transformation matrix. Summary of the Invention

[0004] In view of the above, it is necessary to provide a joint calibration method, apparatus, electronic device and computer-readable storage medium that can solve the technical problems of not being able to obtain camera extrinsic parameters through joint calibration of LiDAR and images and the insufficient accuracy of camera calibration results.

[0005] On one hand, this application proposes a joint calibration method, which includes: acquiring a checkerboard image of a calibration board through an imaging device, and acquiring a skeleton point cloud map corresponding to the checkerboard image through a lidar. The calibration board has an array of checkerboard grids, each checkerboard grid including a central region and a boundary region surrounding the central region. The central region and the boundary region have different reflectivities. Multiple initial corner points and the spatial coordinates of each initial corner point are determined based on the point cloud brightness of each point cloud data in the skeleton point cloud map. The point cloud brightness is positively correlated with the reflectivity. The pixel coordinates corresponding to each initial corner point are obtained from the checkerboard image. Based on the spatial coordinates of each initial corner point and the pixel coordinates corresponding to the initial corner point, the transformation matrix between the camera coordinate system corresponding to the imaging device and the lidar coordinate system corresponding to the lidar is calculated.

[0006] According to an optional embodiment of this application, multiple initial corner points are determined based on the point cloud brightness of each point cloud data in the skeleton point cloud diagram, including: determining multiple horizontal lines and multiple vertical lines based on the point cloud brightness of each point cloud data in the skeleton point cloud diagram, and determining multiple initial corner points based on the intersection points of each horizontal line and each vertical line.

[0007] In this embodiment, since the intersection points of the multiple intersecting horizontal and vertical lines are directly and accurately determined as multiple initial corner points, the corner point positions can be quickly located.

[0008] According to an optional embodiment of this application, the reflectivity of the boundary region of the checkerboard pattern is greater than the reflectivity of the central region of the checkerboard pattern. Multiple horizontal lines and multiple vertical lines are determined based on the point cloud brightness of each point cloud data in the skeleton point cloud image, including: determining the point cloud corresponding to the point cloud brightness greater than a first threshold in the skeleton point cloud image as the first target point cloud, and determining multiple horizontal lines and multiple vertical lines based on the first target point cloud.

[0009] In this embodiment, since the reflectivity of the boundary region of the checkerboard is greater than that of the center region of the checkerboard, the point cloud brightness of the boundary region of each checkerboard in the skeleton point cloud map is brighter than that of the center region of the checkerboard. Therefore, by comparing the point cloud brightness with the first threshold, multiple horizontal lines and multiple vertical lines can be quickly determined.

[0010] According to an optional embodiment of this application, the reflectivity of the boundary region of the checkerboard pattern is less than that of the central region of the checkerboard pattern. Multiple horizontal lines and multiple vertical lines are determined based on the point cloud brightness of each point cloud data in the skeleton point cloud image, including: determining the point cloud corresponding to the point cloud brightness of the skeleton point cloud image that is less than a second threshold as the second target point cloud, and determining multiple horizontal lines and multiple vertical lines based on the second target point cloud.

[0011] In this embodiment, since the reflectivity of the boundary area of ​​the checkerboard is less than that of the center area of ​​the checkerboard, the point cloud brightness of the boundary area of ​​each checkerboard in the skeleton point cloud map is darker than that of the center area of ​​the checkerboard. Therefore, by comparing the point cloud brightness with the second threshold, multiple horizontal lines and multiple vertical lines can be quickly determined.

[0012] According to an optional embodiment of this application, the transformation matrix between the camera coordinate system corresponding to the shooting device and the lidar coordinate system corresponding to the lidar is calculated based on the spatial coordinate value of each initial corner point and the pixel coordinate value corresponding to the initial corner point. This includes: obtaining the intrinsic parameter matrix of the shooting device; determining the first pixel coordinate value corresponding to each initial corner point based on the spatial coordinate value of each initial corner point and the intrinsic parameter matrix; determining the second pixel coordinate value corresponding to each initial corner point based on the pixel coordinate value corresponding to each initial corner point and the first pixel coordinate value; and calculating the transformation matrix between the camera coordinate system corresponding to the shooting device and the lidar coordinate system corresponding to the lidar based on the second pixel coordinate value, the spatial coordinate value of the initial corner point, and the intrinsic parameter matrix.

[0013] In this embodiment, the transformation matrix is ​​calculated based on the spatial coordinates of multiple initial corner points and their corresponding pixel coordinates. Since the multiple intersections of the multiple intersecting horizontal and vertical lines are directly determined as multiple initial corner points, there is no need to manually select the spatial coordinates of the initial corner points and the corresponding pixel coordinates. Therefore, the error caused by manual point selection during manual calibration can be greatly reduced.

[0014] On the other hand, this application also proposes a joint calibration device, which includes: an acquisition unit, used to acquire a checkerboard image of a calibration board through an imaging device, and to acquire a skeleton point cloud map corresponding to the checkerboard image through a lidar, wherein the calibration board has an array of checkerboard grids, each checkerboard grid including a central region and a boundary region surrounding the central region, the central region and the boundary region having different reflectivities; a determination unit, used to determine multiple initial corner points and the spatial coordinate values ​​of each initial corner point based on the point cloud brightness of each point cloud data in the skeleton point cloud map, wherein the point cloud brightness is positively correlated with the reflectivity; the acquisition unit is also used to acquire the pixel coordinate values ​​corresponding to each initial corner point from the checkerboard image; and a calculation unit, used to calculate the transformation matrix between the camera coordinate system corresponding to the imaging device and the lidar coordinate system corresponding to the lidar based on the spatial coordinate values ​​of each initial corner point and the pixel coordinate values ​​corresponding to the initial corner point.

[0015] On the other hand, this application also proposes an electronic device, which includes:

[0016] Memory, which stores computer-readable instructions; and

[0017] The processor executes computer-readable instructions stored in memory to implement the joint calibration method.

[0018] On the other hand, this application also proposes a computer-readable storage medium storing computer-readable instructions, which are executed by a processor in an electronic device to implement a joint calibration method.

[0019] As can be seen from the above technical solution, this application can acquire a checkerboard image of a calibration board using an imaging device, and acquire a skeleton point cloud map corresponding to the checkerboard image using a LiDAR. The calibration board forms an array of checkerboard patterns, with the central and boundary regions of each checkerboard pattern having different reflectivities. Since point cloud brightness is positively correlated with reflectivity, the point cloud brightness of the point cloud data corresponding to the boundary regions of the checkerboard pattern differs from the point cloud brightness of the point cloud data corresponding to the central regions of the checkerboard pattern in the skeleton point cloud map. Therefore, multiple initial corner points can be determined based on the point cloud brightness of each point cloud data in the skeleton point cloud map, and the spatial coordinate values ​​of the initial corner points can be accurately obtained from the point cloud data corresponding to the initial corner points. Furthermore, the pixel coordinate values ​​corresponding to each initial corner point can be obtained from the checkerboard image. Subsequently, based on the spatial coordinate values ​​and corresponding pixel coordinate values ​​of each initial corner point, the transformation matrix between the camera coordinate system corresponding to the imaging device and the LiDAR coordinate system corresponding to the LiDAR is calculated. In summary, in the method provided in this application embodiment, the electronic device can automatically select the initial corner point based on the point cloud brightness of each point cloud data in the skeleton point cloud map, without requiring the user to manually select feature points. This method is simple to operate, reduces point selection errors, and improves the accuracy of the transformation matrix. Attached Figure Description

[0020] Figure 1 This is an application environment diagram of the joint calibration method provided in the embodiments of this application.

[0021] Figure 2 This is a flowchart of the joint calibration method provided in the embodiments of this application.

[0022] Figure 3 This is a schematic diagram of the calibration plate provided in an embodiment of this application.

[0023] Figure 4 This is a schematic diagram of the skeleton point cloud provided in an embodiment of this application.

[0024] Figure 5 This is a functional block diagram of the joint calibration method provided in the embodiments of this application.

[0025] Figure 6 This is a schematic diagram of the structure of an electronic device using the joint calibration method provided in an embodiment of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] like Figure 1The diagram shown illustrates the application environment of the joint calibration method provided in an embodiment of this application. The joint calibration method can be applied to one or more electronic devices 1, which communicate with an imaging device 2 and a lidar 3. The imaging device 2 can be a monocular camera or other devices with imaging capabilities.

[0028] Electronic device 1 is a device that can automatically perform parameter value calculation and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to: microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0029] Electronic device 1 can be any electronic product that can interact with a user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), game console, interactive network television (IPTV), smart wearable device, etc.

[0030] Electronic device 1 may also include network devices and / or user devices. Among them, network devices include, but are not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0031] The network in which electronic device 1 is located includes, but is not limited to: the Internet, wide area network, metropolitan area network, local area network, virtual private network (VPN), etc.

[0032] Electronic device 1 may also include a self-moving device. The self-moving device may be a device incorporating self-movement assistance functionality. This self-movement assistance functionality may be implemented via an in-vehicle terminal, and the corresponding self-moving device may be a vehicle equipped with that in-vehicle terminal. The self-moving device may also be a semi-self-moving device or a fully autonomous device. Examples include lawnmowers, sweepers, and robots with navigation capabilities.

[0033] like Figure 2 The diagram shown is a flowchart of a joint calibration method provided in an embodiment of this application. Depending on different requirements, the order of the steps in this flowchart can be changed, and some steps can be omitted.

[0034] S101 acquires a checkerboard image of the calibration board using an imaging device, and acquires a skeleton point cloud map corresponding to the checkerboard image using a lidar.

[0035] In this embodiment, the imaging device and the lidar can be installed at a designated location on the self-moving device. Each checkerboard grid includes a central region and a boundary region surrounding the central region, with the central region and the boundary region having different reflectivities.

[0036] In at least one embodiment of this application, the calibration plate refers to a checkerboard calibration plate, which has an array of checkerboard patterns. The boundary region and the central region of each checkerboard pattern are composed of materials with different reflectivities. For example, the material of the boundary region of each checkerboard pattern can be an aluminized reflective polyester film, and the material of the central region of the checkerboard pattern can be different from the material of the boundary region, i.e., different materials with different reflectivities. For example, the material of the central region of the checkerboard pattern can be glass, ceramic, or plastic, etc., without limitation. Alternatively, the material of the central region of the checkerboard pattern can be the same as the material of the boundary region, but the thickness of the material in the central region is different from the thickness of the material on the boundary, so that different thicknesses reflect different reflectivities. Figure 3 The diagram shown is a schematic of a calibration board provided in an embodiment of this application. Figure 3 The winning bid board has a checkerboard pattern with alternating black and white squares. Figure 3 The calibration board shown is for illustrative purposes only.

[0037] In at least one embodiment of this application, the electronic device acquires a checkerboard image of the calibration board via an imaging device, including:

[0038] Electronic equipment controls the imaging device to capture images of the calibration board, obtaining a checkerboard pattern image.

[0039] The shooting device can be a monocular camera or other device capable of shooting; this application does not impose any restrictions on it.

[0040] In at least one embodiment of this application, the electronic device acquires the skeleton point cloud map corresponding to the checkerboard image via LiDAR, including:

[0041] Electronic equipment controls the lidar to scan the calibration board and obtain a skeleton point cloud map.

[0042] In this embodiment, because the boundary regions and central regions of each chessboard square are composed of materials with different reflectivities, the point cloud brightness of each chessboard square's boundary regions and central regions differs in the skeleton point cloud map. Therefore, the skeleton point cloud map can clearly display multiple horizontal and vertical lines. The skeleton point cloud map is a point cloud image including both horizontal and vertical lines.

[0043] S102, determine multiple initial corner points and the spatial coordinates of each initial corner point based on the point cloud brightness of each point cloud data in the skeleton point cloud map.

[0044] Among them, point cloud brightness is positively correlated with reflectivity, that is, the greater the reflectivity, the stronger the corresponding point cloud brightness.

[0045] like Figure 4 The image shown is a schematic diagram of a skeleton point cloud provided in an embodiment of this application. Figure 4 It includes multiple intersecting horizontal and vertical lines. Among them, Figure 4 The skeleton point cloud shown is for illustrative purposes only.

[0046] In at least one embodiment of this application, the electronic device determines multiple initial corner points based on the point cloud brightness of each point cloud data in the skeleton point cloud map, including:

[0047] The electronic device determines multiple horizontal and vertical lines based on the point cloud brightness of each point cloud data in the skeleton point cloud map. Furthermore, the electronic device determines multiple initial corner points based on the intersection points of each horizontal and vertical line.

[0048] In this embodiment, the electronic device determines the intersection points of each horizontal line and multiple vertical lines as multiple initial corner points.

[0049] In this embodiment, since the boundary region of each checkerboard grid and the center region of each checkerboard grid are composed of materials with different reflectivities, the relationship between the reflectivity of the boundary region of the checkerboard grid and the reflectivity of the center region of the checkerboard grid includes the following cases: the reflectivity of the boundary region of the checkerboard grid is greater than the reflectivity of the center region of the checkerboard grid, or the reflectivity of the boundary region of the checkerboard grid is less than the reflectivity of the center region of the checkerboard grid.

[0050] In this embodiment, if the reflectivity of the boundary region of the checkerboard pattern is greater than that of the central region of the checkerboard pattern, the electronic device determines multiple horizontal and vertical lines based on the point cloud brightness of each point cloud data in the skeleton point cloud map, including:

[0051] The point cloud corresponding to the point cloud brightness greater than the first threshold in the skeleton point cloud map is determined as the first target point cloud, and multiple horizontal lines and multiple vertical lines are determined based on the first target point cloud.

[0052] In this embodiment, if the reflectivity of the boundary region of the checkerboard pattern is less than that of the central region of the checkerboard pattern, the electronic device determines multiple horizontal and vertical lines based on the point cloud brightness of each point cloud data in the skeleton point cloud map, including:

[0053] The electronic device identifies the point cloud corresponding to the point cloud brightness that is less than the second threshold in the skeleton point cloud map as the second target point cloud, and determines multiple horizontal lines and multiple vertical lines based on the second target point cloud.

[0054] The first and second thresholds can be set arbitrarily, and this application does not impose any restrictions on them. Point cloud brightness refers to the brightness reflected by materials with different reflectivities, and point cloud brightness can characterize the reflection intensity of point clouds.

[0055] In this embodiment, the multiple horizontal lines and multiple vertical lines are all composed of point clouds in the skeleton point cloud map. The point cloud is a number of spatial points in the spatial coordinate system. Therefore, the point cloud data corresponding to each point cloud includes the spatial coordinate value of the point cloud. The electronic device can obtain the spatial coordinate value of the initial corner point from the point cloud data corresponding to each initial corner point.

[0056] S103, obtain the pixel coordinates of each initial corner point from the chessboard image.

[0057] In at least one embodiment of this application, the electronic device constructs a pixel coordinate system uOv with the pixel point O of the first row and first column of the checkerboard image as the origin, the parallel line where the first row of pixels is located as the u-axis, and the vertical line where the first column of pixels is located as the v-axis.

[0058] In at least one embodiment of this application, the electronic device obtains the pixel coordinate values ​​corresponding to each initial corner point from the checkerboard image, including:

[0059] Obtain the position of each initial corner point in the pixel coordinate system as the pixel coordinate value.

[0060] S104, based on the spatial coordinates of each initial corner point and the corresponding pixel coordinates, calculate the transformation matrix between the camera coordinate system corresponding to the shooting device and the lidar coordinate system corresponding to the lidar.

[0061] In at least one embodiment of this application, the transformation matrix refers to the transformation relationship between the camera coordinate system and the lidar coordinate system. The transformation matrix transforms points in the lidar coordinate system to the camera coordinate system, or transforms points in the camera coordinate system to the lidar coordinate system.

[0062] In at least one embodiment of this application, the electronic device calculates the transformation matrix between the camera coordinate system corresponding to the shooting device and the lidar coordinate system corresponding to the lidar based on the spatial coordinate values ​​and corresponding pixel coordinate values ​​of each initial corner point, including:

[0063] Obtain the intrinsic parameter matrix of the imaging device. Based on the spatial coordinates of each initial corner point and the intrinsic parameter matrix, determine the first pixel coordinate value corresponding to each initial corner point. Based on the pixel coordinates of each initial corner point and the first pixel coordinate value, determine the second pixel coordinate value corresponding to each initial corner point. Based on the second pixel coordinate value, the spatial coordinates of the initial corner points, and the intrinsic parameter matrix, calculate the transformation matrix between the camera coordinate system corresponding to the imaging device and the LiDAR coordinate system corresponding to the LiDAR.

[0064] Specifically, the spatial coordinate values ​​include the horizontal axis (e.g., x-axis), vertical axis (e.g., y-axis), and z-axis coordinate values ​​of each initial corner point. The first pixel coordinate value includes the first horizontal coordinate and the first vertical coordinate of each initial corner point. The formula for calculating the first pixel coordinate value is as follows:

[0065]

[0066]

[0067]

[0068] Where U0 represents the first x-coordinate value of each initial corner point, p lidar.x p represents the x-axis coordinate of the initial corner point. lidar.z V0 represents the initial ordinate of the corner point, and p represents the first ordinate of the initial corner point. lidar.y The initial corner point's y-axis coordinates are represented by K, which represents the intrinsic parameter matrix, and f... x f represents the focal length of the imaging device along the U-axis. y represents the focal length of the shooting device along the v-axis, cx represents the horizontal coordinate of the principal point in the pixel coordinate system, and cy represents the vertical coordinate of the principal point in the pixel coordinate system. The principal point is the intersection of the optical axis of the shooting device and the checkerboard image.

[0069] Specifically, the pixel coordinates corresponding to each initial corner point include the initial x-coordinate and the initial y-coordinate, and the second pixel coordinates include the second x-coordinate and the second y-coordinate of each initial corner point. The formula for calculating the second pixel coordinates is as follows:

[0070]

[0071]

[0072] Where U1 represents the second horizontal coordinate value, V1 represents the second vertical coordinate value, x represents the initial horizontal coordinate value, y represents the initial vertical coordinate value, dx represents the length of each pixel in the checkerboard image on the u-axis, and dy represents the length of each pixel in the checkerboard image on the v-axis.

[0073] In this embodiment, the transformation matrix includes a rotation matrix and a translation vector, and the formula for calculating the transformation matrix is ​​as follows:

[0074] P camera =K*(R*P lidar +t);

[0075]

[0076]

[0077] Where R represents the rotation matrix, t represents the translation vector, and P lidar P represents spatial coordinate values. camera This represents the coordinate value of the second pixel.

[0078] In this embodiment, based on the spatial coordinates of multiple initial corner points and the corresponding pixel coordinates of multiple initial corner points, multiple equations concerning the rotation matrix and translation vector are obtained. Each equation corresponds one-to-one with each initial corner point. The multiple equations are solved to obtain the solved rotation matrix and the solved translation vector. The solved rotation matrix and the solved translation vector are then concatenated to obtain the transformation matrix.

[0079] The rotation matrix includes multiple unknown Euler angles, and the translation vector includes multiple unknown parameters. Multiple initial corner points are selected based on the sum of the number of unknown Euler angles and the number of unknown parameters.

[0080] For example, if the rotation matrix includes 3 unknown Euler angles and the translation vector includes 3 unknown parameters, then the sum of the unknowns between the unknown Euler angles and the unknown parameters is 6. Therefore, it is only necessary to select 6 initial corner points. Based on the spatial coordinate values ​​of the 6 initial corner points and the pixel coordinate values ​​corresponding to the 6 initial corner points, 6 equations are obtained. Solving the 6 equations yields the transformation matrix between the camera coordinate system corresponding to the shooting device and the LiDAR coordinate system corresponding to the LiDAR.

[0081] In this embodiment, the unknown quantity sum between the unknown Euler angles and the unknown parameters is calculated. Since the transformation matrix can be obtained by calculating the spatial coordinates and pixel coordinates of the initial corner points with the same quantity as the unknown quantity sum, it is not necessary to include all the spatial coordinates of the initial corner points and the pixel coordinates of all the initial corner points in the calculation. Therefore, the amount of calculation can be reduced, and the transformation matrix between the camera coordinate system corresponding to the shooting device and the LiDAR coordinate system corresponding to the LiDAR can be calculated quickly.

[0082] In other embodiments of this application, the electronic device further calculates the transformation matrix between the camera coordinate system corresponding to the shooting device and the lidar coordinate system corresponding to the lidar based on the spatial coordinate values ​​and corresponding pixel coordinate values ​​of each initial corner point.

[0083] The electronic device calculates the total error based on the coordinates of the first pixel and the second pixel. Furthermore, it optimizes the total error value based on the least squares algorithm until the total error value is minimized, thus obtaining the transformation matrix.

[0084] Specifically, the formula for calculating the total error value is as follows:

[0085]

[0086] E u =|U0-U1|;

[0087] E v =|V0-V1|;

[0088] Among them, E error E represents the total error value, n represents the number of initial corner points, and E represents the total error value. u E represents the difference in x-coordinates between the first x-coordinate and the corresponding second x-coordinate of each initial corner point. v E represents the difference in ordinate between the first ordinate value and the corresponding second ordinate value of each initial corner point. iu E represents the difference in the x-coordinates of the i-th initial corner point among multiple initial corner points. iv This represents the difference in the ordinate of the i-th initial corner point among multiple initial corner points.

[0089] In this embodiment, the total error value is optimized according to the least squares algorithm. Since more initial corner point spatial coordinates and corresponding pixel coordinates are used, the accuracy of the transformation matrix can be improved.

[0090] As can be seen from the above technical solution, this application can acquire a checkerboard image of a calibration board using an imaging device, and acquire a skeleton point cloud map corresponding to the checkerboard image using a lidar. The calibration board is formed with an array of checkerboard patterns, and the boundary region and center region of each checkerboard pattern are composed of materials with different reflectivities. Because the boundary region and center region of each checkerboard pattern are composed of materials with different reflectivities, the point cloud brightness of the point cloud data corresponding to the boundary region of the checkerboard pattern differs from the point cloud brightness of the point cloud data corresponding to the center region of the checkerboard pattern in the skeleton point cloud map. Therefore, multiple initial corner points can be determined based on the point cloud brightness of each point cloud data in the skeleton point cloud map. Since materials with different reflectivities do not affect the lidar's acquisition of spatial coordinate values, the spatial coordinate values ​​of the initial corner points can be accurately obtained from the point cloud data corresponding to the initial corner points. Furthermore, the pixel coordinate values ​​corresponding to each initial corner point can also be obtained from the aforementioned checkerboard image. Subsequently, based on the spatial coordinates and corresponding pixel coordinates of each initial corner point, the transformation matrix between the camera coordinate system of the capturing device and the lidar coordinate system of the lidar is calculated. In summary, in the method provided in this application embodiment, the electronic device can automatically select initial corner points based on the point cloud brightness of each point cloud data in the skeleton point cloud map, eliminating the need for manual selection of feature points by the user. This simplifies the operation, reduces point selection errors, and improves the accuracy of the transformation matrix.

[0091] like Figure 5 The diagram shown is a functional block diagram of the joint calibration method provided in an embodiment of this application. The joint calibration device 11 includes an acquisition unit 110, a determination unit 111, and a calculation unit 112. The module / unit referred to in this application refers to a series of computer-readable instruction segments that can be acquired by the processor 13 and perform a fixed function, and are stored in the memory 12. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0092] The acquisition unit 110 is used to acquire a checkerboard image of the calibration board through an imaging device, and to acquire a skeleton point cloud map corresponding to the checkerboard image through a lidar. The calibration board has an array of checkerboards, and the boundary region and the center region of each checkerboard are composed of materials with different reflectivities.

[0093] The determining unit 111 is used to determine multiple initial corner points and the spatial coordinate values ​​of each initial corner point based on the point cloud brightness of each point cloud data in the skeleton point cloud map.

[0094] In at least one embodiment of this application, the determining unit 111 includes:

[0095] The first determining subunit is used to determine multiple horizontal lines and multiple vertical lines based on the point cloud brightness of each point cloud data in the skeleton point cloud diagram;

[0096] The second determining sub-unit is used to determine multiple initial corner points based on the intersection points of each horizontal line and each vertical line.

[0097] In this embodiment, if the reflectivity of the boundary region of the checkerboard pattern is greater than the reflectivity of the center region of the checkerboard pattern, the determining unit 111 is used to:

[0098] The point cloud corresponding to the point cloud brightness greater than the first threshold in the skeleton point cloud map is determined as the first target point cloud, and multiple horizontal lines and multiple vertical lines are determined based on the first target point cloud.

[0099] In this embodiment, if the reflectivity of the boundary region of the checkerboard pattern is less than the reflectivity of the central region of the checkerboard pattern, the determining unit 111 is further configured to:

[0100] The point cloud corresponding to the point cloud brightness that is less than the second threshold in the skeleton point cloud map is determined as the second target point cloud, and multiple horizontal lines and multiple vertical lines are determined based on the second target point cloud.

[0101] The acquisition unit 110 is used to acquire the pixel coordinate values ​​corresponding to each initial corner point from the chessboard image.

[0102] The calculation unit 112 is used to calculate the transformation matrix between the camera coordinate system corresponding to the shooting device and the lidar coordinate system corresponding to the lidar based on the spatial coordinate value of each initial corner point and the pixel coordinate value corresponding to the initial corner point.

[0103] In at least one embodiment of this application, the computing unit 112 includes:

[0104] The intrinsic parameter acquisition subunit is used to acquire the intrinsic parameter matrix of the shooting device;

[0105] The coordinate determination subunit is used to determine the first pixel coordinate value corresponding to each initial corner point based on the spatial coordinate value and intrinsic parameter matrix of each initial corner point; and to determine the second pixel coordinate value corresponding to each initial corner point based on the pixel coordinate value and the first pixel coordinate value of each initial corner point.

[0106] The matrix transformation subunit is used to calculate the transformation matrix between the camera coordinate system corresponding to the shooting device and the lidar coordinate system corresponding to the lidar, based on the second pixel coordinate value, the spatial coordinate value of the initial corner point, and the intrinsic parameter matrix.

[0107] As can be seen from the above technical solution, this application can acquire a checkerboard image of a calibration board using an imaging device, and acquire a skeleton point cloud map corresponding to the checkerboard image using a LiDAR. The calibration board forms an array of checkerboard patterns, with the central and boundary regions of each checkerboard pattern having different reflectivities. Since point cloud brightness is positively correlated with reflectivity, the point cloud brightness of the point cloud data corresponding to the boundary regions of the checkerboard pattern differs from the point cloud brightness of the point cloud data corresponding to the central regions of the checkerboard pattern in the skeleton point cloud map. Therefore, multiple initial corner points can be determined based on the point cloud brightness of each point cloud data in the skeleton point cloud map, and the spatial coordinate values ​​of the initial corner points can be accurately obtained from the point cloud data corresponding to the initial corner points. Furthermore, the pixel coordinate values ​​corresponding to each initial corner point can be obtained from the checkerboard image. Subsequently, based on the spatial coordinate values ​​and corresponding pixel coordinate values ​​of each initial corner point, the transformation matrix between the camera coordinate system corresponding to the imaging device and the LiDAR coordinate system corresponding to the LiDAR is calculated. In summary, in the method provided in this application embodiment, the electronic device can automatically select the initial corner point based on the point cloud brightness of each point cloud data in the skeleton point cloud map, without requiring the user to manually select feature points. This method is simple to operate, reduces point selection errors, and improves the accuracy of the transformation matrix.

[0108] like Figure 6 The diagram shown is a schematic diagram of the structure of an electronic device using the joint calibration method provided in an embodiment of this application.

[0109] In one embodiment of this application, the electronic device 1 includes, but is not limited to, a memory 12, a processor 13, and computer-readable instructions, such as a joint calibration program, stored in the memory 12 and executable on the processor 13.

[0110] Those skilled in the art will understand that the schematic diagram is merely an example of electronic device 1 and does not constitute a limitation on electronic device 1. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device 1 may also include input / output devices, network access devices, buses, etc.

[0111] Processor 13 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. Processor 13 is the computational core and control center of electronic device 1, connecting various parts of electronic device 1 through various interfaces and lines, and executing the operating system of electronic device 1, as well as various installed application programs and program code.

[0112] For example, computer-readable instructions may be divided into one or more modules / units, one or more of which are stored in memory 12 and executed by processor 13 to complete this application. One or more modules / units may be a series of computer-readable instruction segments capable of performing a specific function, which describe the execution process of the computer-readable instructions in electronic device 1. For example, the computer-readable instructions may be divided into an acquisition unit 110, a determination unit 111, and a calculation unit 112.

[0113] The memory 12 can be used to store computer-readable instructions and / or modules. The processor 13 implements various functions of the electronic device 1 by running or executing the computer-readable instructions and / or modules stored in the memory 12 and by calling the data stored in the memory 12. The memory 12 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. The memory 12 may include non-volatile and volatile memory, such as: hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other storage devices.

[0114] The memory 12 can be the external memory and / or internal memory of the electronic device 1. Furthermore, the memory 12 can be a physical memory, such as a memory stick, a TF card (Trans-flash Card), etc.

[0115] If the modules / units integrated in electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by instructing related hardware through computer-readable instructions. The computer-readable instructions can be stored in a computer-readable storage medium, and when executed by a processor, the computer-readable instructions can implement the steps of the various method embodiments described above.

[0116] Computer-readable instructions include computer-readable instruction code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media can include: any entity or device capable of carrying computer-readable instruction code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), and random access memory (RAM).

[0117] Combination Figure 2 The memory 12 in electronic device 1 stores computer-readable instructions to implement a joint calibration method, and the processor 13 can execute the computer-readable instructions to achieve:

[0118] The calibration board is imaged using a camera, and a corresponding skeleton point cloud image is obtained using a LiDAR. The calibration board has an array of checkerboard grids, each grid consisting of a central region and a boundary region surrounding the central region. The central and boundary regions have different reflectivities. Multiple initial corner points and their spatial coordinates are determined based on the point cloud brightness of each point in the skeleton point cloud image. Point cloud brightness is positively correlated with reflectivity. The pixel coordinates of each initial corner point are obtained from the checkerboard image. Based on the spatial coordinates and pixel coordinates of each initial corner point, a transformation matrix is ​​calculated between the camera coordinate system of the camera and the LiDAR coordinate system.

[0119] Specifically, the specific implementation method of the processor 13 for the above-mentioned computer-readable instructions can be found in [reference]. Figure 2 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0120] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0121] A computer-readable storage medium stores computer-readable instructions, which, when executed by processor 13, are used to perform the following steps:

[0122] The calibration board is imaged using a camera, and a corresponding skeleton point cloud image is obtained using a LiDAR. The calibration board has an array of checkerboard grids, each grid consisting of a central region and a boundary region surrounding the central region. The central and boundary regions have different reflectivities. Multiple initial corner points and their spatial coordinates are determined based on the point cloud brightness of each point in the skeleton point cloud image. Point cloud brightness is positively correlated with reflectivity. The pixel coordinates of each initial corner point are obtained from the checkerboard image. Based on the spatial coordinates and pixel coordinates of each initial corner point, a transformation matrix is ​​calculated between the camera coordinate system of the camera and the LiDAR coordinate system.

[0123] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0124] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0125] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within this application. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0126] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. The multiple units or devices described may also be implemented by a single unit or device through software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.

Claims

1. A joint calibration method, characterized in that, The joint calibration method includes: The calibration board is imaged using a camera and the corresponding skeleton point cloud image is obtained using a lidar. The calibration board has an array of checkerboard grids, each checkerboard grid including a central region and a boundary region surrounding the central region. The central region and the boundary region have different reflectivities. Multiple horizontal and vertical lines are determined based on the point cloud brightness of each point cloud data in the skeleton point cloud map. The multiple horizontal and vertical lines are all composed of point clouds in the skeleton point cloud map. Multiple initial corner points and the spatial coordinate values ​​of each initial corner point are determined based on the intersection points of the multiple horizontal and vertical lines. The point cloud brightness is positively correlated with the reflectivity. Obtain the pixel coordinates of each initial corner point from the chessboard image; Based on the spatial coordinates of each initial corner point and the pixel coordinates corresponding to the initial corner point, calculate the transformation matrix between the camera coordinate system corresponding to the shooting device and the lidar coordinate system corresponding to the lidar. The reflectivity of the boundary region of the chessboard grid is greater than that of the center region of the chessboard grid. The step of determining multiple horizontal lines and multiple vertical lines based on the point cloud brightness of each point cloud data in the skeleton point cloud map includes: determining the point cloud corresponding to the point cloud brightness greater than a first threshold in the skeleton point cloud map as the first target point cloud, and determining multiple horizontal lines and multiple vertical lines based on the first target point cloud.

2. The joint calibration method as described in claim 1, characterized in that, The reflectivity of the boundary region of the checkerboard pattern is less than that of the center region of the checkerboard pattern; the step of determining multiple horizontal and vertical lines based on the point cloud brightness of each point cloud data in the skeleton point cloud map further includes: The point cloud with a brightness less than the second threshold in the skeleton point cloud map is determined as the second target point cloud; Multiple horizontal lines and multiple vertical lines are determined based on the second target point cloud.

3. The joint calibration method as described in claim 1, characterized in that, The step of calculating the transformation matrix between the camera coordinate system corresponding to the capturing device and the lidar coordinate system corresponding to the lidar, based on the spatial coordinate values ​​of each initial corner point and the pixel coordinate values ​​corresponding to the initial corner point, includes: Obtain the intrinsic parameter matrix of the shooting device; Based on the spatial coordinates of each initial corner point and the intrinsic parameter matrix, determine the first pixel coordinates corresponding to each initial corner point; Based on the pixel coordinates of each initial corner point and the first pixel coordinates, determine the second pixel coordinates of each initial corner point; Based on the second pixel coordinate value, the spatial coordinate value of the initial corner point, and the intrinsic parameter matrix, calculate the transformation matrix between the camera coordinate system corresponding to the shooting device and the lidar coordinate system corresponding to the lidar.

4. A combined calibration device, characterized in that, The joint calibration device includes: The acquisition unit is used to acquire a checkerboard image of the calibration board through an imaging device, and to acquire a skeleton point cloud map corresponding to the checkerboard image through a lidar. The calibration board has an array of checkerboards, each checkerboard including a central region and a boundary region surrounding the central region. The central region and the boundary region have different reflectivities. The determining unit includes a first determining subunit and a second determining subunit. The first determining subunit is used to determine multiple horizontal lines and multiple vertical lines based on the point cloud brightness of each point cloud data in the skeleton point cloud map. The multiple horizontal lines and multiple vertical lines are all composed of point clouds in the skeleton point cloud map. The second determining subunit is used to determine multiple initial corner points and the spatial coordinates of each initial corner point based on the intersection points of the multiple horizontal lines and the multiple vertical lines. The point cloud brightness is positively correlated with the reflectivity. The reflectivity of the boundary region of the chessboard is greater than that of the center region of the chessboard. The first determining subunit is used to determine multiple horizontal lines and multiple vertical lines based on the point cloud brightness of each point cloud data in the skeleton point cloud map, including: determining the point cloud corresponding to the point cloud brightness greater than a first threshold in the skeleton point cloud map as the first target point cloud, and determining multiple horizontal lines and multiple vertical lines based on the first target point cloud. The acquisition unit is further configured to acquire the pixel coordinate values ​​corresponding to each initial corner point from the chessboard image; The calculation unit is used to calculate the transformation matrix between the camera coordinate system corresponding to the shooting device and the lidar coordinate system corresponding to the lidar, based on the spatial coordinate value of each initial corner point and the pixel coordinate value corresponding to the initial corner point.

5. The joint calibration apparatus as described in claim 4, characterized in that, The computing unit includes: The intrinsic parameter acquisition subunit is used to acquire the intrinsic parameter matrix of the shooting device; The coordinate determination subunit is used to determine the first pixel coordinate value corresponding to each initial corner point based on the spatial coordinate value and intrinsic parameter matrix of each initial corner point; and to determine the second pixel coordinate value corresponding to each initial corner point based on the pixel coordinate value and the first pixel coordinate value of each initial corner point. The matrix transformation subunit is used to calculate the transformation matrix between the camera coordinate system corresponding to the shooting device and the lidar coordinate system corresponding to the lidar, based on the second pixel coordinate value, the spatial coordinate value of the initial corner point, and the intrinsic parameter matrix.

6. An electronic device, characterized in that, The electronic device includes: Memory, storing at least one instruction; and The processor executes the at least one instruction to implement the joint calibration method as described in any one of claims 1 to 3.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, which is executed by a processor in an electronic device to implement the joint calibration method as described in any one of claims 1 to 3.

Images