Method, system and medium for calibrating extrinsic parameters of a lidar and a structure light camera with blind fill

Abstract

The application discloses a laser radar and a blind area filling structured light camera external parameter calibration method, system and medium, based on a calibration board plane constraint, determining a pose relationship between an RGB camera and the calibration board, determining a pose relationship between a laser radar and the calibration board, thereby obtaining a pose relationship between the laser radar and the RGB camera, and then according to a relative relationship between the RGB camera and a structured light camera, obtaining an external parameter relationship between the laser radar and the structured light camera. Through determination of the external parameter relationship between the laser radar and the structured light camera, data detected by the laser radar and the structured light camera is converted to a common coordinate system, information of the entire visual angle of the robot is obtained in the same dimension, global strategy and judgment are facilitated, in addition, through the height and the visual angle of the laser radar, the blind area distance of the laser radar is determined, through the visual angle, the tilt angle and the height of the structured light camera, the measurement distance of the structured light camera is determined, the blind area distance or the measurement distance is adjusted, and the blind area problem of the robot is solved.

Description

Technical Field

[0001] This invention relates to the field of robot autonomous navigation, and more specifically, to a method, system, and medium for extrinsic parameter calibration of lidar and blind spot structured light cameras. Background Technology

[0002] Currently, existing methods for determining the extrinsic parameters of LiDAR and structured light cameras using calibration boards require a certain shared field of view between the two sensors. However, due to their installation location and inherent field of view, the shared field of view between the sensors is relatively small, resulting in significant errors in the extrinsic parameter relationship between the LiDAR and structured light cameras.

[0003] Therefore, existing technologies have shortcomings and urgently need improvement. Summary of the Invention

[0004] In view of the above problems, the purpose of this invention is to provide a method, system and medium for calibrating the extrinsic parameters of a lidar and a blind spot structured light camera, which can more accurately determine the global field of view of a robot.

[0005] The first aspect of this invention provides a method for calibrating the extrinsic parameters of a lidar and a structured light camera for blind spot compensation, comprising:

[0006] Acquire information about the points where the lidar falls on the blank area of ​​the calibration board, the corner points of the checkerboard, the laser data, and the checkerboard image data with the corresponding timestamp.

[0007] Based on the corner point information of the chessboard, the pose information of the RGB camera and the calibration board is obtained;

[0008] Based on the plane constraints of the preset calibration board, the closed-form initial value information of the extrinsic parameters of the lidar and the RGB camera is obtained according to the information of the point of the lidar falling on the blank area of ​​the preset calibration board and the pose information of the RGB camera and the calibration board.

[0009] Based on the preset algorithm, the external parameter information of the lidar and RGB camera is obtained according to the laser data, the corresponding timestamp chessboard image data and initial values.

[0010] Based on the extrinsic information of the LiDAR and RGB camera and the preset relative relationship information between the RGB sensor and the structured light depth sensor, the extrinsic relationship information of the LiDAR and the structured light camera is obtained.

[0011] The calibration board consists of a blank area and a checkerboard area, and both belong to the same plane.

[0012] This plan also includes:

[0013] The structured light camera is set as the reference point, and coordinate systems are set separately for different sensors, namely the RGB sensor coordinate system. Depth sensor coordinate system and the projector coordinate system

[0014] Set the lidar as the reference point and define the lidar coordinate system T. lidar ;

[0015] Using the preset calibration plate as the reference point, set the calibration plate coordinate system T. board .

[0016] In this scheme, the closed-form initial values ​​of the extrinsic parameters of the LiDAR and RGB camera are specifically as follows:

[0017] Set the relative relationship of the extrinsic parameters between the lidar coordinate system and the RGB camera coordinate system as follows: in Indicates rotation, Let represent translation. The two coordinate systems together form a six-degree-of-freedom pose. The transformation relationship from the camera coordinate system to the lidar coordinate system is then:

[0018] Based on the RGB camera coordinate system, the plane of the calibration plate can be parameterized as π. c =[n c ,d], where n c Let represent the direction of the plane, and d represent the intercept of the plane equation. Then, the equation representing a 3D point on the calibration plate plane in the camera coordinate system is: Where (1) represents the equation number;

[0019] Based on the relative extrinsic parameters of the LiDAR and the RGB camera, and the equations representing the three-dimensional points on the calibration board in the camera coordinate system, we obtain... (2) Number the equation;

[0020] Set the coordinates of the next measurement point in the lidar coordinate system to P. l = [x,y,z], its representation in the camera coordinate system is as follows: The representation of matrix multiplication is as follows:

[0021]

[0022] Where (3) represents the corresponding equation number;

[0023] Based on equations (2) and (3) above, we obtain Where (4) represents the number of the corresponding equation;

[0024] Based on the preset algorithm, multiple sets of LiDAR and corresponding timestamp chessboard image data are substituted into equation (4) to obtain the closed-form initial value argmin of the corresponding LiDAR and RGB camera extrinsic parameters.

[0025] In this solution, the extrinsic parameters of the LiDAR and RGB camera are specifically as follows:

[0026] Let N be the laser point that the i-th frame lidar falls on the calibration board plane. i The equation of the calibration board plane corresponding to the i-th laser frame in RGB camera coordinates is expressed as follows: Its optimization model is expressed as:

[0027]

[0028] The i-th frame of lidar data belongs to the i-th group of data in N groups of lidar data and checkerboard image data.

[0029] This plan also includes:

[0030] Install the lidar horizontally;

[0031] Set the installation height of the lidar to H and the horizontal field of view to θ.

[0032] Let d be the blind zone distance of the lidar directly in front, and its formula is:

[0033] This plan also includes:

[0034] Set the vertical field of view of the structured light camera to β and the tilt angle to α.

[0035] Set the installation height of the structured light camera to h;

[0036] Let the measurement distance of the structured light camera be s, and its formula is: s=h / tan(90-β-α)°, where 0<α<90°.

[0037] A second aspect of the present invention provides an extrinsic parameter calibration system for a lidar and a blind-filling structured light camera, comprising a memory and a processor. The memory stores a extrinsic parameter calibration method program for the lidar and blind-filling structured light camera. When the processor executes the extrinsic parameter calibration method program for the lidar and blind-filling structured light camera, it performs the following steps:

[0038] Acquire information about the points where the lidar falls on the blank area of ​​the calibration board, the corner points of the checkerboard, the laser data, and the checkerboard image data with the corresponding timestamp.

[0039] Based on the corner point information of the chessboard, the pose information of the RGB camera and the calibration board is obtained;

[0040] Based on the plane constraints of the preset calibration board, the closed-form initial value information of the extrinsic parameters of the lidar and the RGB camera is obtained according to the information of the point of the lidar falling on the blank area of ​​the preset calibration board and the pose information of the RGB camera and the calibration board.

[0041] Based on the preset algorithm, the external parameter information of the lidar and RGB camera is obtained according to the laser data, the corresponding timestamp chessboard image data and initial values.

[0042] Based on the extrinsic information of the LiDAR and RGB camera and the preset relative relationship information between the RGB sensor and the structured light depth sensor, the extrinsic relationship information of the LiDAR and the structured light camera is obtained.

[0043] The calibration board consists of a blank area and a checkerboard area, and both belong to the same plane.

[0044] This plan also includes:

[0045] The structured light camera is set as the reference point, and coordinate systems are set separately for different sensors, namely the RGB sensor coordinate system. Depth sensor coordinate system and the projector coordinate system

[0046] Set the lidar as the reference point and define the lidar coordinate system T. lidar ;

[0047] Using the preset calibration plate as the reference point, set the calibration plate coordinate system T. board .

[0048] In this scheme, the closed-form initial values ​​of the extrinsic parameters of the LiDAR and RGB camera are specifically as follows:

[0049] Set the relative relationship of the extrinsic parameters between the lidar coordinate system and the RGB camera coordinate system as follows: in Indicates rotation, Let represent translation. The two coordinate systems together form a six-degree-of-freedom pose. The transformation relationship from the camera coordinate system to the lidar coordinate system is then:

[0050] Based on the RGB camera coordinate system, the plane of the calibration plate can be parameterized as π. c =[n c ,d], where n c Let represent the direction of the plane, and d represent the intercept of the plane equation. Then, the equation representing a 3D point on the calibration plate plane in the camera coordinate system is: Where (1) represents the equation number;

[0051] Based on the relative extrinsic parameters of the LiDAR and the RGB camera, and the equations representing the three-dimensional points on the calibration board in the camera coordinate system, we obtain... (2) Number the equation;

[0052] Set the coordinates of the next measurement point in the lidar coordinate system to P. l = [x,y,z], its representation in the camera coordinate system is as follows: The representation of matrix multiplication is as follows:

[0053] Where (3) represents the corresponding equation number;

[0054] Based on equations (2) and (3) above, we obtain Where (4) represents the number of the corresponding equation;

[0055] Based on the preset algorithm, multiple sets of LiDAR and corresponding timestamp chessboard image data are substituted into equation (4) to obtain the closed-form initial value argmin of the corresponding LiDAR and RGB camera extrinsic parameters.

[0056] In this solution, the extrinsic parameters of the LiDAR and RGB camera are specifically as follows:

[0057] Let N be the laser point that the i-th frame lidar falls on the calibration board plane. i The equation of the calibration board plane corresponding to the i-th laser frame in RGB camera coordinates is expressed as follows: Its optimization model is expressed as:

[0058]

[0059] The i-th frame of lidar data belongs to the i-th group of data in N groups of lidar data and checkerboard image data.

[0060] This plan also includes:

[0061] Install the lidar horizontally;

[0062] Set the installation height of the lidar to H and the horizontal field of view to θ.

[0063] Let d be the blind zone distance of the lidar directly in front, and its formula is:

[0064] This plan also includes:

[0065] Set the vertical field of view of the structured light camera to β and the tilt angle to α.

[0066] Set the installation height of the structured light camera to h;

[0067] Let the measurement distance of the structured light camera be s, and its formula is: s=h / tan(90-β-α)°, where 0<α<90°.

[0068] A third aspect of the present invention provides a computer medium storing a program for extrinsic parameter calibration of a lidar and a blind spot structured light camera. When the program for extrinsic parameter calibration of a lidar and a blind spot structured light camera is executed by a processor, it implements the steps of the extrinsic parameter calibration method of the lidar and blind spot structured light camera as described in any of the preceding claims.

[0069] This invention discloses a method, system, and medium for extrinsic parameter calibration of a lidar and a structured light camera. By determining the extrinsic parameter relationship between the lidar and the structured light camera, the data detected by the lidar and the structured light camera are transformed into a common coordinate system, obtaining information about the robot's entire field of view in the same dimension, which facilitates global strategy and judgment. Furthermore, this invention determines the blind zone distance of the lidar using its height and viewing angle; and determines the measurement distance of the structured light camera using its viewing angle, tilt angle, and height, adjusting the blind zone distance or the measurement distance to solve the robot's blind zone problem. Attached Figure Description

[0070] Figure 1 A flowchart of the extrinsic parameter calibration method for the lidar and blind spot structured light camera of the present invention is shown;

[0071] Figure 2 This diagram illustrates the working principle of complementary field of view between lidar and structured light camera;

[0072] Figure 3 A blind zone analysis diagram of the lidar is shown;

[0073] Figure 4 A schematic diagram showing the installation angle and measurement distance of the structured light camera is provided.

[0074] Figure 5 A schematic diagram of constructing a calibration board with a common viewing area is shown;

[0075] Figure 6 A schematic diagram showing the relationship between the lidar coordinate system, the RGB camera coordinate system, and the calibration board coordinate system is shown.

[0076] Figure 7 A block diagram of the extrinsic parameter calibration system for the lidar and blind spot structured light camera of the present invention is shown. Detailed Implementation

[0077] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0078] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0079] Figure 1 A flowchart of the extrinsic parameter calibration method for the lidar and blind spot structured light camera of the present invention is shown.

[0080] As shown in the figure, this invention discloses an extrinsic parameter calibration method for a lidar and a blind spot-filling structured light camera, including:

[0081] S102, acquire information about the points where the lidar falls on the blank area of ​​the calibration board, information about the corner points of the checkerboard, laser data, and checkerboard image data information with corresponding timestamps;

[0082] S104, based on the checkerboard corner point information, obtains the pose information of the RGB camera and the calibration board;

[0083] S106. Based on the plane constraint of the preset calibration board, according to the information of the point where the lidar falls on the blank area of ​​the preset calibration board and the pose information of the RGB camera and the calibration board, the closed-form solution initial value information of the extrinsic parameters of the lidar and the RGB camera is obtained.

[0084] S108, based on a preset algorithm, obtains the external parameter information of the lidar and RGB camera according to the laser data, the corresponding timestamp checkerboard image data and initial values;

[0085] S110: Based on the extrinsic information of the LiDAR and the RGB camera and the preset relative relationship information between the RGB sensor and the structured light depth sensor, the extrinsic relationship information of the LiDAR and the structured light camera is obtained.

[0086] It should be noted that the calibration board consists of a blank area and a checkerboard area, both belonging to the same plane. The points measured by the lidar fall on the upper half of the blank area of ​​the calibration board, while the checkerboard area is within the field of view of the RGB camera. Although the two sensors observe different parts, they belong to the same calibration board plane; therefore, the extrinsic parameter relationship between the two can be indirectly obtained using planar constraints. The structured light camera includes an RGB sensor, a depth sensor, and a projector sensor. The relative relationships of these three sensors are known, thus obtaining the six-degree-of-freedom relative spatial pose relationship between the lidar coordinate system and the depth sensor data, i.e., the extrinsic parameter relationship between the lidar and the structured light camera.

[0087] According to an embodiment of the present invention, it further includes:

[0088] The structured light camera is set as the reference point, and coordinate systems are set separately for different sensors, namely the RGB sensor coordinate system. Depth sensor coordinate system and the projector coordinate system

[0089] Set the lidar as the reference point and define the lidar coordinate system T. lidar ;

[0090] Using the preset calibration plate as the reference point, set the calibration plate coordinate system T. board .

[0091] It should be noted that a structured light camera comprises three coordinate systems. These represent the coordinate systems of the RGB sensor, depth sensor, and projector, respectively. The relative relationships between these three sensors are pre-defined and known. For the LiDAR, there is one coordinate system: the LiDAR coordinate system T. lidar ; Calibration plate coordinate system T board The six-degree-of-freedom relative spatial pose relationship between the aforementioned lidar coordinate system and the depth sensor data is the parameter that the calibration method described in this patent ultimately requires.

[0092] According to an embodiment of the present invention, the initial closed-form solution of the extrinsic parameters of the lidar and the RGB camera is specifically as follows:

[0093] Set the relative relationship of the extrinsic parameters between the lidar coordinate system and the RGB camera coordinate system as follows: in Indicates rotation, Let represent translation. The two coordinate systems together form a six-degree-of-freedom pose. The transformation relationship from the camera coordinate system to the lidar coordinate system is then:

[0094] Based on the RGB camera coordinate system, the plane of the calibration plate can be parameterized as π. c =[n c ,d], where n c Let represent the direction of the plane, and d represent the intercept of the plane equation. Then, the equation representing a 3D point on the calibration plate plane in the camera coordinate system is: Where (1) represents the equation number;

[0095] Based on the relative extrinsic parameters of the LiDAR and the RGB camera, and the equations representing the three-dimensional points on the calibration board in the camera coordinate system, we obtain... (2) Number the equation;

[0096] Set the coordinates of the next measurement point in the lidar coordinate system to P. l = [x,y,z], its representation in the camera coordinate system is as follows: The representation of matrix multiplication is as follows:

[0097] Where (3) represents the corresponding equation number;

[0098] Based on equations (2) and (3) above, we obtain Where (4) represents the number of the corresponding equation;

[0099] Based on the preset algorithm, multiple sets of LiDAR and corresponding timestamp chessboard image data are substituted into equation (4) to obtain the closed-form initial value argmin of the corresponding LiDAR and RGB camera extrinsic parameters.

[0100] It should be noted that the preset algorithm is a nonlinear least squares (DLT) algorithm. Based on equation (4), the DLT algorithm is used to estimate the 3x3 H matrix (a total of 9 parameters), which becomes a linear least squares problem. The lidar used in this embodiment is a 3D lidar. One frame of lidar can provide three effective constraints. The plane equation can be determined if three points fall on the calibration board plane. Only three frames of lidar and the corresponding timestamp of the checkerboard image data are needed to solve the closed-form solution. This value is set as the initial value argmin of the closed-form solution of the lidar and RGB camera extrinsic parameters.

[0101] According to an embodiment of the present invention, the extrinsic parameters of the lidar and the RGB camera are specifically as follows:

[0102] Let N be the laser point that the i-th frame lidar falls on the calibration board plane. i The equation of the calibration board plane corresponding to the i-th laser frame in RGB camera coordinates is expressed as follows: Its optimization model is expressed as:

[0103]

[0104]

[0105] The i-th frame of lidar data belongs to the i-th group of data in N groups of lidar data and checkerboard image data.

[0106] It should be noted that the i-th data in the N sets of data includes the i-th frame of LiDAR data and the i-th frame of image data corresponding to the timestamp. The laser point of the i-th frame of LiDAR falling on the calibration board plane is set as N. i The equation of the calibration plate plane corresponding to the i-th laser frame is represented in camera coordinates as follows: The expression for the optimization model is as follows:

[0107]

[0108] The above formula represents the calculation of the distance between N laser points on the calibration board plane of the LiDAR point cloud in the i-th frame, based on the relative extrinsic parameters between the current checkerboard and the camera (obtainable from checkerboard calibration). This calculation is performed for one frame, and then accumulated across all N frames. The resulting sum of squared distances is the target for optimization. The minimum sum of squared distances indicates that the LiDAR points on the calibration board plane are well-positioned on the checkerboard plane. Specifically, in the optimization process, the Jacobian error function can be derived using Lie algebra methods, while optimization can employ nonlinear or Gauss-Newton optimization theories, ultimately determining the optimal solution. The optimal solution is set as the extrinsic parameters of the LiDAR and the RGB camera.

[0109] According to an embodiment of the present invention, it further includes:

[0110] Install the lidar horizontally;

[0111] Set the installation height of the lidar to H and the horizontal field of view to θ.

[0112] Let d be the blind zone distance of the lidar directly in front, and its formula is:

[0113] It should be noted that the ranging principle of lidar is based on time-of-flight ranging, calculating the distance based on the emission and reception time of the laser and the speed of light. This embodiment of the invention uses a 3D mechanical lidar as an example. The lidar can obtain measurement data in both the horizontal and vertical directions. A Cartesian coordinate system is established with the center of the lidar to obtain the coordinates of each point. The coordinates of all detected data points form a point cloud.

[0114] According to an embodiment of the present invention, it further includes:

[0115] Set the vertical field of view of the structured light camera to β and the tilt angle to α.

[0116] Set the installation height of the structured light camera to h;

[0117] Let the measurement distance of the structured light camera be s, and its formula is: s=h / tan(90-β-α)°, where 0<α<90°.

[0118] It's important to explain the ranging principle of structured light cameras: A structured light camera generally consists of two parts: a projector and a camera. The projector actively projects an infrared light spot with coded information (similar to a projector). Then, the camera captures the structured light with coded information reflected back from the object being measured. Finally, the depth data of each point is obtained in the computing unit using a matching method. The accuracy of the projected pattern of a structured light camera decreases with increasing distance, making it suitable for short-range measurements and complementing the advantages of lidar ranging.

[0119] According to an embodiment of the present invention, it further includes:

[0120] The difference W is obtained by calculating the difference between the measurement distance of the structured light camera and the blind zone distance of the lidar directly in front;

[0121] Determine whether the difference W is greater than or equal to zero. If yes, then the robot has no blind spots; if no, adjust the installation tilt angle of the structured light camera.

[0122] It should be noted that when W≥0, it means that the blind zone distance of the LiDAR directly in front is less than or equal to the measurement distance of the structured light camera. When W<0, it means that the blind zone distance of the LiDAR directly in front is greater than the measurement distance of the structured light camera. The robot also has a blind zone. The measurement distance of the structured light camera can be increased by adjusting the tilt angle of the structured light camera until W≥0.

[0123] Figure 2 This diagram illustrates the working principle of complementary field of view between lidar and structured light camera;

[0124] like Figure 2 The diagram shows the installation positions of the LiDAR and the structured light camera, as well as their respective visible areas. The LiDAR is installed horizontally, while the structured light camera is installed at an angle. The LiDAR is responsible for mid-to-long-range measurements, while the structured light camera is responsible for short-range measurements. The two complement each other to ensure that the measurement blind spot of the robot body is minimized.

[0125] Figure 3 The blind zone analysis diagram of the lidar is shown.

[0126] like Figure 3As shown, when the horizontal field of view of the lidar is θ = 30 degrees and the installation height is H, then the blind zone distance d directly in front is: d = H / tan15°

[0127] Figure 4 A schematic diagram showing the installation angle and measurement distance of the structured light camera is provided.

[0128] like Figure 4 As shown, when the vertical field of view β of the structured light camera is 57° and the installation height is h, the measurement distance s is s = h / tan(90-57-α)°.

[0129] Figure 5 A schematic diagram of constructing a calibration board with a common viewing area is shown.

[0130] like Figure 5 As shown, since the two sensors do not share a common viewing area in the measurement space, this embodiment of the invention constructs an external parameter calibration board suitable for such sensors without a common viewing area by using an upper blank area plus a bottom checkerboard area on the basis of a common checkerboard calibration board. At the same time that the structured light camera can measure the checkerboard, the lidar can also measure the calibration board that belongs to the same plane.

[0131] Figure 6 A schematic diagram showing the relationship between the lidar coordinate system, the RGB camera coordinate system, and the calibration board coordinate system is presented.

[0132] like Figure 6 As shown, the point measured by the lidar falls on the blank upper half of the calibration board, while the checkerboard on the calibration board is within the field of view of the RGB camera. Although the two sensors observe different parts, they belong to the same calibration board plane. The external parameter relationship between the two can be indirectly obtained by using planar constraints.

[0133] Figure 7 A block diagram of the extrinsic parameter calibration system for the lidar and blind spot structured light camera of the present invention is shown.

[0134] like Figure 7 As shown, the second aspect of the present invention provides an extrinsic parameter calibration system 7 for a lidar and a blind-filling structured light camera, including a memory 71 and a processor 72. The memory stores an extrinsic parameter calibration method program for the lidar and blind-filling structured light camera. When the processor executes the extrinsic parameter calibration method program for the lidar and blind-filling structured light camera, it performs the following steps:

[0135] Acquire information about the points where the lidar falls on the blank area of ​​the calibration board, the corner points of the checkerboard, the laser data, and the checkerboard image data with the corresponding timestamp.

[0136] Based on the corner point information of the chessboard, the pose information of the RGB camera and the calibration board is obtained;

[0137] Based on the plane constraints of the preset calibration board, the closed-form initial value information of the extrinsic parameters of the lidar and the RGB camera is obtained according to the information of the point of the lidar falling on the blank area of ​​the preset calibration board and the pose information of the RGB camera and the calibration board.

[0138] Based on the preset algorithm, the external parameter information of the lidar and RGB camera is obtained according to the laser data, the corresponding timestamp chessboard image data and initial values.

[0139] Based on the extrinsic information of the LiDAR and RGB camera, and the preset relative relationship information between the RGB sensor and the structured light depth sensor, the extrinsic relationship information of the LiDAR and structured light camera is obtained.

[0140] It should be noted that the calibration board consists of a blank area and a checkerboard area, both belonging to the same plane. The points measured by the lidar fall on the upper half of the blank area of ​​the calibration board, while the checkerboard area is within the field of view of the RGB camera. Although the two sensors observe different parts, they belong to the same calibration board plane; therefore, the extrinsic parameter relationship between the two can be indirectly obtained using planar constraints. The structured light camera includes an RGB sensor, a depth sensor, and a projector sensor. The relative relationships of these three sensors are known, thus obtaining the six-degree-of-freedom relative spatial pose relationship between the lidar coordinate system and the depth sensor data, i.e., the extrinsic parameter relationship between the lidar and the structured light camera.

[0141] According to an embodiment of the present invention, it further includes:

[0142] The structured light camera is set as the reference point, and coordinate systems are set separately for different sensors, namely the RGB sensor coordinate system. Depth sensor coordinate system and the projector coordinate system

[0143] Set the lidar as the reference point and define the lidar coordinate system T. lidar ;

[0144] Using the preset calibration plate as the reference point, set the calibration plate coordinate system T. board .

[0145] It should be noted that a structured light camera comprises three coordinate systems. These represent the coordinate systems of the RGB sensor, depth sensor, and projector, respectively. The relative relationships between these three sensors are pre-defined and known. For the LiDAR, there is one coordinate system: the LiDAR coordinate system T. lidar ; Calibration plate coordinate system T boardThe six-degree-of-freedom relative spatial pose relationship between the aforementioned lidar coordinate system and the depth sensor data is the parameter that the calibration method described in this patent ultimately requires.

[0146] According to an embodiment of the present invention, the initial closed-form solution of the extrinsic parameters of the lidar and the RGB camera is specifically as follows:

[0147] Set the relative relationship of the extrinsic parameters between the lidar coordinate system and the RGB camera coordinate system as follows: in Indicates rotation, Let represent translation. The two coordinate systems together form a six-degree-of-freedom pose. The transformation relationship from the camera coordinate system to the lidar coordinate system is then:

[0148] Based on the RGB camera coordinate system, the plane of the calibration plate can be parameterized as π. c =[n c ,d], where n c Let represent the direction of the plane, and d represent the intercept of the plane equation. Then, the equation representing a 3D point on the calibration plate plane in the camera coordinate system is: Where (1) represents the equation number;

[0149] Based on the relative extrinsic parameters of the LiDAR and the RGB camera, and the equations representing the three-dimensional points on the calibration board in the camera coordinate system, we obtain... (2) Number the equation;

[0150] Set the coordinates of the next measurement point in the lidar coordinate system to P. l = [x,y,z], its representation in the camera coordinate system is as follows: The representation of matrix multiplication is as follows:

[0151] Where (3) represents the corresponding equation number;

[0152] Based on equations (2) and (3) above, we obtain Where (4) represents the number of the corresponding equation;

[0153] Based on the preset algorithm, multiple sets of LiDAR and corresponding timestamp chessboard image data are substituted into equation (4) to obtain the closed-form initial value argmin of the corresponding LiDAR and RGB camera extrinsic parameters.

[0154] It should be noted that the preset algorithm is a nonlinear least squares (DLT) algorithm. Based on equation (4), the DLT algorithm is used to estimate the 3x3 H matrix (a total of 9 parameters), which becomes a linear least squares problem. The lidar used in this embodiment is a 3D lidar. One frame of lidar can provide three effective constraints. The plane equation can be determined if three points fall on the calibration plate plane. Only three frames of lidar and the corresponding timestamp of the checkerboard image data are needed to solve the closed-form solution. This value is set as the initial value argmin of the closed-form solution of the lidar and RGB camera extrinsic parameters.

[0155] According to an embodiment of the present invention, the extrinsic parameters of the lidar and the RGB camera are specifically as follows:

[0156] Let N be the laser point that the i-th frame lidar falls on the calibration board plane. i The equation of the calibration board plane corresponding to the i-th laser frame in RGB camera coordinates is expressed as follows: Its optimization model is expressed as:

[0157]

[0158] The i-th frame of lidar data belongs to the i-th group of data in N groups of lidar data and checkerboard image data.

[0159] It should be noted that the i-th data in the N sets of data includes the i-th frame of LiDAR data and the i-th frame of image data corresponding to the timestamp. The laser point of the i-th frame of LiDAR falling on the calibration board plane is set as N. i The equation of the calibration plate plane corresponding to the i-th laser frame is represented in camera coordinates as follows: The expression for the optimization model is as follows:

[0160]

[0161]

[0162] The above formula represents the calculation of the distance between N laser points on the calibration board plane of the LiDAR point cloud in the i-th frame, based on the relative extrinsic parameters between the current checkerboard and the camera (obtainable from checkerboard calibration). This calculation is performed for one frame, and then accumulated across all N frames. The resulting sum of squared distances is the target for optimization. The minimum sum of squared distances indicates that the LiDAR points on the calibration board plane are well-positioned on the checkerboard plane. Specifically, in the optimization process, the Jacobian error function can be derived using Lie algebra methods, while optimization can employ nonlinear or Gauss-Newton optimization theories, ultimately determining the optimal solution. The optimal solution is set as the extrinsic parameters of the LiDAR and the RGB camera.

[0163] According to an embodiment of the present invention, it further includes:

[0164] Install the lidar horizontally;

[0165] Set the installation height of the lidar to H and the horizontal field of view to θ.

[0166] Let d be the blind zone distance of the lidar directly in front, and its formula is:

[0167] It should be noted that the ranging principle of lidar is based on time-of-flight ranging, calculating the distance based on the emission and reception time of the laser and the speed of light. This embodiment of the invention uses a 3D mechanical lidar as an example. The lidar can obtain measurement data in both the horizontal and vertical directions. A Cartesian coordinate system is established with the center of the lidar to obtain the coordinates of each point. The coordinates of all detected data points form a point cloud.

[0168] According to an embodiment of the present invention, it further includes:

[0169] Set the vertical field of view of the structured light camera to β and the tilt angle to α.

[0170] Set the installation height of the structured light camera to h;

[0171] Let the measurement distance of the structured light camera be s, and its formula is: s=h / tan(90-β-α)°, where 0<α<90°.

[0172] It's important to explain the ranging principle of structured light cameras: A structured light camera generally consists of two parts: a projector and a camera. The projector actively projects an infrared light spot with coded information (similar to a projector). Then, the camera captures the structured light with coded information reflected back from the object being measured. Finally, the depth data of each point is obtained in the computing unit using a matching method. The accuracy of the projected pattern of a structured light camera decreases with increasing distance, making it suitable for short-range measurements and complementing the advantages of lidar ranging.

[0173] According to an embodiment of the present invention, it further includes:

[0174] The difference W is obtained by calculating the difference between the measurement distance of the structured light camera and the blind zone distance of the lidar directly in front;

[0175] Determine whether the difference W is greater than or equal to zero. If yes, then the robot has no blind spots; if no, adjust the installation tilt angle of the structured light camera.

[0176] It should be noted that when W≥0, it means that the blind zone distance of the LiDAR directly in front is less than or equal to the measurement distance of the structured light camera. When W<0, it means that the blind zone distance of the LiDAR directly in front is greater than the measurement distance of the structured light camera. The robot also has a blind zone. The measurement distance of the structured light camera can be increased by adjusting the tilt angle of the structured light camera until W≥0.

[0177] A third aspect of the present invention provides a computer medium storing a program for extrinsic parameter calibration of a lidar and a blind spot structured light camera. When the program for extrinsic parameter calibration of a lidar and a blind spot structured light camera is executed by a processor, it implements the steps of the extrinsic parameter calibration method of the lidar and blind spot structured light camera as described in any of the preceding claims.

[0178] This invention discloses a method, system, and medium for extrinsic parameter calibration of a LiDAR and a structured light camera with blind spot compensation. Based on the planar constraints of a calibration board, it determines the pose relationship between the RGB camera and the calibration board, and the pose relationship between the LiDAR and the calibration board, thereby obtaining the pose relationship between the LiDAR and the RGB camera. Then, based on the relative relationship between the RGB camera and the structured light camera, it obtains the extrinsic parameter relationship between the LiDAR and the structured light camera. By determining the extrinsic parameter relationship between the LiDAR and the structured light camera, this invention transforms the data detected by the LiDAR and the structured light camera into a common coordinate system, obtaining information about the robot's entire field of view in the same dimension, which facilitates global strategy and judgment. In addition, this invention determines the blind zone distance of the LiDAR through the height and viewing angle of the LiDAR; and determines the measurement distance of the structured light camera through the viewing angle, tilt angle, and height of the structured light camera, adjusting the blind zone distance or the measurement distance, thus solving the robot's blind zone problem.

[0179] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0180] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0181] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0182] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0183] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

Claims

1. A method for calibrating the extrinsic parameters of a lidar and a structured light camera with blind spot compensation, characterized in that, include: Acquire information about the points where the lidar falls on the blank area of ​​the calibration board, the corner points of the checkerboard, the laser data, and the checkerboard image data with the corresponding timestamp. Based on the corner point information of the chessboard, the pose information of the RGB camera and the calibration board is obtained; Based on the pre-defined calibration board plane constraints, and according to the information of the points where the LiDAR falls on the blank area of ​​the pre-defined calibration board and the pose information of the RGB camera and the calibration board, the initial closed-form solution of the extrinsic parameters of the LiDAR and the RGB camera is obtained; specifically, the relative relationship of the extrinsic parameters from the LiDAR coordinate system to the RGB camera coordinate system is set as follows: ,in Indicates rotation, Let represent translation. Together, they form a six-degree-of-freedom pose. The transformation relationship from the camera coordinate system to the lidar coordinate system is: Based on the RGB camera coordinate system, the plane of the calibration board is parameterizable as... In the formula Let represent the direction of the plane, and d represent the intercept of the plane equation. Then, the equation representing a 3D point on the calibration plate plane in the camera coordinate system is: Where (1) represents the equation number; based on the relative relationship between the external parameters of the lidar and the RGB camera and the equation representing the three-dimensional points on the calibration board in the camera coordinate system, we obtain Where (2) represents the equation number; set the coordinates of the next measurement point in the lidar coordinate system as Its representation in the camera coordinate system is as follows The representation of matrix multiplication is as follows: Where (3) represents the corresponding equation number; according to equations (2) and (3) above, we obtain , where (4) represents the number of the corresponding equation; Based on the preset algorithm, substitute multiple sets of lidar and corresponding timestamp chessboard image data into equation (4) to obtain the closed-form initial value argmin of the external parameters of the lidar and RGB camera. Based on the preset algorithm, the external parameter information of the lidar and RGB camera is obtained according to the laser data, the corresponding timestamp chessboard image data and initial values. Based on the extrinsic information of the LiDAR and RGB camera and the preset relative relationship information between the RGB sensor and the structured light depth sensor, the extrinsic relationship information of the LiDAR and the structured light camera is obtained. The calibration board consists of a blank area and a checkerboard area, and both belong to the same plane.

2. The extrinsic parameter calibration method for lidar and blind-filling structured light camera according to claim 1, characterized in that, Also includes: The structured light camera is set as the reference point, and coordinate systems are set separately for different sensors, namely the RGB sensor coordinate system. Depth sensor coordinate system and the projector coordinate system ; Set the lidar as the reference point and define the lidar coordinate system. ; Use the preset calibration plate as the reference point to set the calibration plate coordinate system. .

3. The extrinsic parameter calibration method for lidar and blind-filling structured light camera according to claim 1, characterized in that, The extrinsic parameters of the LiDAR and RGB camera are as follows: Let the laser point that the i-th frame lidar falls on the calibration plate plane be set as... The equation of the calibration board plane corresponding to the i-th laser frame in RGB camera coordinates is expressed as follows: Its optimization model is expressed as: ; The i-th frame of lidar data belongs to the i-th group of data in N groups of lidar data and checkerboard image data.

4. The extrinsic parameter calibration method for lidar and blind-filling structured light camera according to claim 1, characterized in that, Also includes: Install the lidar horizontally; Set the installation height of the lidar to H, and the horizontal field of view to [value missing]. ; Let d be the blind zone distance of the lidar directly in front, and its formula is: .

5. The extrinsic parameter calibration method for a lidar and a blind-filling structured light camera according to claim 1, characterized in that, Also includes: Set the vertical field of view of the structured light camera to The tilt angle is set to ; Set the installation height of the structured light camera to h; Let the measurement distance of the structured light camera be s, and its formula is: ,in .

6. An extrinsic parameter calibration system for a lidar and blind-filling structured light camera, characterized in that, The system includes a memory and a processor. The memory stores a program for extrinsic parameter calibration of the lidar and the blind-filling structured light camera. When the processor executes the extrinsic parameter calibration program of the lidar and the blind-filling structured light camera, it performs the following steps: Acquire information about the points where the lidar falls on the blank area of ​​the calibration board, the corner points of the checkerboard, the laser data, and the checkerboard image data with the corresponding timestamp. Based on the corner point information of the chessboard, the pose information of the RGB camera and the calibration board is obtained; Based on the pre-defined calibration board plane constraints, and according to the information of the points where the LiDAR falls on the blank area of ​​the pre-defined calibration board and the pose information of the RGB camera and the calibration board, the initial closed-form solution of the extrinsic parameters of the LiDAR and the RGB camera is obtained; specifically, the relative relationship of the extrinsic parameters from the LiDAR coordinate system to the RGB camera coordinate system is set as follows: ,in Indicates rotation, Let represent translation. Together, they form a six-degree-of-freedom pose. The transformation relationship from the camera coordinate system to the lidar coordinate system is: Based on the RGB camera coordinate system, the plane of the calibration board is parameterizable as... In the formula Let represent the direction of the plane, and d represent the intercept of the plane equation. Then, the equation representing a 3D point on the calibration plate plane in the camera coordinate system is: Where (1) represents the equation number; based on the relative relationship between the external parameters of the lidar and the RGB camera and the equation representing the three-dimensional points on the calibration board in the camera coordinate system, we obtain Where (2) represents the equation number; set the coordinates of the next measurement point in the lidar coordinate system as Its representation in the camera coordinate system is as follows The representation of matrix multiplication is as follows: Where (3) represents the corresponding equation number; according to equations (2) and (3) above, we obtain , where (4) represents the number of the corresponding equation; based on the preset algorithm, multiple sets of lidar and corresponding timestamp chessboard image data are substituted into equation (4) to obtain the closed-form initial value argmin of the external parameters of the corresponding lidar and RGB camera; Based on the preset algorithm, the external parameter information of the lidar and RGB camera is obtained according to the laser data, the corresponding timestamp chessboard image data and initial values. Based on the extrinsic information of the LiDAR and RGB camera and the preset relative relationship information between the RGB sensor and the structured light depth sensor, the extrinsic relationship information of the LiDAR and the structured light camera is obtained. The calibration board consists of a blank area and a checkerboard area, and both belong to the same plane.

7. The extrinsic parameter calibration system for a lidar and blind-filling structured light camera according to claim 6, characterized in that, Also includes: The structured light camera is set as the reference point, and coordinate systems are set separately for different sensors, namely the RGB sensor coordinate system. Depth sensor coordinate system and the projector coordinate system ; Set the lidar as the reference point and define the lidar coordinate system. ; Use the preset calibration plate as the reference point to set the calibration plate coordinate system. .

8. A computer medium, characterized in that, The computer medium stores a extrinsic parameter calibration method program for a lidar and a blind spot structured light camera. When the extrinsic parameter calibration method program for a lidar and a blind spot structured light camera is executed by the processor, it implements the steps of the extrinsic parameter calibration method for a lidar and a blind spot structured light camera as described in any one of claims 1 to 5.

Images