Time-space calibration method and device for calibration target object, laser radar and camera

Abstract

The invention belongs to the technical field of intelligent traffic, and particularly relates to a time-space calibration method and device for calibrating a target object, a laser radar and a camera. The method comprises the following steps: respectively acquiring image data and point cloud data of a target area in which a calibration target object is arranged by using a camera and a laser radar, target detection is performed on the calibration target object in the image data and the point cloud data, and position coordinates of the calibration target object in a camera coordinate system and a laser radar coordinate system are acquired; respectively forming an image track and a point cloud track by a series of three-dimensional image points and point cloud points obtained by continuously performing target detection on the calibration target object, and determining external parameters of a laser radar and a camera based on the similarity of the image track and the point cloud track; and track point matching is carried out on the image track and the point cloud track based on the external parameters, and time migration of the camera and the laser radar is determined based on the matched track points.

Description

Technical Field

[0001] The present disclosure belongs to the field of intelligent transportation technology, and specifically relates to a method and device for calibrating a target object and a time-space calibration of a laser radar and a camera. Background Art

[0002] In the fields of autonomous driving and smart transportation, multi-sensor fusion strategies based on LiDAR and cameras are widely used in roadside perception systems. An important prerequisite for successful multi-sensor fusion is spatiotemporal calibration, that is, determining the 6-DOF spatial transformation parameters (i.e., external parameters) between different sensor coordinate systems and determining the time offset between different sensor data.

[0003] However, existing multi-sensor spatiotemporal calibration methods are basically designed for vehicle-mounted sensors and perform poorly in roadside scenarios. For example, roadside sensors usually have a longer perception range than vehicle-side sensors, often exceeding 100 meters. At long distances, the calibration targets are very small in the image and sparse in the point cloud, making them difficult to detect. Roadside scenes contain many planar features, such as traffic signs, which make traditional planar calibration targets difficult to distinguish in point clouds. Therefore, traditional algorithms usually require manual intervention, such as setting ROI (Region of Interest). In addition, the installation locations of roadside sensors vary greatly, and it is impossible to estimate the initial external parameters between sensors through measurement. Summary of the invention

[0004] The disclosed embodiment proposes a spatiotemporal calibration scheme for a laser radar and a camera, so as to provide a spatiotemporal calibration method for a laser radar and a camera suitable for roadside scenes.

[0005] A first aspect of an embodiment of the present disclosure provides a calibration target object, which is set in a target area and includes a circular target with high reflectivity and at least one positioning two-dimensional code, wherein:

[0006] The circular target is used as a target for target detection in the point cloud data of the target area to obtain the coordinates of the calibration target in the laser radar coordinate system;

[0007] The positioning two-dimensional code is used as a target for target detection in the image data of the target area to obtain the coordinates of the calibration target object in the camera coordinate system.

[0008] A second aspect of the embodiments of the present disclosure provides a method for spatiotemporal calibration of a laser radar and a camera, including:

[0009] Using a camera and a laser radar, respectively, image data and point cloud data of a target area of ​​a calibration target object as described in the first aspect of the present disclosure are acquired, target detection is performed on the calibration target object in the image data and the point cloud data, and position coordinates of the calibration target object in a camera coordinate system and a laser radar coordinate system are acquired;

[0010] A series of three-dimensional image points and point cloud points obtained by continuously performing target detection on the calibration target object respectively form an image trajectory and a point cloud trajectory, and determine the external parameters of the laser radar and the camera based on the similarity between the image trajectory and the point cloud trajectory;

[0011] The image trajectory and the point cloud trajectory are matched based on the external parameters, and the time offset of the camera and the laser radar is determined based on the matched trajectory points.

[0012] In some embodiments of the present disclosure, performing target detection on the calibration target in the point cloud data and obtaining the position coordinates of the calibration target in the laser radar coordinate system includes:

[0013] Acquire a first point cloud of the target area, filter points whose intensities are lower than a preset threshold in the first point cloud, and generate a second point cloud;

[0014] In some embodiments of the present disclosure, points corresponding to the circular target are respectively identified in the clustering results of the second point cloud and the first point cloud based on the properties of the circular target and the calibration target object as the third point cloud and the fourth point cloud;

[0015] Remove noise points from the third point cloud and fit a plane, calculate the center of mass and the normal vector of the plane, and determine the transformation matrix for mapping the point cloud of the calibration target object to the XY plane of the three-dimensional coordinate system based on the center of mass and the normal vector; obtain the coordinates of the center of the calibration target object in the three-dimensional coordinate system, multiply the coordinates by the inverse of the transformation matrix to obtain the position coordinates of the calibration target object in the lidar coordinate system, wherein the Z axis of the XY plane is 0.

[0016] In some embodiments of the present disclosure, obtaining the coordinates of the center of the calibration target object in a three-dimensional coordinate system includes:

[0017] In the three-dimensional coordinate system, a point that is not less than a preset distance from the origin is selected as the center of the circle, and a circle is drawn with the radius of the circular target as the radius, and the position of the center of the circle is adjusted so that the circle contains as many of the third point cloud as possible and the points in the fourth point cloud that do not belong to the third point cloud are as many as possible outside the circle;

[0018] Get the X and Y coordinates of the center of the circle, and set the Z coordinate to 0.

[0019] In some embodiments of the present disclosure, determining the external parameters of the laser radar and the camera based on the similarity between the image trajectory and the point cloud trajectory includes:

[0020] Acquire initial values ​​of external parameters of the laser radar and the camera based on minimizing the difference between the image trajectory and the point cloud trajectory;

[0021] Then, Gaussian mixture models are constructed for the image trajectory and the point cloud trajectory respectively, and the final extrinsic parameter is determined based on the statistical similarity between the image trajectory and the point cloud trajectory and the Gaussian mixture model.

[0022] In some embodiments of the present disclosure, the obtaining of initial values ​​of external parameters of the laser radar and the camera based on minimizing the difference between the image trajectory and the point cloud trajectory includes:

[0023] A cost function is expressed using the extrinsic parameters, a plurality of the image trajectories and the trajectory points of the point cloud trajectory;

[0024] Calculate the external parameters when the cost function is the minimum, the external parameters are the initial values ​​of the external parameters of the laser radar and the camera, wherein the cost function is:

[0025]

[0026] Among them, E is the cost function, are the starting point and the stationary point of the point cloud trajectory, are the starting point and the stationary point of the image trajectory, N represents the number of trajectories, R and t represent the rotation and translation parts of the extrinsic parameters respectively, and i is the index of the trajectory point.

[0027] In some embodiments of the present disclosure, constructing Gaussian mixture models for the image trajectory and the point cloud trajectory respectively includes:

[0028] Respectively representing the image trajectory and the point cloud trajectory as weighted sums of Gaussian density functions to construct a first Gaussian mixture model and a second Gaussian mixture model in a three-dimensional space, wherein the number of Gaussian components in the Gaussian mixture model is equal to the number of trajectory points in the trajectory, the mean of each Gaussian component is the coordinate of the corresponding trajectory point, and all the Gaussian components have the same weight and covariance matrix;

[0029] The image trajectory and the point cloud trajectory are respectively projected onto an image plane to form a two-dimensional image trajectory and a two-dimensional point cloud trajectory, and the two-dimensional image trajectory and the two-dimensional point cloud trajectory are represented as a weighted sum of Gaussian density functions to respectively construct a third Gaussian mixture model and a fourth Gaussian mixture model in two-dimensional space.

[0030] In some embodiments of the present disclosure, determining the final external parameter based on the statistical similarity between the image trajectory and the point cloud trajectory and a Gaussian mixture model includes:

[0031] Using the external parameter to express the statistical similarity between the first Gaussian mixture model and the second Gaussian mixture model, to obtain a first distance;

[0032] Using the external parameter to express the statistical similarity between the third Gaussian mixture model and the fourth Gaussian mixture model, to obtain a second distance;

[0033] Determine the external parameter when the sum of the first distance and the second distance is minimized.

[0034] In some embodiments of the present disclosure, performing trajectory point matching on the image trajectory and the point cloud trajectory based on the external parameter includes:

[0035] Acquire the trajectory point whose speed is greater than a preset threshold in the point cloud trajectory;

[0036] The trajectory points are transformed into a camera coordinate system according to the external parameters, and the closest point matching is performed with the trajectory points constituting the image trajectory.

[0037] A third aspect of the embodiments of the present disclosure provides a spatiotemporal calibration device for a laser radar and a camera, comprising:

[0038] an acquisition module, configured to respectively acquire image data and point cloud data of a target area of ​​a calibration target object as described in the first aspect of the embodiment of the present disclosure by using a camera and a laser radar, perform target detection on the calibration target object in the image data and the point cloud data, and acquire position coordinates of the calibration target object in a camera coordinate system and a laser radar coordinate system;

[0039] A first determination module is used to form an image trajectory and a point cloud trajectory respectively from a series of three-dimensional image points and point cloud points obtained by continuously performing target detection on the calibration target object, and determine the external parameters of the laser radar and the camera based on the similarity between the image trajectory and the point cloud trajectory;

[0040] The second determination module is used to match the image trajectory and the point cloud trajectory based on the external parameters, and determine the time offset of the camera and the laser radar based on the matched trajectory points.

[0041] In summary, the spatiotemporal calibration method and device of the laser radar and camera provided in each embodiment of the present disclosure realizes automatic positioning and detection by selecting a calibration target that is more suitable for long-distance target detection, so as to be more suitable for roadside scenes. Specifically, for the camera, AprilTag is selected as the target, which is more suitable for long-distance detection, and can also obtain good detection accuracy in roadside scenes, providing accurate 3D position. For the laser radar, a circular target with high reflectivity is selected according to its physical characteristics. It is not easy to lose information in the point cloud, and the features are obvious and easy to identify, so that automatic positioning and detection can be realized without human intervention. At the same time, in the external parameter calibration stage, after the initial estimation, the external parameters are further optimized with the help of the Gaussian mixture model to minimize the influence of noise and outliers and make the results more stable. In time synchronization, the matching errors caused by low-speed targets are removed to improve the accuracy. In summary, each embodiment of the present disclosure provides a spatiotemporal calibration method suitable for roadside scenes with robustness, stability and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The features and advantages of the present disclosure will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present disclosure in any way. In the accompanying drawings:

[0043] Figure 1 is a schematic diagram of a computer system to which the present disclosure is applicable;

[0044] Figure 2 is a schematic diagram of a calibration target object according to an embodiment of the present disclosure;

[0045] Figure 3 It is an algorithm framework of a spatiotemporal calibration algorithm of a lidar and a camera proposed in the present disclosure;

[0046] Figure 4 is a flow chart of a method for spatiotemporal calibration of a laser radar and a camera according to some embodiments of the present disclosure;

[0047] Figure 5 is an example of a point cloud for calibrating a target object in one embodiment of the present disclosure, wherein there is a large deviation between the center and the centroid of the point cloud;

[0048] Figure 6 Yes Figure 2 Schematic diagram of detecting and locating the calibration target shown;

[0049] Figure 7 The initial estimated value and optimized value of the external parameter are respectively Figure 6 The acquired point cloud is projected onto an instance of the image;

[0050] Figure 8 This is an example of projecting a point cloud onto an image before and after spatiotemporal synchronization;

[0051] Fig. 9 It is a schematic diagram of a spatiotemporal calibration device for a laser radar and a camera according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0052] In the detailed description below, many specific details of the present disclosure are set forth by way of example in order to provide a thorough understanding of the relevant disclosure. However, it will be apparent to one of ordinary skill in the art that the present disclosure can be implemented without these details. It should be understood that the use of the terms "system," "device," "unit," and / or "module" in the present disclosure is a method for distinguishing different parts, elements, parts, or assemblies at different levels in a sequential arrangement. However, these terms may be replaced by other expressions if other expressions can achieve the same purpose.

[0053] It should be understood that when a device, unit or module is referred to as being "on," "connected to," or "coupled to" another device, unit or module, it may be directly on, connected to, coupled to, or communicating with other devices, units or modules, or there may be intermediate devices, units or modules, unless the context clearly indicates an exception. For example, the term "and / or" as used in this disclosure includes any and all combinations of one or more of the relevant listed items.

[0054] The terms used in this disclosure are only for describing specific embodiments and are not intended to limit the scope of this disclosure. As shown in the specification and claims of this disclosure, unless the context clearly indicates an exception, the words "one", "a kind" and / or "the" do not specifically refer to the singular, but may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of clearly identified features, wholes, steps, operations, elements and / or components, and such expressions do not constitute an exclusive list, and other features, wholes, steps, operations, elements and / or components may also be included.

[0055] These and other features and characteristics, methods of operation, functions of related elements of the structure, combinations of parts, and economies of manufacture of the present disclosure may be better understood with reference to the following description and drawings, which form a part of the specification. However, it is to be clearly understood that the drawings are for illustration and description purposes only and are not intended to limit the scope of protection of the present disclosure. It is to be understood that the drawings are not drawn to scale.

[0056] Various structural diagrams are used in this disclosure to illustrate various variations of the embodiments of the present disclosure. It should be understood that the above or below structures are not intended to limit the present disclosure. The scope of protection of the present disclosure shall be subject to the claims.

[0057] Figure 1 is a schematic diagram of a computer system to which the present disclosure is applicable. Figure 1 The computer system shown includes a calibration server connected to the laser radar and image sensor data set on the roadside, and a calibration target for target identification is also set on the road. The calibration server calibrates the laser radar and the image sensor based on the sensor data of the calibration target obtained by the laser radar and the image sensor.

[0058] The road is a road where an intelligent transportation system is deployed, such as a laser radar and an image sensor are deployed on the roadside. The laser radar is used to obtain point cloud data of the target area of ​​the road, and the image sensor is used to obtain image data of the target area. The image sensor can be various monocular, binocular cameras or video cameras. A calibration target is set on the road, and the calibration server performs spatiotemporal calibration of the laser radar and the image sensor based on the sensor data of the calibration target. The calibration server can be any of a stand-alone, cluster or distributed server.

[0059] In roadside scenarios, multi-sensor calibration presents some unique challenges. Roadside sensors typically have a longer sensing range than vehicle-side sensors, often exceeding 100 meters. At long distances, the calibration targets are small in the image and sparse in the point cloud, making them difficult to detect. Roadside scenes contain many planar features, such as traffic signs, which make traditional planar calibration targets difficult to distinguish in point clouds. Therefore, traditional algorithms usually require manual intervention, such as setting ROI (Region of Interest). In addition, the installation positions of roadside sensors vary greatly, and it is impossible to estimate the initial external parameters between sensors by measurement. An effective method for initial estimation of external parameters is needed. Existing calibration methods are basically designed for vehicle-mounted sensors and perform poorly in roadside scenarios.

[0060] In view of the above problems, an embodiment of the present disclosure proposes a calibration target object that can be easily detected by a laser radar and an image sensor, and whose features can be easily extracted and located, such as Figure 2 The calibration target shown has a high-reflectivity circular target in the middle and is surrounded by 4 AprilTags. Circular targets in point clouds are not prone to losing edge information and are easier to detect than rectangles. In order to fully utilize point cloud data and reduce human intervention, the present disclosure designs a high-reflectivity circular target on the calibration target to achieve accurate target detection in the point cloud. For the camera, the present disclosure sets 4 AprilTags on the calibration target, which is more suitable for long-distance roadside scenes. This design enables the monocular camera to accurately obtain the 3D position of the target, while the lidar can automatically detect and locate the target in the environment.

[0061] Based on a uniquely designed calibration target, the present disclosure proposes a method for achieving spatiotemporal synchronization of a camera and a lidar without relying on any environmental features. The method detects the calibration target through a lidar and a camera respectively and tracks the trajectory. By aligning the trajectories detected by different sensors, the spatiotemporal calibration parameters between roadside sensors are initially estimated and optimized to obtain accurate results. The algorithm framework is as follows: Figure 3 shown.

[0062] Figure 4 is a flow chart of a method for spatiotemporal calibration of a laser radar and a camera according to some embodiments of the present disclosure. In some embodiments, the spatiotemporal calibration method of the laser radar and the camera is Figure 1 The calibration server shown is executed, and the spatiotemporal calibration method of the laser radar and the camera includes the following steps:

[0063] S410, using a camera and a lidar to respectively obtain image data and point cloud data of a target area of ​​a calibration target object as described in claim 1, performing target detection on the calibration target object in the image data and the point cloud data, and obtaining position coordinates of the calibration target object in a camera coordinate system and a lidar coordinate system.

[0064] The first step of target-based calibration is for each sensor to identify the calibration target and obtain the detection result in its own coordinate system. The output of this step is a series of 3D points, which represent the position of the center point of the calibration target in the coordinate system of different sensors.

[0065] For camera data, by detecting the corner points of AprilTag in the image, a series of pixel coordinates can be obtained: According to the known size of the calibration target, the coordinates of these corner points in the calibration target coordinate system can be easily obtained: according to and Combined with the camera intrinsic parameters, the PnP algorithm can be used to obtain the transformation parameters from the calibration target coordinate system to the camera coordinate system.

[0066] Considering that the calibration target is a plane, it means The points in are all coplanar, so the present disclosure uses the IPPE algorithm for pose estimation. According to the obtained transformation matrix, the coordinates of the center point of the calibration target in the camera coordinate system can be obtained.

[0067] For LiDAR data, the original point cloud is recorded as Considering that the circular part on the calibration target has a high reflectivity, the present disclosure first filters out the points with low intensity in the point cloud according to a specific threshold, and the filtered points are recorded as Respectively and Perform DBSCAN clustering to obtain the corresponding clustering results and

[0068] Since the calibration targets have unique features in the point cloud, they can be automatically identified from these clusters. First, and The ratio of the number of points should be approximately equal to the area ratio of the high reflectivity target to the entire calibration target. The size of the reflectivity target should match that of the target, which can be achieved by Finally, the point cloud It should present a circular plane distribution, which can be judged by the eigenvalue. Based on the above characteristics, the point cloud representing the calibration target can be identified from the obtained cluster. Recorded as And the corresponding Recorded as

[0069] Considering that the vertical angular resolution of the LiDAR may be uneven, The distribution is uneven and the point cloud may contain noise, so we directly It is not appropriate to use the center of gravity of as the center point of the calibration target, such as Figure 5 shown. Figure 5 The resolution of the point cloud is uneven. The square points are the point clouds corresponding to the calibration target, the circular points represent the actual center of the calibration target, and the triangular points represent the centroid of the point cloud. There is a large deviation between the two. Therefore, in order to detect the center of the circular target, the point cloud of the entire calibration target is first Apply the RANSAC plane fitting algorithm to remove noise points. Then, calculate the normal vector of the plane. Use the normal vector and The center of mass of the target can be calculated to map the point cloud of the calibration target to the XY plane, and ensure that the Z axis is approximately 0. The far point of the coordinate system is used as the initial estimate of the center of the circle and a circle is drawn according to the known radius, taking into account It should be inside the circle. Except The part should belong to the outside of the circle. According to this rule, the coordinates of the center of the circle can be iteratively optimized. Finally, the optimized X and Y coordinates plus the Z coordinate set to 0 constitute the coordinates of the center of the calibration target. Applying the inverse transformation of the previous mapping to this coordinate, the coordinates of the center of the calibration target in the lidar coordinate system can be obtained.

[0070] right Figure 2 The schematic diagram of the calibration target object for detection and positioning is shown in Figure 6 .

[0071] S420, forming an image trajectory and a point cloud trajectory respectively from a series of three-dimensional image points and point cloud points obtained by continuously performing target detection on the calibration target object, and determining the external parameters of the laser radar and the camera based on the similarity between the image trajectory and the point cloud trajectory.

[0072] The movement of the calibration target can generate multiple trajectories. Although there may be time differences, the trajectories themselves are consistent, and they are all observations of the same target. Therefore, when calibrating external parameters, this disclosure only focuses on the coordinates of the trajectory points and temporarily ignores the timestamps. Obtaining an initial estimate of the external parameters between sensors requires some certain correspondence. Therefore, this disclosure uses the static start and end points of the trajectory, which are recorded as and the camera coordinate system The present disclosure obtains an initial estimate of the external parameters by minimizing the following cost function E:

[0073]

[0074] Where N is the number of trajectories, R and t are the rotation and translation parts of the extrinsic parameters, respectively, and i is the index of the trajectory point.

[0075] The next step is to further optimize the external parameters. The present disclosure uses a Gaussian mixture model to represent discrete trajectory points as a continuous density function. Specifically, each trajectory can be represented as a weighted sum of Gaussian density functions, i.e. For 3D trajectories, represents a random variable, represents the mean, represents the covariance matrix, w i represents the weight. The purpose of doing this is twofold. First, the continuous probability density function can better handle the noise and outliers of the calibration target detection results. Second, it eliminates the requirement of nearest point matching in the optimization process and transforms the discrete optimization problem into a more tractable continuous optimization problem. A simple way to construct a Gaussian mixture model from a trajectory is: i) the number of Gaussian components is equal to the number of points in the trajectory, ii) the mean of each component is the coordinate of the corresponding trajectory point, and iii) all Gaussian components have the same weight and covariance matrix. The L2 distance is used to evaluate the statistical similarity between two Gaussian mixture models because the L2 distance has a closed form expression and is defined as:

[0076]

[0077] in, and Respectively represent the target trajectory in the laser radar coordinate system and the camera coordinate system, R and t represent the rotation and translation parts of the external parameters, respectively. Indicates the trajectory according to the external parameters R and t To transform, and represents a Gaussian mixture model. The L2 distance can be easily calculated using the following formula:

[0078]

[0079] in represents Gaussian distribution, μ1 and μ2 represent the mean of Gaussian distribution, and ∑1 and ∑2 represent the variance of Gaussian distribution.

[0080] Each trajectory can calculate an L2 distance. The trajectory points are projected onto the image plane using the following formula:

[0081]

[0082]

[0083] K represents the camera internal parameters, which consists of 4 parameters, including the focal length f x and f y , the principal point coordinates c x and c y , T represents the external parameter of the laser radar to the camera, which consists of two parts: rotation R and translation t. represents the coordinates in the image coordinate system, d i represents the depth after projection, Represents coordinates in the lidar coordinate system.

[0084] Project the image 3D trajectory points and point cloud 3D trajectory points onto the image plane to generate the corresponding image 2D trajectory points and point cloud 2D trajectory points According to the previous principles, the Gaussian mixture model is constructed in the same way, that is, in represents a random variable, represents the mean, represents the covariance matrix, w i Represents the weight. Calculate the L2 distance, expressed as:

[0085]

[0086] The final cost function E is the sum of the L2 distances of the 3D and 2D trajectories, that is

[0087] E=w 3D d 3D (R, t)+w 2D d 2D (R, t),

[0088] Among them, w3D and w 2D represents the weight, d 3D (R, t) and d 2D (R, t) represents the L2 distance between the 3D trajectory Gaussian mixture model and the 2D trajectory Gaussian mixture model, respectively. By minimizing E, the optimal external parameters R and t can be obtained.

[0089] Figure 7 These are example diagrams of projecting a point cloud onto an image based on the initial estimate and optimized value of the external parameters.

[0090] S430, performing trajectory point matching on the image trajectory and the point cloud trajectory based on the external parameters, and determining the time offset between the camera and the laser radar based on the matched trajectory points.

[0091] Once the external parameter calibration is completed, time synchronization is easy to perform. The trajectory detected by the lidar is transformed into the camera coordinate system according to the external parameters, and the closest point is matched with the trajectory detected by the camera. The time deviation between the matching trajectory points can be calculated. The mean of these deviations can be used as the time offset between the two sensors.

[0092] It is worth noting that calibrating the stationary points in the target trajectory will cause incorrect matching, resulting in inaccurate calculated time deviations. Therefore, the present disclosure calculates the speed of each point on the trajectory and matches the points whose speed is greater than a threshold.

[0093] This step can be performed both after the initial estimation and after the optimization. Generally speaking, using more accurate external parameters can lead to more accurate time synchronization results.

[0094] Figure 8 is an example of projecting a point cloud onto an image before and after spatiotemporal synchronization. Figure 8 As shown, before time-space synchronization, when the timestamps of the output data of the two sensors are not synchronized, the moving objects in the environment cannot be aligned. Using the disclosed algorithm for time synchronization and correcting the timestamps of the data can align the data of the two sensors in both space and time.

[0095] Fig. 9 is a schematic diagram of a spatiotemporal calibration device for a laser radar and a camera according to some embodiments of the present disclosure. Fig. 9 As shown, the spatiotemporal calibration device 900 for the laser radar and camera includes an acquisition module 910, a first determination module 920, and a second determination module 930. In some embodiments of the present disclosure, the spatiotemporal calibration function of the laser radar and camera is Figure 1 The calibration server is executed as shown.

[0096] An acquisition module 910 is used to respectively acquire image data and point cloud data of a target area of ​​the calibration target object as claimed in claim 1 by using a camera and a laser radar, perform target detection on the calibration target object in the image data and the point cloud data, and acquire position coordinates of the calibration target object in a camera coordinate system and a laser radar coordinate system;

[0097] A first determination module 920 is used to form an image trajectory and a point cloud trajectory respectively from a series of three-dimensional image points and point cloud points obtained by continuously performing target detection on the calibration target object, and determine the external parameters of the laser radar and the camera based on the similarity between the image trajectory and the point cloud trajectory;

[0098] The second determination module 930 is used to match the image trajectory and the point cloud trajectory based on the external parameters, and determine the time offset between the camera and the laser radar based on the matched trajectory points.

[0099] In summary, the spatiotemporal calibration method and device of the laser radar and camera provided in each embodiment of the present disclosure realizes automatic positioning and detection by selecting a calibration target that is more suitable for long-distance target detection, so as to be more suitable for roadside scenes. Specifically, for the camera, AprilTag is selected as the target, which is more suitable for long-distance detection, and can also obtain good detection accuracy in roadside scenes, providing accurate 3D position. For the laser radar, a circular target with high reflectivity is selected according to its physical characteristics. It is not easy to lose information in the point cloud, and the features are obvious and easy to identify, so that automatic positioning and detection can be realized without human intervention. At the same time, in the external parameter calibration stage, after the initial estimation, the external parameters are further optimized with the help of the Gaussian mixture model to minimize the influence of noise and outliers and make the results more stable. In time synchronization, the matching errors caused by low-speed targets are removed to improve the accuracy. In summary, each embodiment of the present disclosure provides a spatiotemporal calibration method suitable for roadside scenes with robustness, stability and accuracy.

[0100] Although the subject matter described herein is provided in the general context of being executed in conjunction with the execution of operating systems and application programs on computer systems, it will be appreciated by those skilled in the art that other implementations may also be performed in conjunction with other types of program modules. In general, program modules include routines, programs, components, data structures, and other types of structures that perform specific tasks or implement specific abstract data types. It will be appreciated by those skilled in the art that the subject matter described herein may be practiced using other computer system configurations, including handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc., and may also be used in a distributed computing environment in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0101] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0102] It should be understood that the above specific embodiments of the present disclosure are only used to illustrate or explain the principles of the present disclosure, and do not constitute a limitation of the present disclosure. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present disclosure should be included in the protection scope of the present disclosure. In addition, the claims attached to the present disclosure are intended to cover all changes and modifications that fall within the scope and boundaries of the attached claims, or the equivalent forms of such scope and boundaries.

Claims

1. A calibration target object, set in a target area, including a circular target with high reflectivity and at least one positioning QR code, wherein: The circular target is used as a target for target detection in the point cloud data of the target area to obtain the coordinates of the calibration target in the laser radar coordinate system; The positioning two-dimensional code is used as a target for target detection in the image data of the target area to obtain the coordinates of the calibration target object in the camera coordinate system.

2. A method for spatiotemporal calibration of a laser radar and a camera, characterized in that: include: Using a camera and a laser radar, respectively, to obtain image data and point cloud data of a target area of ​​the calibration target object as claimed in claim 1, respectively perform target detection on the calibration target object in the image data and the point cloud data, and obtain the position coordinates of the calibration target object in the camera coordinate system and the laser radar coordinate system; A series of three-dimensional image points and point cloud points obtained by continuously performing target detection on the calibration target object respectively form an image trajectory and a point cloud trajectory, and determine the external parameters of the laser radar and the camera based on the similarity between the image trajectory and the point cloud trajectory; The image trajectory and the point cloud trajectory are matched based on the external parameters, and the time offset of the camera and the laser radar is determined based on the matched trajectory points.

3. The method according to claim 2, characterized in that: Performing target detection on the calibration target in the point cloud data and obtaining the position coordinates of the calibration target in the laser radar coordinate system includes: Acquire a first point cloud of the target area, filter points whose intensities are lower than a preset threshold in the first point cloud, and generate a second point cloud; Based on the properties of the circular target and the calibration target object, points corresponding to the circular target are respectively identified in the clustering results of the second point cloud and the first point cloud as a third point cloud and a fourth point cloud; Remove noise points from the third point cloud and fit a plane, calculate the center of mass and the normal vector of the plane, and determine the transformation matrix for mapping the point cloud of the calibration target object to the XY plane of the three-dimensional coordinate system based on the center of mass and the normal vector; obtain the coordinates of the center of the calibration target object in the three-dimensional coordinate system, multiply the coordinates by the inverse of the transformation matrix to obtain the position coordinates of the calibration target object in the lidar coordinate system, wherein the Z axis of the XY plane is 0.

4. The method according to claim 3, characterized in that: The step of obtaining the coordinates of the center of the calibration target object in the three-dimensional coordinate system includes: In the three-dimensional coordinate system, a point that is not less than a preset distance from the origin is selected as the center of the circle, and a circle is drawn with the radius of the circular target as the radius, and the position of the center of the circle is adjusted so that the circle contains as many of the third point cloud as possible and the points in the fourth point cloud that do not belong to the third point cloud are as many as possible outside the circle; Get the X and Y coordinates of the center of the circle, and set the Z coordinate to 0.

5. The method according to claim 2, characterized in that: The determining of the external parameters of the laser radar and the camera based on the similarity between the image trajectory and the point cloud trajectory includes: Acquire initial values ​​of external parameters of the laser radar and the camera based on minimizing the difference between the image trajectory and the point cloud trajectory; Then, Gaussian mixture models are constructed for the image trajectory and the point cloud trajectory respectively, and the final extrinsic parameter is determined based on the statistical similarity between the image trajectory and the point cloud trajectory and the Gaussian mixture model.

6. The method according to claim 5, characterized in that: The obtaining of initial values ​​of external parameters of the laser radar and the camera based on minimizing the difference between the image trajectory and the point cloud trajectory comprises: The cost function is expressed using the external parameters, the plurality of image trajectories and the trajectory points of the point cloud trajectory; Calculate the external parameters when the cost function is the minimum, the external parameters are the initial values ​​of the external parameters of the laser radar and the camera, wherein the cost function is: Among them, E is the cost function, are the starting point and the stationary point of the point cloud trajectory, are the starting point and the stationary point of the image trajectory, N represents the number of trajectories, R and t represent the rotation and translation parts of the extrinsic parameters respectively, and i is the index of the trajectory point.

7. The method according to claim 5, characterized in that: The constructing of Gaussian mixture models for the image trajectory and the point cloud trajectory respectively comprises: Respectively representing the image trajectory and the point cloud trajectory as weighted sums of Gaussian density functions to construct a first Gaussian mixture model and a second Gaussian mixture model in a three-dimensional space, wherein the number of Gaussian components in the Gaussian mixture model is equal to the number of trajectory points in the trajectory, the mean of each Gaussian component is the coordinate of the corresponding trajectory point, and all the Gaussian components have the same weight and covariance matrix; The image trajectory and the point cloud trajectory are respectively projected onto an image plane to form a two-dimensional image trajectory and a two-dimensional point cloud trajectory, and the two-dimensional image trajectory and the two-dimensional point cloud trajectory are represented as a weighted sum of Gaussian density functions to respectively construct a third Gaussian mixture model and a fourth Gaussian mixture model in two-dimensional space.

8. The method according to claim 5, characterized in that: The determining of the final external parameter based on the statistical similarity between the image trajectory and the point cloud trajectory and the Gaussian mixture model comprises: Using the external parameter to express the statistical similarity between the first Gaussian mixture model and the second Gaussian mixture model, to obtain a first distance; Using the external parameter to express the statistical similarity between the third Gaussian mixture model and the fourth Gaussian mixture model, to obtain a second distance; Determine the external parameter when the sum of the first distance and the second distance is minimized.

9. The method according to claim 2, characterized in that: The performing trajectory point matching on the image trajectory and the point cloud trajectory based on the external parameter comprises: Acquire the trajectory point whose speed is greater than a preset threshold in the point cloud trajectory; The trajectory points are transformed into a camera coordinate system according to the external parameters, and the closest point matching is performed with the trajectory points constituting the image trajectory.

10. A time-space calibration device for a laser radar and a camera, characterized in that: include: An acquisition module, used to respectively acquire image data and point cloud data of a target area of ​​the calibration target object set as claimed in claim 1 by using a camera and a laser radar, perform target detection on the calibration target object in the image data and the point cloud data, and acquire position coordinates of the calibration target object in a camera coordinate system and a laser radar coordinate system; A first determination module is used to form an image trajectory and a point cloud trajectory respectively from a series of three-dimensional image points and point cloud points obtained by continuously performing target detection on the calibration target object, and determine the external parameters of the laser radar and the camera based on the similarity between the image trajectory and the point cloud trajectory; The second determination module is used to match the image trajectory and the point cloud trajectory based on the external parameters, and determine the time offset of the camera and the laser radar based on the matched trajectory points.

Images