A roadside laser radar camera calibration method and system

Abstract

The application discloses a road end laser radar camera calibration method and system, and the method comprises the following steps: for two frames of continuous point clouds, the scene flow between the two frames of point clouds is estimated, a threshold is set to realize dynamic and static separation, and the point cloud scene flow of a moving target is obtained; for two frames of continuous images, the image optical flow of the moving target is obtained by separating dynamic objects through image optical flow; based on the image optical flow and the point cloud scene flow of the moving target, rough calibration is performed on the laser radar and camera external parameters to obtain an initial estimation of the laser radar and camera external parameters; and according to the image optical flow and the point cloud scene flow of the moving target, the initial estimation of the laser radar and camera external parameters is optimized to obtain an accurate estimation of the laser radar and camera external parameters; the present calibration solves the inconvenience of the deployment of the road end sensor, the traffic flow in the traffic scene is extracted by using the optical flow scene flow, and the external parameter calibration of the laser radar and the camera is automatically realized.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of sensor parameter calibration, and particularly relates to a road end laser radar camera calibration method and system. BACKGROUND

[0002] The automatic driving system faces huge safety challenges, and the vehicle-road cooperative perception system is considered to effectively improve the safety of the automatic driving system. The road testing device can bring an ultra-wide view to the automatic driving system, solving the problem of limited single vehicle perception range. Road testing perception, as an infrastructure, can provide real-time information of the road environment for drivers, such as collision warning of people and motor vehicles, congestion warning in front, traffic accident warning, etc.; and can provide monitoring and prediction of the road traffic environment for the traffic management department, such as vehicle flow statistics, vehicle illegal parking detection, interval speed measurement, etc.

[0003] In road end perception, laser radar and camera are common sensor configurations. The camera can provide images containing rich semantic information but lacks depth information, while the laser radar can provide accurate but sparse 3D measurement point cloud but is difficult to perceive detailed semantic information. The perception algorithm based on fusion of laser radar and camera often has better performance, and the external parameter calibration of laser radar and camera is the basis for its implementation. Unlike the laser radar camera calibration on traditional robots, the automation degree of road end sensor calibration has higher requirements. The current method mostly uses a checkerboard to calibrate the laser radar and the camera. The deployment of the checkerboard will affect normal traffic, and once the object is removed, it is difficult to realize re-calibration. At the same time, some methods based on semantic features rely on specific semantic features and are seriously dependent on the data set, and are not strong in universality. With the development of road end perception system, it is necessary to study the automatic laser camera external parameter calibration method.

[0004] The existing laser radar camera calibration technology mostly relies on a checkerboard. This calibration method needs to install a checkerboard, especially for road end devices. The installation height of laser radar and camera is generally high, so the required checkerboard is often large, which will seriously affect normal traffic, and the checkerboard cannot be recalibrated after being removed. Since the road end device is often static after installation, the motion-based calibration method is not suitable for road end devices. The end-to-end calibration method based on learning relies on the already calibrated data as the true value for training, and its generalization performance in different scenes is difficult to guarantee. The method based on semantic feature alignment can extract semantic elements such as vehicles, pedestrians and ground, which is a relatively universal feature. However, both point cloud semantic segmentation and image semantic segmentation rely on a large amount of data labeling, and their segmentation effect also depends on the data set, and the performance in different scene deployment is difficult to guarantee. SUMMARY

[0005] In order to solve the problems in the prior art, the present application provides a laser radar camera external parameter automatic calibration algorithm based on optical flow and scene flow, which solves the inconvenience of existing calibration for road end sensor deployment, uses optical flow scene flow to extract traffic flow (vehicles, pedestrians) in the traffic scene, and automatically realizes the external parameter calibration of the laser radar and the camera.

[0006] To achieve the above object, the technical scheme adopted by the present application is as follows: a road end laser radar camera calibration method, comprising the following steps:

[0007] For two consecutive point clouds, the scene flow between the two frames of point clouds is estimated, a threshold is set to realize dynamic and static separation, and the point cloud scene flow of the moving target is obtained;

[0008] For two consecutive images, the image optical flow of the moving target is obtained by separating dynamic objects through image optical flow;

[0009] Based on the image optical flow and the point cloud scene flow of the moving target, the laser radar and camera external parameters are coarsely calibrated to obtain an initial estimation of the laser radar and camera external parameters;

[0010] According to the image optical flow and the point cloud scene flow of the moving target, the initial estimation of the laser radar and camera external parameters is optimized to obtain an accurate estimation of the laser radar and camera external parameters.

[0011] Further, for two consecutive point clouds, the scene flow between the two frames of point clouds is estimated, a threshold is set to realize dynamic and static separation, and the point cloud scene flow of the moving target is obtained, specifically:

[0012] For point cloud data, first, the ground point estimation method is used to estimate and filter the ground points, and then the outlier removal algorithm is used to remove noise points, so that the point cloud of the object on the ground is relatively pure;

[0013] For the point cloud of the object on the ground which is relatively pure, the continuous point cloud frame is used to estimate the point cloud scene flow by using the neural prior scene flow method, the point cloud scene flow is obtained, and the objects whose speed is lower than the speed threshold are filtered out by setting the speed threshold, so that the point cloud mask ML of the moving target is obtained, and the scene flow and the point cloud of the moving target are obtained.

[0014] Further, for two consecutive images, the image optical flow of the moving target is obtained by separating dynamic objects through image optical flow, specifically:

[0015] For consecutive frames of images, the moving objects are segmented out by optical flow estimation to obtain the mask MI of the dynamic objects in the image, and the motion of the dynamic objects in the image coordinate system and the pixel point representation of the obtained image optical flow are given.

[0016] Furthermore, based on the image optical flow and point cloud scene flow of the moving target, coarse calibration of the extrinsic parameters of the LiDAR and camera is performed to obtain initial estimates of the extrinsic parameters of the LiDAR and camera, including the following steps:

[0017] The initial values ​​of the extrinsic parameters are estimated using a sampling method: The extrinsic parameters of the LiDAR and camera include rotation and translation parameters. If the relative translation between the LiDAR and camera is less than a set translation amount, only the pose is sampled, and the translation parameters are set to [0,0,0]. T For the displacement between the LiDAR and the camera not less than a set translation amount, the range of the relative displacement between the LiDAR and the camera is given, and the relative translation is sampled.

[0018] Estimating the principal directions and centroids of optical flow for moving objects and scene flow: Each moving target is described using its centroid. For image optical flow, Euclidean distance clustering is applied to the pixels of the moving objects to obtain the image optical flow set f for each moving target. k,j For point cloud scene flow, the point clouds of moving objects are clustered using the Euclidean distance method to obtain the point cloud and scene flow set F for each moving object. k,j Then calculate the optical flow centroid and principal direction of each moving target, as well as the scene flow centroid and principal direction of the scene flow;

[0019] The centroid and main direction of the scene flow of the point cloud are projected onto the camera pixel coordinate system using camera intrinsic parameters and sampling extrinsic parameters. The sampling pose is evaluated by calculating the distance between the centroid and main direction of the scene flow after the point cloud is projected and the nearest optical flow centroid and main direction. A series of distance scores for sampling extrinsic parameters are obtained. All the scores of sampling extrinsic parameters are sorted, and the extrinsic parameter with the lowest score is the initial estimate of the extrinsic parameters of the LiDAR and the camera.

[0020] Furthermore, the pixel centroid and principal direction of optical flow for each moving target, as well as the point cloud centroid and principal direction of scene flow, are calculated using the following formulas:

[0021]

[0022]

[0023] Among them, f k,j Representative image I k The set of pixels and optical flow of a moving object j, where an element f is a subset of the set. k,j,i ={p k,j,i ,v k,j,i The image coordinates p of the moving object's pixels k,j,i And the optical flow v of that pixel k,j,i Composition. F k,j Representative point cloud P kA point cloud of a moving object j and a set of flow fields of its scene, one element F k,j,i = {P k,j,i , V k,j,i} consists of a point P k,j,i in the point cloud of the moving object and a flow field V k,j,i of the point.

[0024] Further, the projection formula of the flow field is as follows:

[0025]

[0026]

[0027] Wherein, the point P k.i in the point cloud is projected to its pixel coordinate p k.i by the camera intrinsic K and the extrinsic {R, t} of the camera and the lidar, Z c represents the Z-axis coordinate of the point in the camera coordinate system, and the flow field estimate V k,i of the point is projected in the pixel coordinate system as represents the Z-direction component of the flow field of the point projected in the camera coordinate system;

[0028] The evaluation function is as follows:

[0029]

[0030] Wherein, represents the projection of the point cloud centroid P k,j of the moving object j and the image the image centroid of the nearest moving object, the main direction of the optical flow of the moving object, the projection of the main direction V k,j of the flow field of the moving object in the image, and α and β are adjustment factors for adjusting the weight of the point projection and the flow field projection.

[0031] Further, according to the image optical flow and the point cloud flow field of the moving object, the initial estimate of the extrinsic of the lidar and the camera is optimized to obtain the accurate estimate of the extrinsic of the lidar and the camera, which is as follows:

[0032] An optimization equation is constructed, and a nonlinear optimization method is used to solve it, and an accurate extrinsic estimate is obtained by iterative optimization. The optimization target has two aspects: one is to make the projection of the moving object point cloud obtained by the lidar be completely covered by the corresponding optical flow mask, and the other is to make the flow field of the moving object point cloud projection in the pixel coordinate system and its optical flow as close as possible;

[0033] When constructing the optimization equation, the moving object point cloud is projected onto the pixel coordinate system. The nearest neighbor search method is used to obtain the pixel coordinates of the moving object closest to the moving object point cloud. The pixel distance and the difference between the projection and optical flow estimated by the scene flow are calculated. The optimization equation is as follows:

[0034]

[0035] in, Point P in the point cloud representing a moving object m k,m,i Projection in the image The nearest point, For pixels Optical flow estimation, Representative point P k,m,i Scene Stream V k,m,i In the projection onto the image, μ and γ are adjustment factors used to adjust the weights of point projection and scene flow projection in the loss function. During optimization, after obtaining a local optimum each time, the local optimum is used as the initial value for the next iteration. Through iterative optimization, the global optimum is obtained, and finally, an accurate estimate of the extrinsic parameters of the LiDAR and camera is obtained.

[0036] Based on the above concept, the present invention also provides a roadside lidar camera calibration system, including a point cloud scene flow estimation module, an image optical flow estimation module, a coarse calibration module, and a fine calibration module;

[0037] The point cloud scene flow estimation module is used to estimate the scene flow between two consecutive point clouds, set a threshold to achieve motion-static separation, and obtain the point cloud scene flow of the moving target.

[0038] The image optical flow estimation module is used to separate dynamic objects from moving targets based on two consecutive images to obtain the image optical flow of the moving target.

[0039] The coarse calibration module performs coarse calibration on the extrinsic parameters of the lidar and camera based on the image optical flow and point cloud scene flow of the moving target to obtain initial estimates of the extrinsic parameters of the lidar and camera;

[0040] The fine calibration module is used to optimize the initial estimates of the extrinsic parameters of the lidar and camera based on the image optical flow and point cloud scene flow of the moving target, so as to obtain accurate estimates of the extrinsic parameters of the lidar and camera.

[0041] The present invention can also provide a computer device, including a processor and a memory, wherein the memory is used to store a computer executable program, the processor reads the computer executable program from the memory and executes it, and the processor can implement the roadside lidar camera calibration method of the present invention when executing the computer executable program.

[0042] The application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the road end laser radar camera calibration method.

[0043] Compared with the prior art, the application has at least the following beneficial effects:

[0044] The method uses low-level general feature optical flow and scene flow as calibration features, and the laser radar and the camera are basically stationary in the road end device, and there are a large number of moving targets such as pedestrians and vehicles, so that the segmentation of the moving objects can be easily realized by using the optical flow and scene flow estimation, and the dependence of the optical flow and scene flow on the data set is not strong, and the generalization of the deep learning method in different scenes can be ensured; the method provided by the application has more general features, moving objects are generally present in the traffic scene, and the dependence of the optical flow and scene flow estimation on the data set is smaller, so that the method can realize fully automatic calibration without human intervention, and is more suitable for the deployment of the road end perception system. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 FIG. 1 is a flowchart of a road end laser radar camera automatic calibration method based on image optical flow and point cloud scene flow.

[0046] Figure 2 FIG. 3 is a calibration process in a pixel coordinate system when the image optical flow and the point cloud scene flow are used for calibration.

[0047] Figure 3 FIG. 5 is a flowchart of a coarse calibration method using the image optical flow and the point cloud scene flow.

[0048] Figure 4 FIG. 7 is an effect diagram of the coarse calibration method on simulation data and real data.

[0049] Figure 5 FIG. 9 is a flowchart of a fine calibration method using the image optical flow and the point cloud scene flow.

[0050] Figure 6 FIG. 11 is a result of calibration using the image optical flow and the point cloud scene flow on real data.

[0051] Figure 7 FIG. 13 is a consistency evaluation result of the method in an actual scene. DETAILED DESCRIPTION

[0052] The exemplary embodiments of this application are described in detail below with reference to the accompanying drawings and specific implementations, including various details of the embodiments of this application to aid understanding. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art fall within the scope defined by the appended claims.

[0053] Figure 1 This is a flowchart of the automatic calibration method for roadside LiDAR cameras based on image optical flow and point cloud scene flow, as proposed in this invention. This invention is applicable to the calibration of roadside LiDAR and camera sensors. It comprises three parts: data preprocessing and optical flow / scene flow estimation, coarse calibration, and fine calibration. The input to the proposed algorithm is two consecutive frames of point cloud data. and two consecutive RGB images I k , Where N p N represents the number of points in the point cloud. h N w This represents the image size, and we also need to know the camera's intrinsic parameters, denoted as K∈R. 3×3 The goal is to estimate the 6-DOF extrinsic parameters R and t between the lidar and the camera.

[0054] Since the movement of roadside equipment is minimal, with a speed almost zero, dynamic objects such as pedestrians and vehicles can be easily segmented using point cloud scene flow and image optical flow. For two consecutive point cloud frames, the scene flow F between the two frames is first estimated using a point cloud scene flow estimation algorithm. Pk ={P k V k The algorithm sets a threshold to achieve motion-static separation and obtains a dynamic point cloud mask. This mask is then used to capture the point cloud of the moving object obtained by the lidar. For two consecutive frames, the optical flow is estimated using an optical flow estimation method. Image mask M is obtained by separating dynamic objects using image optical flow. I Motion segmentation of images and point clouds is achieved using image optical flow and point cloud scene flow. The extrinsic parameters of LiDAR and camera are solved based on the images and point clouds of moving objects.

[0055] Solving for the extrinsic parameters of the LiDAR and camera involves two steps: coarse calibration and fine calibration. The first step involves statistically calculating the principal vectors and geometric centers of dynamic objects in the point cloud scene flow and image optical flow, respectively. Then, the extrinsic parameters are sampled, and the principal velocities and geometric centers of the point cloud scene flow are projected onto the image. Based on the projection, the sampled poses are scored, and the pose with the highest score is used as the initial estimate {R}. init ,t initSecond part is fine calibration, using the initial estimate obtained in the first step as the initial value, the point cloud and scene flow are projected to the pixel coordinate system. The nearest neighbor search is used to realize the matching of the point cloud to the pixel, and the re-projection error iteration optimization is constructed to obtain the accurate external parameters of the radar and the camera. Finally, the simulation platform Carla and the actual scene are tested to verify the effectiveness of the application. Details of the application are as follows:

[0056] Step 1: The data collected by the sensor is processed, the data noise is filtered, and then the image optical flow and point cloud scene flow are estimated by using the optical flow scene flow estimation algorithm, and the specific details are as follows:

[0057] For continuous frame images, a typical image optical flow algorithm such as RAFT is used, and the public data set (FlyingChairs, FlyingThing3d, Sintel) is trained. Since the optical flow is a relatively low-level and general feature, it can have good effect without fine-tuning in real data.

[0058] Since the road end laser radar and the camera are almost stationary, the moving objects (vehicles, pedestrians) are segmented out by the optical flow estimation to obtain the mask MI of the dynamic objects in the image, and the motion of the dynamic objects in the image coordinate system can be given. The pixel and optical flow estimation of the moving objects in the image can be obtained through the mask, which is represented as f i,k ={p i,k ,v i,k}∈f k .

[0059] For point cloud data, unlike image data, there are many noise points and useless ground points in the point cloud data. First, the ground point estimation algorithm (such as PatchWork) is used to estimate and filter the ground points, and then the outlier removal algorithm is used to remove the noise points, so that the relatively pure point cloud of the objects on the ground can be obtained.

[0060] The relatively pure point cloud of the objects on the ground is estimated by using the continuous point cloud frames, and the neural prior is used to estimate the point cloud scene flow. The method is Neural Scene Flow Prior, which can be estimated online to obtain the point cloud scene flow, which is represented as By setting the speed threshold, the static or low motion speed objects are filtered out, so that the point cloud mask M L of the moving target is obtained, and the point cloud and scene flow estimation of the moving target is obtained.

[0061] Step 2: The image optical flow and point cloud scene flow of the moving target obtained in step 1 are used for coarse calibration, and the algorithm flow is as shown in Figure 3 , and the specific details are as follows:

[0062] 1) External parameter sampling: The purpose of coarse calibration is to find an initial value close to the real value of external parameters. In order to make the estimation fully automatic, the present invention uses sampling method to estimate the initial value of external parameters. The external parameters of LiDAR and camera include rotation parameters R and translation parameters t. The possible value range of rotation external parameter R is limited, while the relative translation parameter t can have infinite range without installation parameters. Therefore, in the present invention, only the pose is sampled, and the translation parameter is set to [0, 0, 0] T It is effective when the relative displacement of LiDAR and camera is small (displacement < 50 cm). On the contrary, for large relative translation of LiDAR and camera (displacement > 50 cm), it is necessary to give the approximate range of the relative displacement of LiDAR and camera, and sample the relative translation.

[0063] 2) Estimate the main direction and centroid of the motion object optical flow and scene flow: In order to reduce the amount of calculation, only the centroid of the motion object is used to describe each motion target during initial value estimation. For image optical flow, the Euclidean distance clustering method is used on the motion object mask to obtain the image optical flow set f k,j For point cloud scene flow, the point cloud of the motion object is clustered by Euclidean distance clustering method to obtain the point cloud and scene flow set F k,j .

[0064] Then the optical flow centroid and main direction of each motion target and the scene flow centroid and main direction are calculated by the following formula:

[0065]

[0066]

[0067] where f k,j represents the pixel and optical flow set of motion object j in image I k , and an element f k,j,i = {p k,j,i , v k,j,i} in the set is composed of the image coordinate p k,j,i of the motion object pixel and the optical flow v k,j,i of the pixel. F k,j represents the point cloud and scene flow set of motion object j in point cloud P k , and an element F k,j,i = {P k,j,i , V k,j,i} in the set is composed of the point P k,j,i in the motion object point cloud and the scene flow V k,j,i of the point.

[0068] Further, the optical flow centroid and the main direction of the optical flow of each moving target in the image and the scene flow centroid and the main direction of the scene flow are obtained.

[0069] 3) After the main direction of the optical flow and the main direction of the scene flow are obtained, the scene flow main direction and the moving object point cloud centroid are projected to the camera pixel coordinate system through the camera intrinsic parameter K and the sampling extrinsic parameters R and t, and the scene flow projection formula is as follows:

[0070]

[0071]

[0072] wherein the point P in the point cloud k.i is projected to its pixel coordinate p through the camera intrinsic parameter K and the extrinsic parameters {R, t} of the camera and the laser radar k.i , Z c represents the Z-axis coordinate of the point in the camera coordinate system. The scene flow estimation V of the point is k,i The projection of the pixel coordinate system is represented as The Z-direction component of the scene flow of the point projected in the camera coordinate system is represented as

[0073] The scene flow projection formula ignores high-order small amounts. Through the projection, the scene flow centroid and the main direction after the projection of the point cloud and the distance from the nearest optical flow centroid and the main direction are calculated, so as to evaluate the sampling pose. The evaluation function is as follows:

[0074]

[0075] wherein, represents the projection of the point P k,m,i in the image in the point cloud of the moving object m the nearest point, is the optical flow estimation of the pixel point , and represents the projection of the scene flow V k,m,i of the point P k,m,i in the image. μ and γ are adjustment factors for adjusting the weight of the point projection and the scene flow projection in the loss function. During optimization, after the local optimal value is obtained each time, the local optimal value is used as the initial value for the next iteration for optimization. Through continuous iteration and optimization, the global optimal value is obtained, and finally the accurate estimation of the extrinsic parameters of the laser radar and the camera is obtained.

[0076] The evaluation function uses the consistency evaluation of the point cloud scene flow and the optical flow to evaluate the sampling pose, and the scores D of all sampling extrinsic parameters are sorted, and the extrinsic parameters corresponding to the final score D with the lowest score are the initial estimation of the extrinsic parameters of the laser radar and the camera.

[0077] To verify the effectiveness of the initial value estimation algorithm, data collected in the simulation environment Carla is used to verify the effectiveness of the method. Specifically, in the simulation environment, a laser radar and a camera are set up at a typical traffic intersection, the height is set to 9 m, the horizontal elevation angle is -30°, the rotation in the extrinsic parameter of the laser radar and the camera is represented by the quaternion q = [0.500, -0.500, 0.500, 0.500], and the displacement is t = [-0.600, 0.354, -0.354] T .

[0078] In the simulation environment, the true value of the extrinsic parameter can be obtained, and the initial value estimation result is set to be within 20° of the true value of the extrinsic parameter rotation error. The experimental results of the initial value estimation are as follows:

[0079]

[0080] In the experiment, 156 frames of point cloud and image data are tested, and the effectiveness of the proposed method is proved from the experimental results. The experiment also shows the influence of the sampling interval of the extrinsic parameter pose on the success rate and the operation time, which needs to be selected according to the actual situation in actual use. The experimental results of the initial value estimation in the simulation environment are shown in Figure 4 .

[0081] Step 3: Using the initial value of the extrinsic parameter obtained in step 2 and the optical flow and scene flow obtained in step 1, the extrinsic parameter is optimized to obtain more accurate parameter estimation. The process is shown in Figure 5 , the principle is shown in Figure 2 , and the specific process is as follows:

[0082] Using the initial value of the extrinsic parameter obtained in step 2, the point cloud of the moving object and the scene flow estimation are projected to the image coordinate system. Since there is a certain gap between the initial estimated extrinsic parameter and the true extrinsic parameter, the projected point cloud does not completely coincide with the moving object point cloud, as shown in Figure 6 (a). When the laser radar and the camera are calibrated, the moving object point cloud mask in the image should completely cover the moving object point cloud, as shown in Figure 6 (c).

[0083] To achieve accurate calibration of the extrinsic parameter of the laser radar and the camera, an optimization equation is constructed, and a nonlinear optimization method is used to solve it. Iterative optimization is used to obtain accurate extrinsic parameter estimation. The optimization target has two aspects: one is to make the projection of the moving object point cloud obtained by the laser radar completely covered by the corresponding optical flow mask, and the other is to make the scene flow of the moving object point cloud projection as close as possible to the pixel coordinate system and its optical flow.

[0084] In constructing the optimization equation, the moving object point cloud is projected to the pixel coordinate system, the nearest neighbor searching method is used to obtain the pixel coordinates of the moving object closest to the moving object point cloud, the difference between the pixel distance and the scene flow estimation projection and the optical flow is calculated, and the matching process is as shown in Figure 2 The optimization equation is as follows:

[0085]

[0086] Wherein, represents the point P k,m,i in the moving object point cloud The projection in the image The nearest point, The optical flow estimation of the pixel point represents the scene flow V k,m,i of the point P k,m,i The projection in the image. μ and γ are adjustment factors for adjusting the weight of the point projection and the scene flow projection in the loss function.

[0087] Since there is an error in the optical flow scene flow estimation, when constructing the scene flow projection error, the data with large error needs to be filtered out. Since the matching is completed by the nearest neighbor method, this method cannot completely reflect the real data matching. Therefore, after completing the optimization once, the local optimal value is obtained, and the value is used as the initial value for the next optimization and data association matching for re-optimization, and the global optimal value is obtained through continuous iteration optimization. Finally, the accurate estimation of the laser radar and camera external parameters is obtained, and the estimation result is as shown in Figure 6 (b).

[0088] The present application has been tested in detail in simulation environment and real scene. In the simulation environment, the true value of the external parameters of the laser radar and the camera can be known. Through the test of 125 frames of laser camera data, the method disclosed by the present application can realize the effect of an average translation error of 6.24 cm and an average rotation error of 1.73°. In real data, the true value of the external parameters of the laser radar and the camera cannot be obtained, so the consistency of the algorithm is tested, and the traditional manual calibration method is compared, and the comparison effect is as shown in Figure 6 . Through the test of about 300 frames of data, the consistency is good, and the consistency result is as shown in Figure 7 . The translation and rotation are evaluated by the error of the consistency and the results of the manual calibration.

[0089] Based on the concept of the method, the present application provides a road end laser radar camera calibration system, comprising a point cloud scene flow estimation module, an image optical flow estimation module, a coarse calibration module and a fine calibration module.

[0090] The point cloud scene flow estimation module is configured to estimate scene flow between two frames of point clouds according to the two frames of continuous point clouds, to set a threshold to realize dynamic and static separation, and to obtain point cloud scene flow of the moving target.

[0091] The image optical flow estimation module is configured to obtain image optical flow of the moving target by separating dynamic objects through image optical flow according to two frames of continuous images.

[0092] The coarse calibration module is configured to perform coarse calibration on the laser radar and camera extrinsic parameters based on the image optical flow and the point cloud scene flow of the moving target to obtain initial estimation of the laser radar and camera extrinsic parameters.

[0093] The fine calibration module is configured to optimize the initial estimation of the laser radar and camera extrinsic parameters based on the image optical flow and the point cloud scene flow of the moving target to obtain accurate estimation of the laser radar and camera extrinsic parameters.

[0094] The present application can also provide a computer device comprising a processor and a memory, the memory being configured to store computer executable programs, the processor being configured to read part or all of the computer executable programs from the memory and execute, and the processor being configured to implement the road end laser radar camera calibration method when executing part or all of the computer executable programs.

[0095] In another aspect, the present application provides a computer readable storage medium, the computer readable storage medium being configured to store computer programs, and the computer programs being configured to be executed by a processor to implement the road end laser radar camera calibration method.

[0096] The computer device can be a notebook computer, a desktop computer, a vehicle-mounted computer or a workstation.

[0097] The processor can be a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC) or a ready programmable gate array (FPGA).

[0098] The memory can be an internal storage unit of a notebook computer, a desktop computer, a vehicle-mounted computer or a workstation, such as a memory or a hard disk, or can be an external storage unit, such as a mobile hard disk or a flash card.

[0099] The computer-readable storage medium can include computer storage media and communication media. The computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. The computer-readable storage medium can include read-only memory (ROM), random access memory (RAM), solid state disk (SSD), optical disk, etc. Among them, the random access memory can include resistance random access memory (ReRAM) and dynamic random access memory (DRAM).

[0100] In summary, in the first aspect, the application provides a general moving object extraction method for traffic scenes, which comprises the following steps: estimating the motion of vehicles and pedestrians in consecutive frame images by optical flow, obtaining the motion object mask and its motion speed in the image. Remove the ground points in the point cloud by using the point cloud ground segmentation algorithm, and remove the noise points and useless points by using the outlier removal method, to obtain the point cloud of the objects on the ground. Perform scene flow estimation on the filtered consecutive frame point cloud to obtain the point cloud motion information. Set a speed threshold to filter the static point cloud to obtain a dynamic object point cloud mask and the motion of the moving object point cloud.

[0101] In the second aspect, the application provides a coarse calibration method based on optical flow scene flow. Based on the motion object image mask and the motion object point cloud mask obtained by optical flow and scene flow, each motion object in the image and the radar point cloud is obtained, and the motion main direction and the centroid of the image and the point cloud are estimated. Sample the possible external parameters of the laser radar and camera external parameters to obtain all possible coarse external parameters. Align the centroids and main directions of the point cloud motion objects to evaluate all sampling poses according to the distance, and select the optimal possible external parameter as the initial estimation of the external parameters of the laser radar and camera.

[0102] In the third aspect, the application provides a laser radar camera fine calibration method based on optical flow and scene flow. The internal correlation of the motion object expressed by the point cloud scene flow and the image optical flow is utilized, the motion object point cloud and the scene flow are projected onto the image, and the pixel distance and the optical flow distance of the motion object in the image are optimized to obtain the accurate external parameter estimation.

[0103] The application acquires moving objects in a traffic scene through optical flow and scene flow, and realizes automatic laser camera calibration. Unlike traditional calibration methods, the method proposed by the application is more general in terms of required features, moving objects are generally present in traffic scenes, and the dependence of optical flow scene flow estimation on data sets is smaller. The method proposed in the application can realize fully automatic calibration without human intervention, and is more suitable for deployment of road end perception systems.

[0104] The above is only a preferred embodiment of the application, but the protection scope of the application is not limited thereto, any person skilled in the art can make equivalent replacement or change within the technical range disclosed by the application according to the technical solution and inventive concept of the application, which should be covered within the protection scope of the application.

Claims

1. A method for calibrating a roadside lidar camera, characterized in that, Includes the following steps: For two consecutive point cloud frames, estimate the scene flow between the two point cloud frames, set a threshold to achieve motion-static separation, and obtain the point cloud scene flow of the moving target. For two consecutive frames of images, the image optical flow of the moving target is obtained by separating the dynamic object through image optical flow; Based on the image optical flow and point cloud scene flow of moving targets, the extrinsic parameters of the lidar and camera are coarsely calibrated to obtain initial estimates of the extrinsic parameters of the lidar and camera; Based on the image optical flow and point cloud scene flow of the moving target, the initial estimates of the extrinsic parameters of the lidar and camera are optimized to obtain accurate estimates of the extrinsic parameters of the lidar and camera. The initial estimates of the extrinsic parameters of the LiDAR and camera are obtained by coarsely calibrating the extrinsic parameters of the LiDAR and camera based on the image optical flow and point cloud scene flow of the moving target. This includes the following steps: The initial values ​​of the extrinsic parameters are estimated using a sampling method: the extrinsic parameters of the LiDAR and camera include rotation and translation parameters. If the relative translation between the LiDAR and camera is less than a set translation amount, only the pose is sampled, and the translation parameters are set to... For the displacement between the LiDAR and the camera not less than a set translation amount, the range of the relative displacement between the LiDAR and the camera is given, and the relative translation is sampled. Estimating the principal directions and centroids of optical flow for moving objects and scene flow: Each moving target is described using its centroid. For image optical flow, Euclidean distance clustering is applied to the pixels of the moving objects to obtain the image optical flow set for each moving target. For point cloud scene flow, the point clouds of moving objects are clustered using the Euclidean distance method to obtain the point cloud and scene flow set for each moving object. Then calculate the optical flow centroid and principal direction of each moving target, as well as the scene flow centroid and principal direction of the scene flow; The centroid and main direction of the scene flow of the point cloud are projected onto the camera pixel coordinate system using camera intrinsic parameters and sampling extrinsic parameters. The sampling pose is evaluated by calculating the distance between the centroid and main direction of the scene flow after the point cloud is projected and the nearest optical flow centroid and main direction. A series of distance scores for sampling extrinsic parameters are obtained. All the scores of sampling extrinsic parameters are sorted, and the extrinsic parameter with the lowest score is the initial estimate of the extrinsic parameters of the LiDAR and the camera.

2. The method for calibrating a roadside lidar camera according to claim 1, characterized in that, For two consecutive point cloud frames, the scene flow between the two frames is estimated, a threshold is set to achieve motion-static separation, and the point cloud scene flow of the moving target is obtained, specifically as follows: For point cloud data, we first use the point cloud ground estimation method to estimate and filter ground points, and then use the outlier removal algorithm to remove noise points, thus obtaining a relatively clean point cloud of objects on the ground. For a relatively clean point cloud of objects on the ground, the neural prior scene flow method is used to estimate the point cloud scene flow using continuous point cloud frames. By setting a velocity threshold to filter out stationary objects or objects with a velocity speed lower than the velocity threshold, the point cloud mask of the moving target is obtained, thus obtaining the scene flow and point cloud of the moving target.

3. The method for calibrating a roadside lidar camera according to claim 1, characterized in that, For two consecutive images, the image optical flow of the moving target is obtained by separating the dynamic object through image optical flow, specifically as follows: For consecutive frames of images, moving objects are segmented using optical flow estimation to obtain masks of dynamic objects in the image. MI This gives the pixel representation of the motion of a dynamic object in the image coordinate system and the optical flow of the resulting image.

4. The method for calibrating a roadside lidar camera according to claim 1, characterized in that, The following formulas are used to calculate the pixel centroid and principal optical flow direction of each moving target, as well as the point cloud centroid and principal scene flow direction: in, Representative image The set of pixels and optical flow of a moving object j, where one element is a pixel in the set. Image coordinates of the pixels of the moving object and the optical flow of that pixel composition, Representative point cloud The point cloud of a moving object j and its scene flow set, where one element is a point cloud of the moving object j. Points in the point cloud of the moving object and the scene flow at that point composition.

5. The method for calibrating a roadside lidar camera according to claim 1, characterized in that, The scene flow projection formula is as follows: Among them, the points in the point cloud Through camera internal parameters K The extrinsic parameters of the camera and LiDAR { R,t The pixel coordinates are obtained by projection. , This represents the Z-axis coordinate in the camera coordinate system at that point, while the scene flow estimation at that point... The projection in the pixel coordinate system is represented as , The Z-direction component of the scene flow projected onto the camera coordinate system at that point; The evaluation function is as follows: in, The centroid of the point cloud representing the moving object j Projection in the image The image centroid of the nearest moving object This is the principal direction of the optical flow of the moving object. The main direction of the scene flow for the moving object Projection in the image, and This is an adjustment factor used to adjust the weights of point projection and scene flow projection.

6. The method for calibrating a roadside lidar camera according to claim 1, characterized in that, Based on the image optical flow and point cloud scene flow of the moving target, the initial estimates of the extrinsic parameters of the LiDAR and camera are optimized to obtain accurate estimates of the extrinsic parameters of the LiDAR and camera, as follows: An optimization equation is constructed and solved using a nonlinear optimization method. Iterative optimization is used to obtain accurate external parameter estimates. The optimization objectives are twofold: first, to ensure that the projection of the moving object point cloud obtained by the LiDAR is completely covered by the corresponding optical flow mask; and second, to make the scene flow of the moving object point cloud projection as close as possible to its optical flow in the pixel coordinate system. When constructing the optimization equation, the moving object point cloud is projected onto the pixel coordinate system. The nearest neighbor search method is used to obtain the pixel coordinates of the moving object closest to the moving object point cloud. The pixel distance and the difference between the projection and optical flow estimated by the scene flow are calculated. The optimization equation is as follows: in, Representative and moving objects m Points in a point cloud Projection in the image The nearest point, For pixels Optical flow estimation, Representative point Scene flow Projection in the image, and The adjustment factor is used to adjust the weights of point projection and scene flow projection in the loss function. During optimization, after obtaining a local optimum each time, the local optimum is used as the initial value for the next iteration. Through iterative optimization, the global optimum is obtained, and finally, an accurate estimate of the extrinsic parameters of the LiDAR and camera is obtained.

7. A roadside lidar camera calibration system, characterized in that, It includes a point cloud scene flow estimation module, an image optical flow estimation module, a coarse calibration module, and a fine calibration module; The point cloud scene flow estimation module is used to estimate the scene flow between two consecutive point clouds, set a threshold to achieve motion-static separation, and obtain the point cloud scene flow of the moving target. The image optical flow estimation module is used to separate dynamic objects from moving targets based on two consecutive images to obtain the image optical flow of the moving target. The coarse calibration module, based on the image optical flow and point cloud scene flow of the moving target, performs coarse calibration on the extrinsic parameters of the LiDAR and camera to obtain initial estimates of their extrinsic parameters. Specifically, this includes estimating the initial values ​​of the extrinsic parameters using a sampling method: the extrinsic parameters of the LiDAR and camera include rotation and translation parameters. If the relative translation between the LiDAR and camera is less than a set translation amount, only the pose is sampled, and the translation parameters are set to... For a displacement between the LiDAR and the camera that is not less than a set translation amount, the range of relative displacement between the LiDAR and the camera is given, and the relative translation is sampled; the principal directions and centroids of the moving object optical flow and scene flow are estimated: each moving target is described using the centroid of the moving object, and for image optical flow, the Euclidean distance clustering method is used to obtain the image optical flow set of each moving target for the moving object pixels. For point cloud scene flow, the point clouds of moving objects are clustered using the Euclidean distance method to obtain the point cloud and scene flow set for each moving object. Then, the optical flow centroid and principal direction of each moving target, as well as the scene flow centroid and principal direction of the scene flow, are calculated. The point cloud scene flow centroid and principal direction of the scene flow are projected onto the camera pixel coordinate system through camera intrinsic parameters and sampling extrinsic parameters. The distance between the projected scene flow centroid and principal direction of the point cloud and the nearest optical flow centroid and principal direction is calculated to evaluate the sampling pose, resulting in a series of distance scores for the sampling extrinsic parameters. The scores of all sampling extrinsic parameters are sorted, and the extrinsic parameter with the lowest score is the initial estimate of the lidar and camera extrinsic parameters. The fine calibration module is used to optimize the initial estimates of the extrinsic parameters of the lidar and camera based on the image optical flow and point cloud scene flow of the moving target, so as to obtain accurate estimates of the extrinsic parameters of the lidar and camera.

8. A computer device, characterized in that, It includes a processor and a memory, the memory being used to store a computer-executable program, the processor reading the computer-executable program from the memory and executing it, and the processor executing the computer-executable program being able to implement the roadside lidar camera calibration method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, A computer-readable storage medium stores a computer program that, when executed by a processor, enables the roadside lidar camera calibration method as described in any one of claims 1-6.

Images