An online calibration method for external parameters between laser radar and camera in road environment

Abstract

An online calibration method for extrinsic parameters between a laser radar and a camera in a road environment, which analyzes the representation and detection accuracy of the semantic information of the collected road environment in the laser radar point cloud segmentation results and the camera's image detection results to obtain analysis results; performs offline calibration of the extrinsic parameters between the laser radar and the camera according to checkerboard markers as the initial values ​​of the extrinsic parameters for online calibration; uses the analysis results and the initial values ​​of the extrinsic parameters to construct a 2D-3D association relationship between the camera and the laser radar data; uses an error construction method in combination with the representation method and the association relationship to obtain an optimization target for nonlinear optimization; and optimizes the backend by maintaining a sliding window with a fixed number of frames. Each optimization constructs mutual projections between frames through the laser radar pose, supplements the sparse point cloud of each frame through reprojection, and uses the error construction method to calculate the error of each point to implement inter-frame extrinsic parameter constraints. At the same time, the extrinsic parameters of multiple frames are optimized by a nonlinear optimization method that minimizes the error.

Description

Technical Field

[0001] The present invention belongs to the technical field of robots, and in particular relates to an online calibration method for external parameters between a laser radar and a camera in a road environment. Background Art

[0002] With the advancement of automotive technology and the sensor industry, people have a higher demand for automated driving, which has increased the continuous research on the application of autonomous driving in people's daily lives. The biggest difficulty in realizing autonomous driving technology is the improvement of robustness and maintainability. A complete autonomous driving system requires the coordinated fusion of multiple sensors. Compared with other sensors that directly measure vehicle motion information, lidar and cameras can perceive the surrounding environment and provide point clouds with depth information and images with dense textures respectively. The principle of collecting information makes them have the characteristics of independent frames in the original data, with the advantages of error-free accumulation and easy state estimation. The fusion of data is inseparable from coordinate transformation, and frequent coordinate transformation is inseparable from the accurate estimation of external parameters.

[0003] At present, there have been a lot of studies trying to solve the calibration problem between LiDAR and cameras. For example, the invention patent with patent application number CN202010084392.X discloses a calibration method based on spherical markers. By using a camera at the same time to capture a single-frame image of the spherical marker and a LiDAR to scan the point cloud data of the spherical marker, the two-dimensional coordinates of the center of the spherical marker in the camera coordinate system and the three-dimensional coordinates in the LiDAR coordinate system are obtained, thereby forming a pair of corresponding 2D-3D position data pairs, adjusting the posture of the spherical marker, and obtaining at least four pairs of corresponding 2D-3D position data pairs, thereby solving the conversion relationship between the LiDAR coordinate system and the camera coordinate system. However, this calibration method requires the target object to be fixed in advance, which has high requirements and is not suitable for real-time monitoring and estimation of external parameters. The invention patent with patent application number CN201810426821.X discloses an online calibration method based on the edge of laser radar and camera data. The method associates the edge extraction results of laser point cloud data with the image edge through the external parameters of offline calibration, and uses the information of the two sensors to conduct a comprehensive analysis of a scanned object to optimize the external parameters of the laser and camera. However, the operating range of autonomous driving covers the entire urban area. The urban environment is complex and changeable, with many dynamic objects. The robustness of the association method that only relies on the edge of the scanned object is not high, and it does not take into account the interference of other dynamic objects on the road and the requirements of the autonomous driving system for computing power and stability.

[0004] Therefore, the existing online calibration methods for road environments still have defects. Summary of the invention

[0005] In order to solve the above problems existing in existing robot applications, a method for online calibration of external parameters between a laser radar and a camera in a road environment is provided, which uses the semantic information of the road environment for online calibration.

[0006] In order to achieve the above-mentioned invention object, the present invention adopts the following technical solution: a method for online calibration of external parameters between a laser radar and a camera in a road environment, comprising the following steps:

[0007] Step 1: Analyze the representation and detection accuracy of the collected road environment semantic information in the laser radar point cloud segmentation results and the camera image detection results to obtain the analysis results; Step 2: Offline calibrate the external parameters between the laser radar and the camera according to the checkerboard markers as the initial values ​​of the external parameters for online calibration;

[0008] Step 3: Use the analysis results and initial values ​​of external parameters to build the 2D-3D relationship between camera and lidar data;

[0009] Step 4: Use the error construction method in combination with the representation method and the correlation relationship to obtain the optimization target of nonlinear optimization;

[0010] Step 5: Perform backend optimization and maintain a sliding window with a fixed number of frames. Each optimization builds mutual projections between frames through the lidar pose, supplements the sparse point cloud of each frame through reprojection, and uses the error designed in step 4 to calculate the error of each point to achieve inter-frame extrinsic parameter constraints. At the same time, optimize the multi-frame extrinsic parameters through a nonlinear optimization method that minimizes the error.

[0011] Furthermore, the road environment semantic information in step one is lane line, sign and lamp pole information provided by an upstream perception algorithm.

[0012] Furthermore, the step 1 is specifically as follows:

[0013] S11, analyzing the input road environment semantic information, combining the rules of objects in the actual scene, defining the road environment semantic information according to the three options of point, line and surface features, defining the road environment semantic information to be represented as pixel-level semantic connected blocks in image detection, and represented as annotated scattered points in point cloud segmentation;

[0014] S12. Set a threshold for the depth value of the semantic information of the road environment in the laser radar coordinates, and use the threshold to filter out the semantic information with low detection quality due to the long distance; since the signs and lamp poles in the semantic information of the road environment will be misdetected in the point cloud segmentation results, the signs and lamp poles are classified as one category, and the lane lines are classified as another category, and the two categories are associated separately to improve the detection accuracy.

[0015] Furthermore, the step 2 is specifically as follows: before the vehicle departs, a chessboard pattern plate with a known real pattern size is used to automatically extract the center point and normal vector of the chessboard pattern plate from the lidar point cloud, and the corresponding feature points in the image and point cloud are obtained from the external parameters of the camera and the lidar. The precise point correspondence relationship is obtained, and the external parameter model corresponding to the minimized point-to-point projection error is obtained through the principle of nonlinear optimization, and the external parameter results of the offline calibration of the external parameter model are used as the initial values ​​of the external parameters for online calibration.

[0016] Furthermore, the external parameter model is data of six dimensions of rotation and translation between the laser radar coordinate system and the camera coordinate system at this time.

[0017] Furthermore, the step three is specifically as follows:

[0018] S31, using the association method to perform 2D-3D association, since the point cloud obtained in step 1 is a scattered point, and the image is a pixel-level semantically connected block, combined with the initial value of the external parameter in step 2, the 3D scattered points are projected onto the 2D image plane to obtain the 2D projection point coordinates of each 3D point on the pixel plane;

[0019] S32. Calculate the nearest perception area for each 2D projection point to achieve the association between the 3D point and the 2D area.

[0020] Furthermore, in step S31, a matching association method between different modalities is adopted. Based on the analysis result in step one, it can be known that the image is a semantically connected block, the point cloud is a scattered point, and the semantically connected block and the scattered points are representations of different modalities.

[0021] Furthermore, the step 4 is specifically as follows:

[0022] S41, for the two categories of signs, lamp poles and lane lines in the semantic information of the road environment, for each 3D point, respectively calculate the distance value from its 2D projection point to the nearest perception area of ​​the same type as the direct distance error;

[0023] S42, using exponential distance error for signs and light poles; specifically, reading the minimum distance value from the 3D projection point in the non-perception area of ​​the image, that is, the background area, to the perception area, reading the minimum distance value from the projection point in the perception area to the perception area, and using an exponential function for the exponential distance error; at the same time, using the error at the boundary to connect the background area and the perception area;

[0024] S43, calculating the epipolar constraint error for the sign area whose area is larger than the preset value; specifically, extracting feature points from the large sign extraction area, matching the previous and next frames based on the optical flow method, each point matching pair needs to comply with the epipolar constraint, and using the epipolar constraint formula value as the epipolar constraint error.

[0025] Furthermore, the preset value in step S43 is 20 pixels square.

[0026] Furthermore, the step five is specifically as follows: maintaining a sliding window with a fixed number of frames, using the radar posture to perform coordinate conversion on the point clouds of different frames, converting them to the coordinate systems of other frames in the sliding window, and completing the mutual complementation of the point clouds between frames; performing 3D-2D projection on all points, and calculating the errors, thereby realizing mutual constraints between frames; optimizing the external parameters with a fixed number of frames in the sliding window at the same time each time the optimization is performed, and the initial value of the external parameters of each frame newly added to the sliding window is the result of one optimization of the previous frame.

[0027] The technical solution of the present invention has the following beneficial effects: the method utilizes the semantic information of three categories of relatively stable lane lines, signs and lamp poles on the road to perform online calibration of the laser radar and camera extrinsic parameters of the road environment, which not only ensures the staticity and maintainability of the extracted features, but also effectively reduces the amount of calculation and ensures the lightweight algorithm compared to the traditional method of extracting features using all the data information in the road environment; the lane lines are evenly distributed laterally within the viewing angle, and the longitudinal constraints of the lamp poles and signs are strong, which ensures the constraints of the features in six degrees of freedom; the point-to-region and point-to-point errors are designed mainly according to the errors of semantic information of different categories, and at the same time, through the laser radar posture, the multi-frame point cloud blocks and the perception area in the sliding window with a fixed frame size are mutually projected and associated to construct the same error term, and at the same time, the extrinsic parameters of the multi-frames in the sliding window are optimized to ensure the smoothness and accuracy of the inter-frame transformation of the extrinsic parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a flowchart of the online calibration algorithm of LiDAR and camera extrinsics based on road semantic information;

[0029] Figure 2 It is a schematic diagram of missed detection in image detection;

[0030] Figure 3a It is the effect of projecting 3D point cloud onto the imaging plane;

[0031] Figure 3b It is a binary image generated based on the image detection results;

[0032] Figure 4 is the potential energy map of distance inside and outside the semantic region required by the exponential distance error;

[0033] Figure 5a It is a mask that only retains some pixels of the sign;

[0034] Figure 5b It is to extract feature points only for the sign part;

[0035] Figure 5cIt is a schematic diagram of the horizontal connection lines of the matching points of the front and rear frames;

[0036] Figure 5d It is a schematic diagram of the longitudinal connection lines of the matching point pairs of the front and rear frames;

[0037] Figure 6 It is a schematic diagram of the optimizer design scheme;

[0038] Figure 7 It evaluates the convergence direction of external parameters from six degrees of freedom. DETAILED DESCRIPTION

[0039] The specific embodiments of the specific scheme of the present invention are further elaborated in conjunction with the accompanying drawings to make the technical scheme clearer and more understandable. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0040] like Figure 1-7 As shown, this embodiment relates to an online calibration method for external parameters between a laser radar and a camera in a road environment, which uses semantic information of the road environment and is mainly used in actual scenarios. Taking an L4 truck as a platform, a Velodyne HDL-32 linear rotating laser radar is configured with a vertical viewing angle of 40°, which can detect obstacles up to 80m away, and a PointGrey color monocular camera, which are connected to the vehicle through brackets. Data is collected at a frequency of 10Hz, and data synchronization is done in advance through hardware. Motion compensation of the point cloud is preprocessed. The data environment is a highway in North America, driving at an average speed of 50km / h. The error of the external parameter comes from the jitter of the sensor bracket when the truck body is running, which is consistent with rigid body motion.

[0041] The implementation process of this method is as follows.

[0042] Step 1. This method uses the semantic extraction results of 32-line laser radar and monocular camera, and intends to use the semantic information obtained by the two sensors to construct a 3D-2D correspondence, so as to calculate the external parameter value between the laser radar and the camera. Considering that the stable static object information in the dynamic environment is more conducive to constructing the correct 3D-2D correspondence, this method chooses to use three types of semantic information: signs, lane lines, and light poles. Traffic signs appear frequently in relatively complex intersection environments. The special pattern design makes its pixels more valuable. The design is often bright in color, and the shapes are mainly rectangular, circular, and triangular, which are easy to extract. Since the repeatability of signs is much lower than that of lane lines and light poles, and they mainly appear on both sides of the road and in complex intersections, they supplement the longitudinal constraints of semantic information and are very important semantic features. Lane lines appear frequently in urban environments, and their simple shapes play an important role in vehicle operation and sensor alignment. At the same time, lane lines are concentrated in the lower half of the picture, with a good distribution in the horizontal direction, but fewer longitudinal constraints. Light poles appear frequently and have a good distribution in the vertical direction, so they are also important semantic information.

[0043] The real-time semantic processing technology for 3D point clouds can be divided into two categories according to the representation method: semantic segmentation of point clouds represented as scattered points and semantic detection of point clouds represented as rectangular 3D detection frames. Considering that the semantic information in the form of 3D detection frames loses the contour structure of the object, only the matching 2D semantic information can be found for the 8 corner points of the detection frame, and due to the limited viewing angle of single-frame point cloud data, the size of the detection frame is inevitably different from the actual object situation, and such detection frame corner point association is not accurate enough. In addition, the rotation angle information provided by the detection frame is difficult to use for association with a monocular camera, because the semantic processing of the monocular camera cannot include the object posture information. In contrast, the semantic segmentation result of the point cloud in the form of scattered points retains the contour information of the object, all points can be used to build associations with 2D information, and the original radar scanning results are directly used, and the accuracy of the point information is higher. Therefore, the image processing framework of the present invention selects the real-time point cloud semantic segmentation result as the upstream input, selects the open source segmentation algorithm PolarNet without neighbors that takes into account both real-time performance and accuracy, and designs the 3D-2D association method to build associations for each 3D point.

[0044] Therefore, the semantic information of LiDAR is represented as follows: in addition to the three-dimensional information and reflection intensity of each point in the point cloud of each frame in the local coordinate system, each point also has a semantic annotation, indicating that the point belongs to a certain semantic object, including vehicles, pedestrians, signs, lamp poles, ground, vegetation, railings and other objects in various road environments. Then, the ground point cloud extracts the corner points of the dotted lane line and the point series of the solid lane line according to the reflection intensity.

[0045] Since real-time segmentation is used and only single-frame data is used, the accuracy is inevitably not ideal. A common problem is the misdetection between the sign and the pole supporting the sign. According to the principle of prior annotation, the sign should be one category, and the pole and the light pole should be one category, with different semantic annotations. However, in reality, it occasionally happens that the supporting pole is considered to be a sign, and the lower half of the sign near the pole is marked with the sign semantic annotation. These misdetections will have a great impact on the error design and optimization effect. When designing the error, you can consider putting the sign and the light pole together for association; while the lane line will not confuse the detection, it can be associated separately to improve the matching accuracy.

[0046] Existing image semantic extraction methods can be classified into semantic detection and semantic segmentation according to the representation form, which provide semantic information in the form of a minimum enclosing rectangle 2D detection frame and a scattered point form respectively. Point cloud semantic processing uses point cloud semantic information represented by scattered points, so it is necessary to calculate a corresponding 2D semantic information for each 3D point. For 2D semantic information in the form of a detection frame, the detection frame is usually larger than the actual pixel area occupied by the object. If the distance between two objects in the actual space is too close, the two detection frames on the imaging plane will have an intersection area, so the 3D points projected to the intersection area cannot determine which detection frame should correspond to. At the same time, the point cloud data is sparse and messy, and some abnormal scanning points on the edge of the object should be removed, but the detection frame loses the object contour information and cannot distinguish the abnormal points. The semantic segmentation technology of pixel-level understanding of images retains the contour information of the object, and each 3D point can be mapped to a unique semantic pixel point. Therefore, the present invention uses the real-time image semantic segmentation result as the upstream input. The real-time image semantic segmentation technology open source framework used in the present invention is Tusimple-DUC, a real-time image semantic segmentation technology based on FCN.

[0047] Therefore, the camera semantic information representation is as follows:

[0048] (1) Real-time detection of lane lines in images mainly uses bright colors to extract lane line pixels. Whether it is a solid or dotted lane line, it is detected in pixels to determine whether it belongs to the detection area. Multiple semantically connected blocks are obtained. The edges of actual semantically connected blocks are often not smooth and do not necessarily conform to quadrilaterals. In particular, the dotted lane line blocks in the distance have a very small pixel area and are often irregular polygons.

[0049] (2) In the image, the light poles often have a certain width variation and occupy multiple pixel blocks. Therefore, they are still represented by pixel-level semantically connected blocks and the semantic annotations are read pixel by pixel.

[0050] (3) Image semantic sign detection mainly uses special shapes and colors to extract pixels belonging to the sign from the complex background environment, and then fits the minimum circumscribed matrix and saves the pixel coordinates of the four corner points of the matrix.

[0051] As for the level of online semantic detection technology of images, some more specific features can be extracted. However, considering that the online detection error of radar is large and there is an edge expansion problem, since this embodiment only extracts semantic areas from images, and also because it is only based on the image data of the current frame, the accuracy needs to be improved. The biggest problem is missed detection. When the camera is far away from the lamp pole or sign, it is easy to miss detection, such as Figure 2 The sign in the distance on the right side of the center has a point cloud scan result, and is divided into semantic information labeled as a sign through point cloud segmentation technology. However, it is missed by the image detection technology because the pixel area it occupies on the imaging plane is too small. When different objects are close to each other on the imaging plane, only a part of them may be detected due to occlusion. Missed detection will have a great interference with the correctness of subsequent 3D-2D association. The threshold can be set by the projection distance, that is, by setting a threshold for the depth value of the semantic information in the lidar coordinates, to avoid interference caused by missed detection. At the same time, the dotted lane line in the distance will also be mis-extracted because the pixel distance between the dotted blocks is too close. When designing errors, it is necessary to consider the detection quality to reduce the negative impact of false detection.

[0052] Finally, the analysis results of the current road environment are obtained: various existing signs, lamp poles, lane lines, and lane lines include dotted lane lines and solid lane lines.

[0053] Step 2: The key to calibration is to establish accurate 2D-3D correspondence from sparse radar point clouds. Offline calibration is performed by placing black and white checkerboards of known size in advance. Since the colors are very bright, both the image and the point cloud can accurately extract the grid corners. Since the size of the checkerboard and the number of corner points are known, multiple pairs of 2D-3D accurate matching points can be obtained by minimizing the error between corner points:

[0054] P c =RP L +t

[0055]

[0056] loss = (x1-x2) 2 +(y1-2) 2

[0057] Where P L The 3D corner point extracted from a point cloud has its coordinates P in the camera coordinate system obtained through the external reference R,t between the laser radar and the camera c ,After converting to normalized coordinates, the pixel coordinates (x1, y1) on the camera imaging plane are obtained through the camera intrinsic parameters, and the square of the Euclidean distance with the 2D corner point (x2, y2) extracted from the corresponding image is calculated as the error, and the external parameter R,t model that minimizes the error is obtained.

[0058] Step 3: Use the external parameters obtained from the offline calibration in step 2 to associate the semantic information extracted from the point cloud with the image semantic information. First, use the external parameters obtained from the offline calibration as the initial value of the external parameters for the online calibration, and project the 3D points onto the image plane. For each 3D point in the point cloud, PL is the radar coordinate system at the scanning time, and can be converted to the camera coordinate system through the camera external parameter R,t to obtain point P':

[0059] P′=RP L +t

[0060] Then, the camera internal parameter K is used to reverse the pixel coordinates (u, v) based on the monocular imaging principle:

[0061]

[0062] Project all the point cloud extraction results of signs, light poles, and lane lines onto the image, as shown in Figure 3(a), and then find the nearest perceptual connected area. Because the external parameter error of offline calibration will not be particularly large, and each type of semantic information is also relatively scattered on a map, the association between nearby points and regions is generally not wrong. By extracting the perceptual connected areas of the image, a binary map of semantics is generated, as shown in Figure 3(b), where the white area is the perceptual area with a pixel value of 0, and the black area is the non-perceptual area with a non-zero pixel value. By calculating the distance from each non-zero point in the image to the nearest zero point, a distance potential map for each pixel point can be obtained to realize the association between the projection point and the pixel area block, and complete the construction of the 2D-3D association relationship between the camera and lidar data.

[0063] Step 4: This embodiment proposes three error modes:

[0064] (1) After the distance potential map is generated, each 3D point in the point cloud is projected to obtain a pixel value (u, v). The distance P(u, v) to the nearest perception area can be directly read from the distance potential map. This straight distance value is defined as the straight distance error. Its optimization goal is very direct, which is to make all the projected points fall within the perception area and make the sum of the distances as small as possible. A one-dimensional error can be constructed for each projected point.

[0065] (2) Direct distance For the point projected into the background area, the distance from the nearest point in the perception area is calculated. For the point projected into the perception area, the error is zero. This method can achieve good results for narrow lane lines and lamp poles. However, for signs with a certain area, even if the external parameters are disturbed, most of the points can fall within the perception area. The error value of this part is 0, which is a zero space and has no effect on optimization. Relying only on the small part of the edge points that fall outside the perception area to reduce the error will result in the property that the local optimal solution is not unique. The entire sign may be projected slightly upward or slightly to the left, and the total error value remains unchanged. Therefore, consider designing an exponential distance error that fully utilizes the points in the perception area: for the points projected into the background area, read the minimum distance value from the point to the perception area, and the error is taken as an exponential function with a base greater than 1. The larger the direct distance value, the larger the error and the faster the growth rate. For the points projected into the perception area, read the minimum distance value from the point to the background area, and the error is taken as an exponential function with a base of 0 to 1. In this way, the larger the direct distance error value, the smaller the error. The error at the boundary is 1 to smoothly connect the two parts.

[0066]

[0067] Where L represents the error, p is the point where the sign is projected inside the sensing area, m(p) is the minimum distance from the read point to the background area, α is the exponential function base which is a number between 0 and 1, q is the point where the sign is projected outside the sensing area, n(q) is the minimum distance from the read point to the sensing area, and β is the exponential function base which is a number greater than 1. Therefore, two potential energy diagrams need to be generated, such as Figure 4 , depending on whether the projection position belongs to the perception area, read the distance value on different potential energy graphs.

[0068] (3) Both the radial distance error and the exponential distance error are weak constraints from point to region, and there is no strict one-to-one correspondence. Try to use some other image pixel information to extract some point-to-point correspondences and establish connections with external parameters. Traffic signs have a large area in the image, and the patterns are simple and clear. Different sign patterns are highly distinguishable, and it is easy to extract feature points. Try to use only the pixels within the rectangular frame of the sign to extract FAST feature points, and match the previous and next frames based on the optical flow method. Because it is known that the extracted points are sign points, they conform to the mapping relationship model between the two planes, and the false matches can be screened out based on the basic matrix and the homography matrix. First, only use the sign image detection results to generate a mask, such as Figure 5a , and superimposed with the image, only the pixels inside the sign in the image are retained. Because the epipolar constraint has extremely high pixel accuracy requirements, some signs that are too far away and too small in the image are filtered out, and only key frames with enough extracted feature points are retained, such as Figure 5bAfter testing, even if only a small number of pixels are retained to extract feature points, 150-300 pairs of matching pairs can still be obtained when the distance between the signs is appropriate. Figure 5c , Figure 5d After obtaining the pixel matching point pairs, each matching point pair of the previous and next frames should meet the epipolar constraints:

[0069]

[0070] Where K represents the camera intrinsic parameter, R,t is the transformation between camera k and k+1 frames, and p1 and p2 are the pixel coordinates of a pair of matching points. The input information of the system contains relatively accurate radar inter-frame poses, and the inter-frame camera pose transformation can be obtained through two frames of extrinsic parameters. The present invention uses the value of this epipolar constraint formula as the epipolar constraint error, which should be as close to 0 as possible through optimization.

[0071] Step 5: Nonlinear optimization puts all variables to be optimized into a state variable:

[0072] χ={x1,x2,…,x n}={(R1,t1),(R2,t2),…,(R n , t n )}

[0073] Each observation has Z k，j =h(P j，Xk )+V k，j , where Pj represents each 3D point, V k，j Represents the noise term. Assuming that the noise conforms to the Gaussian distribution, the maximum likelihood estimate of the state can be obtained according to its probability density function expansion form:

[0074]

[0075] This formula is equivalent to minimizing the square of the noise term, that is, a least squares problem. The objective function of the whole problem is composed of multiple square sums of errors as above. The entire state variable dimension is very high, but each error term is only related to one or two state variables, such as the straight distance error and the exponential distance error are only related to the current frame external parameters, and the epipolar constraint error is related to the external parameters of the two frames before and after. It is a small-scale constraint, so it is only necessary to put the small Jacobian matrix of a single error term into the corresponding part of the overall Jacobian matrix. The incremental equation solution of the overall error is sparse, so even if the scale is large, it can be solved. The present invention will use the Ceres optimization library, which is a commonly used optimization algorithm library based on graph optimization, which can independently define parameter blocks (optimization variables corresponding to each error term), least squares error terms, Jacobian matrices, and solution process parameters.

[0076] In the backend optimization, a sliding window of fixed size is maintained, and each camera frame is saved as a Frame class, which includes the current frame's lidar pose, external parameters from the radar to the camera, camera internal parameters, 3D point clouds of lane lines, signs, and light poles in the radar segmentation results, and the straight distance error potential map obtained by image segmentation. Each optimization builds mutual projections between frames through the lidar pose, external parameters from the radar to the camera (optimization variables), and camera internal parameters, that is, the current 3D-2D projection is performed, which is equivalent to supplementing the sparse point cloud of each frame, and achieving the purpose of optimization through different error designs.

[0077] In the process of optimizing the design, this embodiment considers: (1) when the sliding window size is designed to be 10, the number of points projected to each other is often in the order of 20,000 to 30,000, that is, there are 20,000 to 30,000 errors, each error corresponds to a 1*70 Jacobian matrix, and the number of optimizations is set to 50 to have a better optimization effect. Such a large amount of calculation cannot guarantee real-time optimization; secondly, at the speed of the experimental vehicle, usually only the semantics of the previous five frames will be projected onto the current frame imaging plane, so the sliding window size of the present invention is designed to be 5. (2) If the association between adjacent frames is only a projection between frames, then there is no mutual constraint on the optimization variable values, and there will be the disadvantage of unlimited changes between adjacent frames.

[0078] Therefore, the optimizer designed in this embodiment optimizes the external parameters of 5 frames in the sliding window at the same time each time, and the initial value of the external parameter of a frame newly added to the sliding window is the result of the optimization of the previous frame, such as Figure 6 Such an optimization design not only ensures rationality but also provides a weak constraint on the numerical value of the inter-frame external parameters. The amount of calculation is relatively small and can ensure real-time.

[0079] Since the vehicle jitter is irregular, the true value cannot be obtained. Therefore, the evaluation method in this embodiment is: manually add noise to the initial value of the external parameter. According to the jitter error model, the error is mainly concentrated in the x and y directions. Through optimization, it should converge quickly in the correct direction and float around the external parameter value calibrated offline. If there is such a convergence trend and the floating range is consistent with the error magnitude, it means that the optimization is effective, and the convergence of the six degrees of freedom is observed and analyzed, such as Figure 7 , the six dimensions can basically be guaranteed to float between noise and true values, x, y, z are kept within ±10cm of the true value, roll, yaw, and pitch are kept within ±0.1rad. According to the jitter tolerance of the sensor mounting frame, this accuracy is in line with the actual error range.

[0080] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made to those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims. More specifically, within the scope of the present application disclosure, drawings and claims, various modifications and improvements may be made to the components and / or layout of the subject combination layout. In addition to the modifications and improvements made to the components and / or layout, other uses will also be obvious to those skilled in the art.

Claims

1. A method for online calibration of external parameters between a laser radar and a camera in a road environment, characterized by: The following steps are included: Step 1: Analyze the representation and detection accuracy of the collected road environment semantic information in the laser radar point cloud segmentation results and the camera image detection results to obtain the analysis results; Step 2: perform offline calibration of the external parameters between the laser radar and the camera according to the checkerboard markers, which serve as the initial values ​​of the external parameters for online calibration; Step 3: Use the analysis results and initial values ​​of external parameters to build the 2D-3D relationship between camera and lidar data; Step 4: Use the error construction method in combination with the representation method and the correlation relationship to obtain the optimization target of nonlinear optimization; Step 5: Perform backend optimization and maintain a sliding window with a fixed number of frames. Each optimization builds mutual projections between frames through the LiDAR pose, supplements the sparse point cloud of each frame through reprojection, and uses the error designed in step 4 to calculate the error of each point to implement inter-frame extrinsic parameter constraints. At the same time, optimize the extrinsic parameters of multiple frames through a nonlinear optimization method that minimizes the error. The step three is specifically as follows: S31, using the association method to perform 2D-3D association, since the point cloud obtained in step 1 is a scattered point, and the image is a pixel-level semantically connected block, combined with the initial value of the external parameter in step 2, the 3D scattered points are projected onto the 2D image plane to obtain the 2D projection point coordinates of each 3D point on the pixel plane; S32, calculating the nearest perception area for each 2D projection point, and realizing the association between the 3D point and the 2D area; The step 4 is specifically as follows: S41, for the two categories of signs, lamp poles and lane lines in the semantic information of the road environment, for each 3D point, respectively calculate the distance value from its 2D projection point to the nearest perception area of ​​the same type as the direct distance error; S42. Use exponential distance error for signs and light poles; Specifically, the minimum distance value from the 3D projection point falling in the background area to the perception area is read, and the minimum distance value from the projection point falling in the perception area to the perception area is read. The exponential distance error uses an exponential function; at the same time, the error at the boundary is used to connect the background area and the perception area. S43, calculating the epipolar constraint error for the sign area whose area is larger than the preset value; specifically, extracting feature points from the large sign extraction area, matching the previous and next frames based on the optical flow method, each point matching pair needs to comply with the epipolar constraint, and using the epipolar constraint formula value as the epipolar constraint error.

2. The method for online calibration of external parameters between a laser radar and a camera in a road environment according to claim 1, characterized in that: The road environment semantic information in step 1 is the lane line, sign and lamp pole information provided by the upstream perception algorithm.

3. The method for online calibration of external parameters between a laser radar and a camera in a road environment according to claim 2, characterized in that: The step 1 specifically comprises: S11, analyzing the input road environment semantic information, combining the rules of objects in the actual scene, defining the road environment semantic information according to the three options of point, line and surface features, defining the road environment semantic information to be represented as pixel-level semantic connected blocks in image detection, and represented as annotated scattered points in point cloud segmentation; S12. Set a threshold for the depth value of the semantic information of the road environment in the laser radar coordinates, and use the threshold to filter out the semantic information with low detection quality due to the long distance; since the signs and lamp poles in the semantic information of the road environment will be misdetected in the point cloud segmentation results, the signs and lamp poles are classified as one category, and the lane lines are classified as another category, and the two categories are associated separately to improve the detection accuracy.

4. The method for online calibration of external parameters between a laser radar and a camera in a road environment according to claim 3, characterized in that: The step 2 specifically comprises: Before the vehicle sets off, a chessboard pattern plate with known real pattern size is used to automatically extract the center point and normal vector of the chessboard pattern plate from the lidar point cloud. The corresponding feature points in the image and point cloud are obtained from the external parameters of the camera and lidar. The precise point correspondence is obtained, and the external parameter model corresponding to the minimized point-to-point projection error is obtained through the principle of nonlinear optimization. The external parameter results of the offline calibration of the external parameter model are used as the initial values ​​of the external parameters for online calibration.

5. The method for online calibration of external parameters between a laser radar and a camera in a road environment according to claim 4, characterized in that: The external parameter model is the data of six dimensions of rotation and translation between the laser radar coordinate system and the camera coordinate system.

6. The method for online calibration of external parameters between a laser radar and a camera in a road environment according to claim 1, characterized in that: In the step S31, a matching association method between different modalities is adopted. Based on the analysis result in step 1, it can be known that the image is a semantically connected block and the point cloud is a scattered point. The semantically connected block and the scattered points are representation forms of different modalities.

7. The method for online calibration of external parameters between a laser radar and a camera in a road environment according to claim 1, characterized in that: The preset value in step S43 is 20 pixels square.

8. The method for online calibration of external parameters between a laser radar and a camera in a road environment according to claim 1, characterized in that: The step five is specifically as follows: Maintain a sliding window with a fixed number of frames, use the radar posture to convert the coordinates of the point clouds of different frames, and convert them to the coordinate systems of other frames in the sliding window to complete the mutual complementation of the point clouds between frames; perform 3D-2D projection on all points and calculate the errors to achieve mutual constraints between frames; each optimization also optimizes the external parameters with a fixed number of frames in the sliding window, and the initial value of the external parameters of each frame newly added to the sliding window is the result of one optimization of the previous frame.

Images