Laser radar and camera joint calibration method based on background point cloud refinement processing

Abstract

The application discloses a kind of based on background point cloud fine processing laser radar and camera joint calibration method, comprising: using laser radar and visible light camera to build experimental device, and construct the optimization equation for solving calibration matrix;The foreground point cloud of the calibration object scanned by laser radar is fitted as a straight line;For the straight line fitted, obtain the background point cloud of laser radar in the same scan line with it;According to horizontal angle, solve the edge point of calibration object;The three-dimensional feature point of calibration object is solved using the obtained edge point;The three-dimensional feature point obtained and the corresponding two-dimensional feature point obtained by visible light camera are brought into optimization equation to solve calibration matrix.The application can obtain the edge point of calibration object scanned by laser radar with high precision and solve accurate three-dimensional feature point, avoid the problem of insufficient feature point accuracy caused by low horizontal resolution of laser radar, thereby improve joint calibration accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot vision and sensor calibration, and particularly relates to a laser radar and camera joint calibration method based on background point cloud fine processing. BACKGROUND

[0002] Information fusion of laser radar and visible light camera is widely used in three-dimensional reconstruction, autonomous navigation and positioning, and unmanned vehicles in the field of robot vision. A single sensor has limitations and cannot meet complex environmental perception tasks. For example, a visible light camera is easily affected by light and lacks environmental depth information, and laser radar has sparse data points and lacks color information. The fusion of the two can make up for their respective defects and achieve more accurate environmental perception.

[0003] Calibration of the external parameters of laser radar and visible light camera is the basis for external information fusion. The mainstream external parameter calibration of laser radar and visible light camera is based on calibration objects. Based on calibration objects, a geometrically constrained calibration object is used to obtain three-dimensional feature points and their corresponding two-dimensional feature points for external parameter calculation. The depth discontinuity of laser radar at the edge is used to extract points with distance mutations as edge points for calibration feature points, and the edge information extracted from the image is registered. However, due to the influence of the horizontal angle resolution of laser radar, the points located at the edge of the object will have distance mutations, so that the real edge points cannot be well positioned, which greatly affects the accuracy of subsequent feature point solving and makes the calibration effect poor and unstable.

[0004] As shown in Figure 1 Due to the existence of the horizontal resolution angle of laser radar, it cannot well measure the edge points of the object.

[0005] The straight line in the near distance is the object to be measured, referred to as the foreground; the straight line in the far distance is the non-measured object, referred to as the background; the thicker points are the measurement points of the laser radar, falling in the foreground are referred to as foreground edge points, and falling in the background are referred to as background edge points; the thinner points are the actual edge points of the object to be measured, referred to as real edge points; and θ is the horizontal resolution angle of the laser radar, i.e. the theoretical angle between the two closest scanning rays.

[0006] Due to the existence of the horizontal resolution angle, the laser radar cannot correctly capture the real edge points. The distance between the thicker foreground edge points and the thinner real edge points is defined as the theoretical error, which is briefly analyzed as shown in Figure 2

[0007] ​The straight lines l1 and l2 are two adjacent laser radar scanning lines, h is the distance of the laser radar to the foreground, d is the horizontal distance of the far scanning line l1 to the position of the laser radar, and e is the theoretical error. According to the geometric relationship, the relationship between e, h, d and theta is as follows:

[0008]

[0009] Taking d as a fixed length of 500 mm, h is 3000 mm, 6000 mm, 10000 mm and 20000 mm respectively, and theta is 0.1°, 0.2° and 0.4° respectively, the above theoretical error e is obtained as follows:

[0010] Table 1 Value of e when d = 500 mm

[0011] θ (°) \ h (mm) 3000 6000 10000 20000 0.1 5.380 10.543 17.495 34.927 0.2 10.757 21.083 34.988 69.851 0.4 21.501 42.155 69.964 139.69

[0012] Since the object measures the left and right edges, as shown in Figure 3 , the maximum theoretical error needs to be enlarged by two times, as shown in the following table.

[0013] Table 2 Maximum theoretical error enlarged by two times

[0014] θ (°) \ h (mm) 3000 6000 10000 20000 0.1 5.380*2 10.543*2 17.495*2 34.927*2 0.2 10.757*2 21.083*2 34.988*2 69.851*2 0.4 21.501*2 42.155*2 69.964*2 139.69*2

[0015] It can be found from the above table that the inherent property (angular resolution) of the laser radar causes it to fail to correctly capture the edge points of the object to be measured. In addition, the above data is based on the fixed angular resolution (i.e. only 0.1 / 0.2 / 0.4 three values), in fact, the angular resolution of the laser radar is not fixed, and the above maximum theoretical error will be larger in practice. The error caused by the horizontal resolution angle of the laser radar will affect the solution of the three-dimensional feature points.

[0016] Therefore, the accurate solution of the three-dimensional feature points is the key to the joint calibration of the laser radar and the visible light camera, and is the primary problem to be solved in the joint calibration of the laser radar and the visible light camera. SUMMARY

[0017] In view of the above problems, the present application provides a laser radar and camera joint calibration method based on background point cloud fine processing, introduces the background point cloud of the calibration object, and performs fine processing on the foreground point cloud to solve the edge points. The accurate edge points are used to solve the feature points of the calibration object, and the feature points are used to establish an optimization equation with the re-projection error as the optimization index to solve the joint calibration parameters, thereby improving the joint calibration accuracy.

[0018] To solve the above technical problems, embodiments of the present application provide the following solutions:

[0019] A laser radar and camera joint calibration method based on background point cloud refinement processing, comprising the following steps:

[0020] S1, an experimental device is built by using a laser radar and a visible light camera, and an optimization equation for solving a calibration matrix is constructed;

[0021] S2, the foreground point cloud of the calibration object scanned by the laser radar is fitted as a straight line;

[0022] S3, for the fitted straight line, the laser radar background point cloud in the same scanning line is obtained;

[0023] S4, the edge points of the calibration object are solved according to the horizontal angle;

[0024] S5, the three-dimensional feature points of the calibration object are solved by using the obtained edge points;

[0025] S6, the obtained three-dimensional feature points and the corresponding two-dimensional feature points obtained by the visible light camera are brought into the optimization equation to solve the calibration matrix.

[0026] Preferably, in the step S1, the experimental device is placed in front of the calibration object, the calibration object is a square plate placed at an angle of 45°, located in the field of view of the laser radar and the visible light camera, and a background wall is arranged behind the calibration object.

[0027] Preferably, in the step S1, the optimization equation for solving the calibration matrix comprises:

[0028] The conversion from the three-dimensional feature points (x, y, z) to the two-dimensional feature points (u, v) is represented as formula (1):

[0029]

[0030] Where s is a scaling factor, K is a visible light camera intrinsic matrix obtained in advance, and an optimal calibration matrix [R * |T * ] needs to be solved, satisfying formula (2):

[0031]

[0032] Where R * is the best rotation matrix, T * is the best translation matrix, P is a three-dimensional feature point set, p i (x i , y i , z i ) ∈ P; Q is a two-dimensional feature point set, q i (u i , v i ) ∈ Q; h(·) is a two-dimensional distance function, as shown in formula (3):

[0033]

[0034] Formula (2) above can be transformed into an optimization problem as shown in formula (4):

[0035]

[0036] Using f(Q, P) as the optimization equation, the calibration matrix [R] is obtained by solving it. * |T * ].

[0037] Preferably, in step S2, the foreground cloud of the i-th frame of the calibration object is fitted into a straight line. i Two symbols are defined to describe the two steps of line fitting: the addition symbol ⊕ and the deletion symbol. The addition symbol ⊕ indicates the scan line of the i-th frame before merging; the deletion symbol... This indicates outlier removal from the data after the ⊕ operation; the foreground cloud information of the i-th frame obtained from the above two steps is then fitted with a RANSAC line to obtain the straight line line. i The information is as follows:

[0038]

[0039]

[0040]

[0041] in, Let (x, y, z) be the direction vector of a line, and any point (x, y, z) on the line can be represented by (x0, y0, z0) and (x0, y0, z0). This indicates that the fitted straight line will be used subsequently. i Find the edge point p m (x m ,y m , z m ).

[0042] Preferably, in step S3, for the fitted straight line... i Based on the ID information of each scan line given by the LiDAR, the background point cloud of the LiDAR on the background wall that is on the same scan line is found, which is used to extract the background edge points.

[0043] Preferably, in step S4, on the straight line... i Find the points with horizontal angles α and β respectively as the background edge points p. b and foreground edge point p f The projection point;

[0044]

[0045] Specifically, p b The horizontal angle α passes through p b 3D coordinates (x) b ,y b , z b The result is obtained by solving formula (6), and then t can be solved by substituting t into formula (5) to obtain p. b The projection point p b' Similarly, p can be solved. f The projection point p f' ; will p b' and p f' The geometric midpoint p m Save p as the edge point of the current frame. m To the set of edge points S edge In the middle; continue to obtain the line of the next frame. i+1 After the same processing, the edge points of the new frame are also saved to the edge point set S. edge In the process, until a specified number of frames is reached; solve for the set of edge points S. edge The centroid of the scan line is taken as the true edge point and saved to edgepoints; similarly, the edge points of the other scan lines are solved and saved to edgepoints.

[0046] Preferably, in step S5, the calibration object is a square plate, and the edge points of the four sides of the calibration object are obtained through the above steps and all are saved in edgepoints; the obtained edge points are fitted to the four side lines l1-l4 of the calibration object using the least squares method, and the three-dimensional feature points p1-p4 are represented by the midpoint of the perpendicular bisector of the two spatial lines.

[0047] Preferably, in step S6, the position of the calibration object is changed, and more three-dimensional feature points are obtained in the same way. The obtained three-dimensional feature points and the corresponding two-dimensional feature points obtained by the visible light camera are substituted into the optimization equation shown in formula (4) to solve the calibration matrix.

[0048] Preferably, in step S6, two-dimensional feature points are extracted from the image acquired by the visible light camera using edge detection and / or corner detection methods.

[0049] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0050] In this embodiment of the invention, the background point cloud of the LiDAR-scanned calibration object is utilized and refined together with the foreground point cloud to obtain accurate edge points of the calibration object. Linear fitting is performed on the edge points to obtain the three-dimensional feature points on the calibration object. An optimization equation is established using the three-dimensional feature points obtained from multi-view measurements and their corresponding two-dimensional feature points, minimizing the reprojection error to obtain the joint calibration result. This invention can extract the edge points of the LiDAR scan on the calibration object with high precision and robustness, avoiding the problem of insufficient feature point accuracy caused by the low horizontal angular resolution of the LiDAR. By obtaining accurate edge points, more accurate calibration results can be obtained, thereby improving the joint calibration accuracy. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a schematic diagram of the horizontal resolution angle of a lidar.

[0053] Figure 2 This is a schematic diagram of theoretical error analysis for lidar;

[0054] Figure 3 This is a schematic diagram showing the measurement of the left and right edges of an object;

[0055] Figure 4 This is a flowchart of the joint calibration method for lidar and camera based on background point cloud refinement provided in the embodiments of the present invention;

[0056] Figure 5 This is a schematic diagram illustrating the specific implementation process of the joint calibration method provided in this embodiment of the invention;

[0057] Figure 6 This is a three-dimensional schematic diagram of the experimental apparatus in an embodiment of the present invention;

[0058] Figure 7 This is a schematic diagram of the arrangement of the experimental apparatus in an embodiment of the present invention;

[0059] Figure 8a and Figure 8b These are the front view and left view of the experimental apparatus and edge point positions provided in the embodiments of the present invention;

[0060] Figure 9a and Figure 9b These are feature point distribution maps and edge point distribution maps provided in embodiments of the present invention.

[0061] As shown in the figure, specific structures and devices are marked in the figure to clearly illustrate the structure of the embodiments of the present invention. However, this is only for illustrative purposes and is not intended to limit the present invention to the specific structure, device and environment. According to specific needs, those skilled in the art can adjust or modify these devices and environments, and such adjustments or modifications are still included in the protection scope of the present invention. Detailed Implementation

[0062] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0063] Embodiments of the present invention provide a joint calibration method for lidar and camera based on background point cloud refinement processing, such as... Figure 4 and Figure 5 As shown, the method includes the following steps:

[0064] S1. Build an experimental setup using lidar and a visible light camera, and construct an optimization equation for solving the calibration matrix.

[0065] In this step, the experimental setup is first constructed and brought to a stable working state, such as... Figure 6 and Figure 7 As shown, the lidar and visible light camera are rigidly connected and placed in front of a calibration object. The calibration object is a flat square plate, placed at a 45° angle, within the field of view of both the lidar and the visible light camera. A background wall is placed behind the calibration object. The front view and left view of the arrangement of the lidar, visible light camera, and calibration object are shown below. Figure 8a and Figure 8b As shown.

[0066] The optimization equations used to solve the calibration matrix include:

[0067] The transformation from three-dimensional feature points (x, y, z) to two-dimensional feature points (u, v) is expressed by formula (1):

[0068]

[0069] Where s is the scaling factor, K is the pre-acquired intrinsic parameter matrix of the visible light camera, and an optimal calibration matrix [R] needs to be solved. * |T * ], or the optimal rigid transformation matrix [R] * |T * ], satisfying the formula (2):

[0070]

[0071] Where R * For the optimal rotation matrix, T * Let P be the optimal translation matrix, and P be the set of 3D feature points. i (x i y i , z i )∈P; Q is a set of two-dimensional feature points, q i (u i v i )∈Q; h(·) is a two-dimensional distance function, as shown in formula (3):

[0072]

[0073] Formula (2) above can be transformed into an optimization problem as shown in formula (4):

[0074]

[0075] Using f(Q, P) as the optimization equation, the calibration matrix [R] is obtained by solving it. * |T * ].

[0076] Figure 9a and Figure 9b This is a distribution diagram of feature points and edge points in an embodiment of the present invention. As can be seen from the diagram, p1-p4 are the feature points to be acquired; these four points are the points where the four lines l1-l4 intersect pairwise. In visible light camera images, two-dimensional feature points can be easily extracted using edge detection and / or corner detection methods. However, three-dimensional feature points require the intersection of 3D spatial edge lines to be obtained. Since l1-l4 are lines in space, their intersection is represented by the midpoint of the perpendicular bisector of two spatial lines, a function already available in the Point Cloud Library. The key to the method of this invention lies in how to obtain accurate edge points to fit the lines l1-l4, thereby solving for the three-dimensional feature points.

[0077] S2. Fit the foreground cloud of the calibration object scanned by the lidar to a straight line.

[0078] In this step, the foreground cloud of the i-th frame of the calibration object is fitted into a straight line. i Two symbols are defined to describe the two steps of line fitting: the addition symbol ⊕ and the deletion symbol. The ⊕ symbol indicates the scan lines of the i-th frame before merging, for example... Figure 8a In the frame i Applying a ⊕ symbol at a certain position merges the scan lines of the previous i frames together; deleting the symbol... This indicates outlier removal from the data after the ⊕ operation; the foreground cloud information of the i-th frame obtained from the above two steps is then fitted with a RANSAC line to obtain the straight line line. i The information is as follows:

[0079]

[0080]

[0081]

[0082] in, Let (x, y, z) be the direction vector of a line, and any point (x, y, z) on the line can be represented by (x0, y0, z0) and (x0, y0, z0). This indicates that the fitted straight line will be used subsequently. i Find the edge point p m (x m ,y m , z m ).

[0083] S3. For the fitted straight line, obtain the background point cloud of the LiDAR that is on the same scan line.

[0084] In this step, for the fitted straight line... i Based on the ID information of each scan line provided by the LiDAR, the background point cloud of the LiDAR on the same scan line is found on the background wall and used to extract background edge points, for reference. Figure 9b .

[0085] S4. Solve for the edge points of the calibration object based on the horizontal angle.

[0086] In this step, on the line i Find the points with horizontal angles α and β respectively as the background edge points p. b and foreground edge point p f The projection point;

[0087]

[0088] Specifically, p b The horizontal angle α passes through p b 3D coordinates (x) b ,y b , z b The result is obtained by solving formula (6), and then t can be solved by substituting t into formula (5) to obtain p. b The projection point p b' ,like Figure 8b The point in the middle circle; similarly, solve for p. f The projection point p f' Because pf' exist Figure 8b The position in and p f Similar, not marked; p b' and p f' The geometric midpoint p m Save p as the edge point of the current frame. m To the set of edge points S edge In the middle; continue to obtain the line of the next frame. i+1 After the same processing, the edge points of the new frame are also saved to the edge point set S. edge In the middle, until a specified number of frames is reached, such as line 20 Solve for the set of edge points S edge The centroid of the scan line is taken as the true edge point and saved to edgepoints; similarly, the edge points of the other scan lines are solved and saved to edgepoints.

[0089] The algorithm for solving the edge points is as follows:

[0090]

[0091] S5. Use the obtained edge points to solve for the three-dimensional feature points of the calibration object.

[0092] In this step, the edge points of the four sides of the calibration object are obtained through the above steps and all are saved in edgepoints. The obtained edge points are then fitted to the four side lines l1-l4 of the calibration object using the least squares method. The midpoints of the perpendicular bisectors of the two spatial lines are used to represent the three-dimensional feature points p1-p4, as follows: Figure 9a As shown.

[0093] S6. Substitute the obtained three-dimensional feature points and the corresponding two-dimensional feature points obtained by the visible light camera into the optimization equation to solve for the calibration matrix.

[0094] In this step, the position of the calibration object is changed, and more three-dimensional feature points are obtained in the same way. The obtained three-dimensional feature points and the corresponding two-dimensional feature points obtained by the visible light camera are substituted into the optimization equation shown in formula (4) to solve the calibration matrix.

[0095] This invention utilizes the background point cloud of the calibration object scanned by a lidar scanner. It performs refined edge processing on this background point cloud and the foreground point cloud to obtain points closer to the calibration object's edge, and then fits a straight line to obtain accurate 3D feature points. By acquiring 2D and 3D feature points of the calibration object from images captured by a visible light camera and point clouds scanned by the lidar scanner, respectively, optimization equations are established using reprojection error as the optimization criterion to find the optimal rotation and translation matrices. This invention can obtain the edge points of the calibration object scanned by the lidar with high accuracy and solve for accurate 3D feature points, avoiding the problem of insufficient feature point accuracy caused by the inherent low horizontal resolution of lidar, thereby improving the joint calibration accuracy.

[0096] It should be noted that the use of terms such as "an embodiment," "an embodiment," "an exemplary embodiment," and "some embodiments" in the specification indicates that the described embodiment may include a specific feature, structure, or characteristic, but not every embodiment necessarily includes that specific feature, structure, or characteristic. Furthermore, when a specific feature, structure, or characteristic is described in conjunction with an embodiment, implementing such a feature, structure, or characteristic in conjunction with other embodiments (whether explicitly described or not) should be within the knowledge of those skilled in the art.

[0097] Generally, terms can be understood at least partly from their use in context. For example, depending at least partly on the context, the term "one or more" as used herein can be used to describe any feature, structure, or characteristic in a singular sense, or a combination of features, structures, or characteristics in a plural sense. Additionally, the term "based on" can be understood not necessarily to convey an exclusive set of factors, but rather, alternatively, depending at least partly on the context, to allow for the presence of other factors that are not necessarily explicitly described.

[0098] As used herein, the term "nominal / nominally" refers to the expected or target value of a characteristic or parameter for the operation of a component or process, set during the design phase of the production or manufacturing process, and the range of values ​​higher and / or lower than the expected value. The range of values ​​may be due to slight variations in the manufacturing process or tolerances. As used herein, the term "about" indicates a value of a given quantity that can vary based on a specific technology node associated with the subject semiconductor device. Based on a specific technology node, the term "about" can indicate a value of a given quantity that varies, for example, within 5% to 15% of the value (e.g., ±5%, ±10%, or ±15% of the value).

[0099] It is understood that the meanings of “on”, “above” and “above” in this disclosure should be interpreted in the broadest sense, such that “on” means not only “directly on” something, but also includes something with an intermediary feature or layer, and that “above” or “above” means not only “on” something, but also includes something “above” or “above” without an intermediary feature or layer.

[0100] Furthermore, spatially related terms such as “below,” “under,” “lower,” “above,” and “upper” are used herein for convenience to describe the relationship of one element or feature to one or more other elements or features, as illustrated in the accompanying drawings. Spatially related terms are intended to cover different orientations in the use or operation of the device other than those depicted in the accompanying drawings. The device may be oriented in other ways, and the spatially related descriptive terms used herein can be interpreted similarly.

[0101] This invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of this invention. To provide the public with a thorough understanding of this invention, specific details are described in detail in the following preferred embodiments; however, those skilled in the art will fully understand the invention even without these details. Furthermore, to avoid unnecessary misunderstanding of the essence of this invention, well-known methods, processes, procedures, components, and circuits are not described in detail.

[0102] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc.

[0103] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A joint calibration method for lidar and camera based on refined background point cloud processing, characterized in that, Includes the following steps: S1. Build an experimental setup using lidar and a visible light camera, and construct an optimization equation for solving the calibration matrix; In step S1, constructing the optimization equation for solving the calibration matrix includes: The transformation from three-dimensional feature points (x, y, z) to two-dimensional feature points (u, v) is expressed by formula (1): Where s is the scaling factor, K is the pre-acquired intrinsic parameter matrix of the visible light camera, and an optimal calibration matrix [R] needs to be solved. * |T * ], satisfying the formula (2): Where R * For the optimal rotation matrix, T * Let P be the optimal translation matrix, and P be the set of 3D feature points. i (x i ,y i ,z i )∈P; Q is a set of two-dimensional feature points, q i (u i ,v i )∈Q; h(·) is a two-dimensional distance function, as shown in formula (3): Formula (2) above can be transformed into an optimization problem as shown in formula (4): Using f(Q,P) as the optimization equation, the calibration matrix [R] is obtained by solving it. * |T * ]; S2. Fit the foreground cloud of the calibration object scanned by the lidar to a straight line; S3. For the fitted straight line, obtain the background point cloud of the LiDAR that is on the same scan line; S4. Determine the edge points of the calibration object based on the horizontal angle; S5. Use the obtained edge points to solve for the three-dimensional feature points of the calibration object; S6. Substitute the obtained three-dimensional feature points and the corresponding two-dimensional feature points obtained by the visible light camera into the optimization equation to solve for the calibration matrix.

2. The joint calibration method for lidar and camera based on background point cloud refinement processing according to claim 1, characterized in that, In step S1, the experimental device is placed in front of the calibration object, which is a square plate placed at a 45° angle and located within the field of view of the lidar and the visible light camera. A background wall is provided behind the calibration object.

3. The joint calibration method for lidar and camera based on background point cloud refinement processing according to claim 1, characterized in that, In step S2, the foreground cloud of the i-th frame of the calibration object is fitted into a straight line. i Two symbols are defined to describe the two steps of line fitting: the increment symbol and the decrement symbol. and deletion symbol Add symbol Indicates the scan lines of the i-th frame before merging; delete symbols. Indicates the process After processing, outlier points are removed from the data; the foreground cloud information of the i-th frame obtained from the above two steps is then fitted with a RANSAC line to obtain the straight line. i The information is as follows: in, Let (x, y, z) be the direction vector of a line, and any point (x, y, z) on the line can be represented by (x0, y0, z0) and (x0, y0, z0). This indicates that the fitted straight line will be used subsequently. i Find the edge point p m (x m ,y m ,z m ).

4. The joint calibration method for lidar and camera based on background point cloud refinement processing according to claim 3, characterized in that, In step S3, for the fitted straight line... i Based on the ID information of each scan line given by the LiDAR, the background point cloud of the LiDAR on the background wall that is on the same scan line is found, which is used to extract the background edge points.

5. The joint calibration method for lidar and camera based on background point cloud refinement processing according to claim 4, characterized in that, In step S4, in the straight line i Find the points with horizontal angles α and β respectively as the background edge points p. b and foreground edge point p f The projection point; Specifically, p b The horizontal angle α passes through p b 3D coordinates (x) b ,y b ,z b The result is obtained by solving formula (6), and then t can be solved by substituting t into formula (5) to obtain p. b The projection point p b' Similarly, p can be solved. f The projection point p f' ; will p b' and p f' The geometric midpoint p m Save p as the edge point of the current frame. m To the set of edge points S edge middle; Continue to obtain the line in the next frame. i+1 After the same processing, the edge points of the new frame are also saved to the edge point set S. edge In the process, until a specified number of frames is reached; solve for the set of edge points S. edge The centroid of the scan line is taken as the true edge point and saved to edgepoints; similarly, the edge points of the other scan lines are solved and saved to edgepoints.

6. The joint calibration method for lidar and camera based on background point cloud refinement processing according to claim 5, characterized in that, In step S5, the calibration object is a square plate. The edge points of the four sides of the calibration object are obtained through the above steps and all are saved in edgepoints. The obtained edge points are fitted to the four side lines l1-l4 of the calibration object using the least squares method. The midpoint of the perpendicular bisector of the two spatial lines is used to represent the three-dimensional feature points p1-p4.

7. The joint calibration method for lidar and camera based on background point cloud refinement processing according to claim 6, characterized in that, In step S6, the position of the calibration object is changed, and more three-dimensional feature points are obtained in the same way. The obtained three-dimensional feature points and the corresponding two-dimensional feature points obtained by the visible light camera are substituted into the optimization equation shown in formula (4) to solve the calibration matrix.

8. The joint calibration method for lidar and camera based on background point cloud refinement processing according to claim 7, characterized in that, In step S6, two-dimensional feature points are extracted from the image acquired by the visible light camera using edge detection and / or corner detection methods.

Images