A fusion method based on fisheye camera and lidar applied to unmanned road cleaning vehicles

Abstract

This invention discloses a fisheye camera and lidar fusion method suitable for unmanned road cleaning vehicles. The method includes joint calibration of the lidar and fisheye camera, projection of the lidar point cloud onto the fisheye camera image plane, selection of point clouds within the "person" and "fire hydrant" detection frames, pixel-based tracking based on the specific clothing of the cleaning personnel, and deriving their three-dimensional coordinates. This method overcomes the limitations of a single sensor in terms of range and accuracy, improving the overall accuracy of the perception system.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned vehicles, and in particular to a fusion method based on a fisheye camera and a laser radar for an unmanned road cleaning vehicle. Background Art

[0002] With the accelerating pace of urbanization, road cleaning and maintenance have become crucial public services in urban management. Traditional cleaning methods rely primarily on manual labor, which is labor-intensive and inefficient. In recent years, with the rapid development of unmanned driving technology, the use of dedicated unmanned vehicles in urban management has increased, especially in road cleaning. These vehicles offer a high degree of automation and intelligence, significantly improving cleaning efficiency and operational quality. However, in complex urban environments, further improving the unmanned vehicles' ability to perceive operators, pedestrians, and the surrounding environment remains a pressing challenge. This is also the key issue that the present invention aims to address. Summary of the Invention

[0003] The technical problem that the present invention aims to solve is to provide a fusion method based on fisheye camera and lidar suitable for unmanned road cleaning vehicles, which overcomes the limitations of a single sensor in terms of perception range and accuracy and improves the overall accuracy of the perception system.

[0004] To address the aforementioned technical issues, this paper proposes a fusion method that combines wide-angle images captured by a fisheye camera with precise distance information provided by a lidar. This method uses the YOLOv8 algorithm to identify objects in the image and presents the recognition results in a 3D lidar point cloud in real time. Simultaneously, the 3D physical coordinates of the objects are published for use by downstream decision-making and planning algorithms. The specific steps are as follows:

[0005] Step S1: Joint calibration of lidar and fisheye camera;

[0006] Step S11: Calibrate the checkerboard. During the calibration process, the image data captured by the fisheye camera and the point cloud data collected by the lidar are synchronously recorded; these data are processed to separate the image and point cloud data within the same frame; and the extrinsic parameter matrix between the lidar and the fisheye camera is obtained, including rotation and translation.

[0007] Placing the checkerboard at different orientations and poses within the fisheye camera's field of view to fully cover its wide-angle imaging area;

[0008] Using the extrinsic calibration tool, the extrinsic parameter matrix between the fisheye camera and the lidar is calculated to describe the spatial relationship between the two, that is, their relative position and posture in three-dimensional space.

[0009] Step S12: Manually calibrate the fisheye camera's internal parameters to ensure the accuracy of image projection and lay the foundation for subsequent data fusion. Manually calibrate all checkerboard corner points of the image covering the fisheye lens imaging area, including edge areas that are difficult for the automatic detection algorithm to handle;

[0010] Step S2: The lidar point cloud is projected onto the fisheye camera image plane, and the point clouds belonging to the “person” and “fire hydrant” detection boxes are filtered out;

[0011] Step S21: Preprocess the LiDAR point cloud, including removing the vehicle's own point cloud and performing through-filtering and radius filtering to obtain a preprocessed point cloud set. Extract the point cloud within the detection frame identified by the fisheye camera;

[0012] Step S22: Cluster the point cloud. Extract the point cloud close to the camera and use the RANSAC method to perform cylindrical clustering on the candidate point cloud. If the clustering is successful, calculate the median of the cluster points as the 3D coordinates of the target.

[0013] If clustering fails, flat clustering is used as a backup solution;

[0014] Step S3: performing pixel-based tracking of the road cleaning vehicle operator;

[0015] Step S31: Before the system is started, the pixel values ​​corresponding to the target operator's uniform are predetermined;

[0016] Step S32: After the system is started, pixel values ​​are extracted from the detected "person" area and compared with the preset pixel values. If a match is found, the target is identified as an operator and its three-dimensional coordinates are detected.

[0017] The superior effects of the present invention are:

[0018] (1) Using fisheye cameras as the main visual sensor: Fisheye cameras can capture wide-angle images, significantly expanding the field of view while effectively reducing camera costs. This provides a more cost-effective solution for the widespread use of professional unmanned road cleaning vehicles;

[0019] (2) Accurate intrinsic calibration of the fisheye camera: To address the limitations of the automatic checkerboard recognition method in the edge areas of the fisheye camera, the present invention manually calibrates all checkerboard corner points of 169 images with high accuracy to obtain accurate intrinsic parameter information. This ensures full utilization of the wide field of view of the fisheye camera and greatly improves the overall accuracy of the perception system;

[0020] (3) Innovative application of sensor fusion in special vehicles: The fusion method of fisheye camera and lidar proposed in this invention overcomes the limitations of a single sensor in terms of perception range and accuracy, and provides a new technical path for the application of unmanned road cleaning special vehicles in complex urban environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are intended to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings:

[0022] Figure 1 It is a workflow diagram of a specific embodiment of the present invention.

[0023] Figure 2 It is a simplified mobile cleaning vehicle platform used in the present invention.

[0024] Figure 3 It is the final result diagram of the present invention. DETAILED DESCRIPTION

[0025] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0026] The present invention provides a fusion method based on fisheye camera and laser radar applied to special unmanned road cleaning vehicles, such as Figure 2 As shown, a simplified mobile cleaning vehicle platform is used as the experimental object.

[0027] like Figure 1 The specific steps are as follows:

[0028] Step S1: Joint calibration of the lidar and fisheye camera. The specific steps are as follows:

[0029] Step S11: A black and white checkerboard with a size of 10×10 is used for calibration, and the side length of each grid is 5 cm. In order to ensure the accuracy of the calibration results, the checkerboard is placed in different directions and postures of the fisheye camera's field of view to fully cover its wide-angle imaging area. During the calibration process, the image data captured by the fisheye camera and the point cloud data collected by the lidar are recorded synchronously. Subsequently, MATLAB software is used to process these data to separate the image and point cloud data of each frame. With the help of MATLAB's external parameter calibration tool, the external parameter matrix between the fisheye camera and the lidar is accurately calculated. This matrix describes the relative position and posture relationship between the two in three-dimensional space, including the rotation matrix R and the translation vector T. The specific formula is:

[0030]

[0031] Then for a point P(xl ,y l ,z l ), its coordinates in the fisheye camera coordinate system are (x c ,y c ,z c ), the conversion relationship between the two is expressed as:

[0032]

[0033] Step S12: Since the Scaramuzza model of MATLAB in the existing fisheye camera model is relatively complex and difficult to implement, the present invention adopts the Kannala-Brandt model widely used in the OpenCV library, which has good computational efficiency and accuracy in the field of fisheye camera calibration and is suitable for internal parameter calibration in practical applications. However, the automatic checkerboard corner detection algorithm has insufficient detection accuracy in the edge area of ​​the fisheye camera and may not even be able to detect corners at all. In order to ensure the high accuracy of the calibration results, the present invention calibrates the fisheye camera manually, and manually calibrates all checkerboard corner points of 169 images to ensure that the imaging area of ​​the entire fisheye lens is covered, especially the edge areas that are difficult for the automatic detection algorithm to handle. After manual calibration, the internal parameter accuracy of the fisheye camera is ensured within the entire field of view, so that the internal parameter results are more reliable and comprehensive. Then for the point P obtained in the previous part at the fisheye camera coordinate (x c ,y c ,z c ), the goal of intrinsic calibration is to determine its coordinates (u, v) in the fisheye camera pixel coordinate system. The relevant formula is as follows:

[0034]

[0035] r 2 =a 2 +b 2

[0036] θ=arctan(r)

[0037] θ d =θ(1+k1θ 2 +k2θ 4 +k3θ 6 +k4θ 8 )

[0038]

[0039] The internal parameter mapping method of the above fisheye camera is represented by K;

[0040] Step S2: Figure 1As shown in the figure, the lidar point cloud is projected onto the fisheye camera image plane, and the point clouds belonging to the "person" and "fire hydrant" detection boxes are filtered out. The specific steps are as follows:

[0041] Step S21: Preprocess the original point cloud P(t) at time t. First, remove the vehicle point cloud to obtain P1(t). Then perform straight-through filtering and radius filtering to obtain P2(t) and P3(t) in sequence. According to the formula Project it into the fisheye lens and then determine point p i (t)=[u i ,v i ] T Whether it falls within the detection frame of "person" or "fire hydrant". If the point is within the detection frame of "person" or "fire hydrant", the corresponding point cloud Add the point cloud P to be clustered final (t) medium;

[0042] Step S22: Treat the clustered point cloud P final (t) Extract the point cloud close to the camera

[0043] Then, P near (t) Cylindrical clustering based on RANSAC method:

[0044] C cy1 (t) = RANSAC(P near (t),cylindermodel)

[0045] Then all the point clouds C after clustering are obtained cy1 (t) The median P of the coordinate center (t)=median(C cy1 (t)), which are the 3D coordinates of the “person” or “fire hydrant” in the fisheye lens. If cylindrical clustering fails, planar clustering is initiated as a backup.

[0046] Step S3: pixel-based tracking based on the specific clothing of the cleaning staff;

[0047] Step S31: Pre-acquire target cleaning personnel R Target_person Pixel V Target_person =pixel(R Target_person );

[0048] Step S32: In the new frame t, extract the detection box R belonging to the “person” person (t) clothing pixel value V(t) = pixel(R person (t)), and compare this value with VTarfet_person For comparison, if the person is within a certain range, the “person” in the detection frame is considered to be the target cleaning staff, and the above-mentioned fusion method is used to detect their three-dimensional coordinates, thereby effectively improving the vehicle’s perception of the operator.

[0049] The final result is as follows Figure 3 As shown, when two "people" and a "fire hydrant" are detected in the fisheye camera, this method presents the corresponding positions of this information in the three-dimensional point cloud map in real time and publishes the three-dimensional coordinates of this information to the downstream.

[0050] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that the present invention is susceptible to various modifications and variations. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A method for fusing fisheye cameras and laser radar for unmanned road cleaning vehicles, comprising the following steps: Step S1: Joint calibration of lidar and fisheye camera; Step S11: Calibrate the checkerboard. During the calibration process, synchronously record the image data captured by the fisheye camera and the point cloud data collected by the lidar; process these data to separate the image and point cloud data within the same frame; and obtain the extrinsic parameter matrix between the lidar and the fisheye camera. Step S12: manually calibrating the fisheye camera's internal parameters, and manually calibrating all checkerboard corner points of the image covering the fisheye lens imaging area; Step S2: The lidar point cloud is projected onto the fisheye camera image plane, and the point clouds belonging to the "person" and "fire hydrant" detection boxes are filtered out; Step S21: Preprocessing the lidar point cloud, including removing the vehicle's own point cloud and performing through-filtering and radius filtering to obtain a preprocessed point cloud set, from which the point cloud within the detection frame recognized by the fisheye camera is extracted; Step S22: clustering the point cloud, extracting the point cloud close to the camera, and using the RANSAC method to perform cylindrical clustering on the candidate point cloud; if clustering is successful, the median of the cluster points is calculated as the three-dimensional coordinates of the target; Step S3: performing pixel-based tracking based on the specific clothing of the cleaning staff and obtaining its three-dimensional coordinates; Step S31: Before the system is started, the pixel values ​​corresponding to the target operator's uniform are predetermined; Step S32: After the system is started, pixel values ​​are extracted from the detected "person" area and compared with the preset pixel values. If a match is found, the target is identified as an operator and its three-dimensional coordinates are detected.

2. The method for fusing fisheye cameras and laser radar for unmanned road cleaning vehicles according to claim 1, characterized in that: The external parameter matrix of step S11 includes the rotation matrix R and the translation vector T: For a point P(x l ,y l , z l ), its coordinates in the fisheye camera coordinate system are (x c ,y c , z c ), the conversion relationship between the two is expressed as:

3. A computer system comprising a memory, a processor, and a computer program stored in the memory, wherein: The processor executes the computer program to implement the steps of the fusion method according to any one of claims 1 to 2.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the fusion method according to any one of claims 1 to 2 are implemented.

5. A computer program product comprising a computer program, characterized in that: When the computer program is executed by a processor, the steps of the fusion method according to any one of claims 1 to 2 are implemented.

Images