Fusion method based on fisheye camera and laser radar and applied to unmanned road cleaning vehicle

Abstract

The invention discloses a fusion method based on a fisheye camera and a laser radar and suitable for an unmanned road cleaning vehicle, and the method comprises the steps: carrying out the joint calibration of the laser radar and the fisheye camera, projecting the point cloud of the laser radar to an image plane of the fisheye camera, screening out the point cloud belonging to the detection frames of a person and a fire hydrant, and carrying out the recognition of the point cloud. Pixel-based tracking is carried out according to specific clothes of cleaning personnel, and three-dimensional coordinates of the clothes are obtained. The method has the advantages that the limitation of a single sensor in the aspects of sensing range and precision is overcome, and the overall accuracy of a sensing system is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned vehicles, and in particular to a fusion method based on a fisheye camera and a laser radar for an unmanned road cleaning vehicle. Background Art

[0002] With the continuous acceleration of urbanization, road cleaning and maintenance have become vital public services in urban management. Traditional cleaning methods mainly rely on manual operations, which are labor-intensive and inefficient. In recent years, with the rapid development of unmanned driving technology, the application of special unmanned vehicles in the field of urban management has increased, especially in road cleaning. Such vehicles have the advantages of high automation and intelligence, which significantly improves cleaning efficiency and work quality. However, in complex urban environments, how to further improve the perception of unmanned vehicles to operators, pedestrians and the surrounding environment is still a difficult problem to be solved. This is also the key problem that the present invention aims to solve. Summary of the invention

[0003] The technical problem that the present invention aims to solve is to provide a fusion method based on fisheye camera and laser radar suitable for unmanned road cleaning vehicles, which overcomes the limitations of a single sensor in terms of perception range and accuracy and improves the overall accuracy of the perception system.

[0004] To solve the above technical problems, the present invention proposes a fusion method, which combines the wide-angle image captured by the fisheye camera with the precise distance information provided by the lidar, uses the YOLOv8 algorithm to identify the target in the image, and presents the recognition result in real time in the three-dimensional lidar point cloud. At the same time, the three-dimensional physical coordinates of the target are published for use by the downstream decision-making and planning algorithm. The specific steps are as follows:

[0005] Step S1: Joint calibration of LiDAR and fisheye camera;

[0006] Step S11: Calibrate the chessboard. During the calibration process, synchronously record the image data captured by the fisheye camera and the point cloud data collected by the laser radar; process these data to separate the image and point cloud data in the same frame; obtain the external parameter matrix between the laser radar and the fisheye camera, including rotation and translation;

[0007] Placing the checkerboard at different orientations and attitudes within the fisheye camera’s field of view to fully cover its wide-angle imaging area;

[0008] Through the external parameter calibration tool, the external parameter matrix between the fisheye camera and the lidar is calculated to describe the spatial relationship between the two, that is, the relative position and posture relationship between the two in three-dimensional space;

[0009] Step S12: Manually calibrate the fisheye camera internal parameters to ensure the accuracy of image projection and lay the foundation for subsequent data fusion. Manually calibrate the checkerboard corner points of all images covering the fisheye lens imaging area, including edge areas that are difficult for the automatic detection algorithm to handle;

[0010] Step S2: The laser radar point cloud is projected onto the fisheye camera image plane, and the point clouds belonging to the “person” and “fire hydrant” detection boxes are screened out;

[0011] Step S21: preprocessing the laser radar point cloud, including removing the point cloud of the vehicle itself and performing through filtering and radius filtering to obtain a preprocessed point cloud set, from which the point cloud within the detection frame recognized by the fisheye camera is extracted;

[0012] Step S22: cluster the point cloud. Extract the point cloud close to the camera and use the RANSAC method to perform cylindrical clustering on the candidate point cloud. If the clustering is successful, calculate the median of the cluster points as the three-dimensional coordinates of the target;

[0013] If clustering fails, flat clustering is used as a backup solution;

[0014] Step S3: performing pixel-based tracking of the road cleaning vehicle operator;

[0015] Step S31: before the system is started, the pixel values ​​corresponding to the uniforms of the target operator are predetermined;

[0016] Step S32: After the system is started, pixel values ​​are extracted from the detected "person" area and compared with the preset pixel values. If they match, the target is identified as an operator and its three-dimensional coordinates are detected.

[0017] The superior effects of the present invention are:

[0018] (1) Using fisheye cameras as the main visual sensor: Fisheye cameras can capture wide-angle images, significantly expanding the field of view while effectively reducing camera costs. This provides a more cost-effective solution for the widespread use of professional unmanned road cleaning vehicles;

[0019] (2) Accurate internal calibration of fisheye cameras: In view of the limitations of the automatic checkerboard recognition method in the edge area of ​​the fisheye camera, the present invention manually calibrates all the checkerboard corner points of the 169 images with high accuracy to obtain accurate internal parameter information. This ensures full utilization of the wide field of view of the fisheye camera and greatly improves the overall accuracy of the perception system;

[0020] (3) Innovative application of sensor fusion in special vehicles: The fusion method of fisheye camera and lidar proposed in this invention overcomes the limitations of a single sensor in terms of sensing range and accuracy, and provides a new technical path for the application of unmanned road cleaning vehicles in complex urban environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings constituting a part of the present application are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0022] Figure 1 It is a workflow diagram of a specific embodiment of the present invention.

[0023] Figure 2 It is a simplified mobile cleaning vehicle platform used in the present invention.

[0024] Figure 3 It is the final result diagram of the present invention. DETAILED DESCRIPTION

[0025] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0026] The present invention provides a fusion method based on fisheye camera and laser radar applied to a special unmanned road cleaning vehicle, such as Figure 2 As shown, a simplified mobile cleaning vehicle platform is used as the experimental object.

[0027] like Figure 1 As shown, the specific steps are as follows:

[0028] Step S1: Joint calibration of LiDAR and fisheye camera. The specific steps are as follows:

[0029] Step S11: A black and white checkerboard with a size of 10×10 is used for calibration, and the side length of each grid is 5 cm. In order to ensure the accuracy of the calibration results, the checkerboard is placed in different directions and postures of the fisheye camera's field of view to fully cover its wide-angle imaging area. During the calibration process, the image data captured by the fisheye camera and the point cloud data collected by the lidar are recorded synchronously. Subsequently, the data are processed using MATLAB software to separate the image and point cloud data of each frame. With the help of MATLAB's external parameter calibration tool, the external parameter matrix between the fisheye camera and the lidar is accurately calculated. This matrix describes the relative position and posture relationship between the two in three-dimensional space, including the rotation matrix R and the translation vector T. The specific formula is:

[0030]

[0031] Then for a point P(xl ,y l ,z l ), its coordinates in the fisheye camera coordinate system are (x c ,y c ,z c ), the conversion relationship between the two is expressed as:

[0032]

[0033] Step S12: Since the Scaramuzza model of MATLAB in the existing fisheye camera model is relatively complex and difficult to implement, the present invention adopts the Kannala-Brandt model widely used in the OpenCV library, which has good computational efficiency and accuracy in the field of fisheye camera calibration and is suitable for intrinsic parameter calibration in practical applications. However, the automatic checkerboard corner point detection algorithm has insufficient detection accuracy in the edge area of ​​the fisheye camera and may not even be able to detect the corner points at all. In order to ensure the high accuracy of the calibration results, the present invention calibrates the fisheye camera manually, and manually calibrates all the checkerboard corner points of 169 images to ensure that the entire imaging area of ​​the fisheye lens is covered, especially the edge areas that are difficult to handle with the automatic detection algorithm. After manual calibration, the intrinsic parameter accuracy of the fisheye camera within the entire field of view is ensured, so that the intrinsic parameter results are more reliable and comprehensive. Then for the point P obtained in the previous part at the fisheye camera coordinate (x c ,y c ,z c ), the goal of the intrinsic calibration is to determine its coordinates (u, v) in the fisheye camera pixel coordinate system. The relevant formula is as follows:

[0034]

[0035] r 2 =a 2 +b 2

[0036] θ=arctan(r)

[0037] θ d =θ(1+k1θ 2 +k2θ 4 +k3θ 6 +k4θ 8 )

[0038]

[0039] The internal parameter mapping method of the above fisheye camera is represented by K;

[0040] Step S2: Figure 1As shown in the figure, the LiDAR point cloud is projected onto the fisheye camera image plane, and the point clouds belonging to the “person” and “fire hydrant” detection boxes are filtered out. The specific steps are as follows:

[0041] Step S21: Preprocess the original point cloud P(t) at time t. First, remove the vehicle point cloud to obtain P1(t). Then, perform straight-through filtering and radius filtering to obtain P2(t) and P3(t). According to the formula Project it into the fisheye lens and then determine point p i (t)=[u i ,v i ] T Whether it falls within the detection box of "person" or "fire hydrant". If the point is within the detection box of "person" or "fire hydrant", the corresponding point cloud Add the point cloud P to be clustered final (t) medium;

[0042] Step S22: Treat the clustered point cloud P final (t) Extract the point cloud close to the camera Then, P near (t) Cylindrical clustering based on the RANSAC method:

[0044] C cy1 (t) = RANSAC(P near (t), cylinder model)

[0045] Then all the point clouds C after clustering are obtained cy1 (t) Median P of the coordinate center (t)=median(C cy1 (t)), which is the 3D coordinate of the “person” or “fire hydrant” in the fisheye lens. If cylindrical clustering fails, planar clustering is started as a backup solution.

[0046] Step S3: performing pixel-based tracking according to the specific clothing of the cleaning staff;

[0047] Step S31: Pre-acquire target cleaning personnel R Target_person Pixel V Target_person =pixel(R Target_person );

[0048] Step S32: In the new frame t, extract the detection box R belonging to the “person” person (t) clothing pixel value V(t) = pixel(R person (t)), and compare this value with V Tarfet_personFor comparison, if it is within a certain range, the “person” in this detection frame is considered to be the target cleaning staff, and its three-dimensional coordinates are detected using the above-mentioned fusion method, thereby effectively improving the vehicle’s perception of the operator.

[0049] The final result is as follows Figure 3 As shown, when two "people" and one "fire hydrant" are detected in the fisheye camera, this method presents the corresponding positions of this information in the three-dimensional point cloud map in real time and publishes the three-dimensional coordinates of this information to the downstream.

[0050] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A fusion method based on a fisheye camera and a laser radar for an unmanned road cleaning vehicle, comprising the following steps: Step S1: Joint calibration of LiDAR and fisheye camera; Step S11: Calibrate the chessboard. During the calibration process, synchronously record the image data captured by the fisheye camera and the point cloud data collected by the laser radar; process these data to separate the image and point cloud data in the same frame; and obtain the external parameter matrix between the laser radar and the fisheye camera; Step S12: manually calibrating the fisheye camera's internal parameters, and manually calibrating all checkerboard corner points of the image covering the fisheye lens imaging area; Step S2: The laser radar point cloud is projected onto the fisheye camera image plane, and the point clouds belonging to the "person" and "fire hydrant" detection boxes are screened out; Step S21: preprocessing the laser radar point cloud, including removing the point cloud of the vehicle itself and performing through filtering and radius filtering to obtain a preprocessed point cloud set, from which the point cloud within the detection frame recognized by the fisheye camera is extracted; Step S22: clustering the point cloud, extracting the point cloud close to the camera, and using the RANSAC method to perform cylindrical clustering on the candidate point cloud; if the clustering is successful, the median of the cluster point is calculated as the three-dimensional coordinate of the target; Step S3: performing pixel-based tracking according to the specific clothing of the cleaning staff and obtaining its three-dimensional coordinates; Step S31: before the system is started, the pixel values ​​corresponding to the uniforms of the target operator are predetermined; Step S32: After the system is started, pixel values ​​are extracted from the detected "person" area and compared with the preset pixel values. If a match is found, the target is identified as an operator and its three-dimensional coordinates are detected.

2. The fusion method based on fisheye camera and laser radar applicable to unmanned road cleaning vehicles according to claim 1 is characterized in that: The external parameter matrix of step S11 includes the rotation matrix R and the translation vector T: For a point P(x l ,y l , z l ), its coordinates in the fisheye camera coordinate system are (x c ,y c , z c ), the conversion relationship between the two is expressed as:

3. A computer system comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the fusion method according to any one of claims 1 to 2.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the fusion method described in any one of claims 1 to 2 are implemented.

5. A computer program product, comprising a computer program, characterized in that: When the computer program is executed by a processor, the steps of the fusion method described in any one of claims 1 to 2 are implemented.

Images