A monocular camera surround view compensation device and method for a long and narrow environment laser radar

Abstract

The application discloses a monocular camera look-around compensation device and method for a long and narrow environment laser radar, wherein the compensation device comprises an airborne laser radar and a point cloud compensation device, the monocular camera is fixed on a holder, the holder is used for dragging the monocular camera to rotate, and the look-around effect is achieved; the problem that the coincidence degree of the field of view of the panoramic camera and the scanning field of the laser radar is not high is avoided, and the problem that the field of view angle of the fixed monocular camera is not enough is also solved. The compensation method is divided into a monocular camera look-around point cloud compensation method and a long and narrow environment compensation method; under the condition that camera-laser radar calibration is performed once, laser radar compensation of a single shaft full angle is realized, and the scene scanning density of a low line number radar can be compensated at a relatively low cost.

Description

Technical Field

[0001] The present invention belongs to the field of point cloud densification, and in particular relates to a monocular camera surround view compensation device and method for a laser radar in a narrow and long environment. Background Art

[0002] With the continuous advancement of autonomous driving and navigation technologies, lidar (LiDAR) is widely used in autonomous driving and unmanned navigation systems as a highly efficient three-dimensional surface scanning device. Currently, mechanical LiDARs used for perception typically have a specified number of scan lines, which typically do not exceed 64. Furthermore, high-line-count LiDARs are bulky and expensive, posing challenges in cost and volume control for small unmanned vehicle systems. Therefore, in some scenarios where low-line-count LiDARs are used, point cloud compensation using camera pixel projection is required. Current point cloud compensation methods typically utilize panoramic (surround view) cameras or fixed monocular cameras. The former typically has a compensation angle of no more than 180 degrees, making it difficult for its field of view to overlap with the LiDAR's scan field over a large area and limiting its mounting location. Fixed monocular cameras, however, struggle to compensate for the LiDAR's wider viewing angle. In narrow and long environments, using panoramic cameras for point cloud compensation can introduce unnecessary field-of-view imaging content due to the limited point cloud scan field, while the use of multiple fixed monocular cameras imposes computational pressure on the system. Summary of the Invention

[0003] To address the above-mentioned issues, the present invention discloses a monocular camera surround view compensation device for a laser radar in a narrow and long environment. The device fixes the monocular camera on a gimbal and uses the gimbal to drag the monocular camera in rotation to achieve a surround view effect. The present invention can avoid the problem of low overlap between the panoramic camera's field of view and the laser radar's scanning field, and can also solve the problem of insufficient field of view of a fixed monocular camera. At the same time, in view of the obvious depth direction of narrow and long environments, the device can automatically achieve point cloud density compensation under a given field of view according to algorithm requirements as needed.

[0004] To achieve the above object, the technical solution of the present invention is as follows:

[0005] A monocular camera surround view compensation device for a narrow and long environment laser radar, comprising:

[0006] LiDAR, used to collect point cloud images within the LiDAR scanning field;

[0007] A point cloud compensation device is placed above or below the laser radar and is used for point cloud compensation;

[0008] A controller, provided on the wheeled chassis, is used for communication, control and data acquisition and processing;

[0009] A wheeled chassis with a power source, a support disc, a lower controller and a moving assembly for supporting the movement of an unmanned system;

[0010] The controller is electrically connected with the laser radar, the point cloud compensation device and the wheeled chassis.

[0011] Further, the compensation device comprises a monocular camera, a transparent shell for perspective, a holder for dragging the monocular camera to rotate, an upper and lower bottom plate for mounting and supporting, a through strut, and a sheet metal for mounting the monocular camera, the through strut is fixedly connected with the holder and the upper bottom plate, the upper and lower bottom plates are fixedly connected with the transparent shell, the monocular camera is mounted on the sheet metal, the sheet metal is fixedly connected with the holder and penetrated by the through strut, the holder drags the monocular camera to rotate, and the rotation axis of the holder is coaxial with the axis of the laser radar.

[0012] The compensation method using the above-mentioned compensation device comprises a monocular camera ring-viewing point cloud compensation method and a long and narrow environment compensation method.

[0013] The monocular camera ring-viewing point cloud compensation method (hereinafter referred to as the point cloud compensation method) comprises the following steps:

[0014] Step one, starting the system, calibrating the laser radar and the monocular camera;

[0015] Step two, after completing step one, continue to collect laser radar point cloud data and monocular camera data, and use a program to control the holder to rotate to obtain the relative rotation angle of the holder;

[0016] Step three, after completing step two, map the points in the laser radar coordinate system to the camera coordinate system according to the relative rotation angle of the holder and the calibration result;

[0017] Step four: after projecting the laser points into the camera coordinate system, sample the pixel points near the laser points, and project the sampling structure back to the laser radar coordinate system to obtain the compensated point cloud.

[0018] The long and narrow environment compensation method comprises the following steps:

[0019] Step (1), mounting the compensation device on the carrier system, and deploying the carrier system in a long and narrow environment;

[0020] Step (2), starting the carrier system, calibrating according to step one of the monocular camera ring-viewing point cloud compensation method, then dragging the monocular camera to rotate to a specified angle by the holder, and compensating the long and narrow part of the laser radar scanning field.

[0021] Further, the starting system and calibrating the laser radar and the monocular camera in step one are as follows: Further, the compensation device comprises a monocular camera, a transparent shell for perspective, a holder for dragging the monocular camera to rotate, an upper and lower bottom plate for mounting and supporting, a through strut, and a sheet metal for mounting the monocular camera, the through strut is fixedly connected with the holder and the upper bottom plate, the upper and lower bottom plates are fixedly connected with the transparent shell, the monocular camera is mounted on the sheet metal, the sheet metal is fixedly connected with the holder and penetrated by the through strut, the holder drags the monocular camera to rotate, and the rotation axis of the holder is coaxial with the axis of the laser radar.

[0022] First, calibrate the monocular camera to obtain the camera's intrinsic parameter matrix K and extrinsic parameter rotation matrix R and translation vector

[0023] Secondly, calibrate the laser radar and the monocular camera to obtain the rotation matrix φ of the laser radar coordinate system relative to the camera coordinate system and the translation vector of the laser radar coordinate system relative to the camera coordinate system

[0024] Furthermore, the relative rotation angle θ described in step 2 is calculated as follows:

[0025] Continue to collect lidar point cloud data With monocular camera data And use the program to control the rotation of the pan-tilt head to obtain the relative rotation angle θ of the pan-tilt head.

[0026] Furthermore, the result of mapping the points in the lidar coordinate system to the camera coordinate system in step 3 is as follows:

[0027]

[0028] Obviously, the relative rotation angle of the gimbal does not produce any change in attitude relative to the z-axis. Therefore, the angle containing the parameter θ is the sum of the initial rotation angle along the z-axis and the relative rotation angle θ, and this part is already included in R and φ. Assuming the original displacement You can get the original xOy plane camera angle The angle after the gimbal rotates is γ=θ0+θ, and the final displacement is

[0029]

[0030] in

[0031] Furthermore, step four is as follows:

[0032] After projecting the laser point to the camera coordinate system, the pixel points near the laser point are sampled to obtain Projecting it back to the LiDAR coordinate system, the final projection result is as follows:

[0033]

[0034] Furthermore, step (2) is specifically as follows:

[0035] The range of the point cloud scanning field is judged, and the part exceeding the threshold is defined as the depth range of the scanning field. The monocular camera is dragged by the gimbal to rotate to the angle of the depth range, and the compensation result is obtained according to steps 2, 3, and 4 of the point cloud compensation method.

[0036] The beneficial effects of the present invention are:

[0037] (1) When the laser radar and the pan-tilt head rotation axis are coaxial, the present invention does not limit the upper / lower installation position of the compensation device;

[0038] (2) Based on the point cloud compensation method of the present invention, it is possible to achieve single-axis full-angle lidar compensation under the condition of performing a single camera-lidar calibration;

[0039] (3) The present invention can compensate for the scene scanning density of low-line-count radar at a relatively low cost;

[0040] (4) The present invention has a clear depth characteristic for narrow and long environments and can rotate spontaneously in narrow and long environments to achieve dense compensation of low-line-count mechanical laser radar point clouds in the depth direction or target direction field of view. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a schematic diagram of an embodiment of the present invention applied to an unmanned wheeled vehicle system;

[0042] Figure 2 This is a structural diagram of a laser radar and compensation device according to an embodiment of the present invention;

[0043] Figure 3 This is a schematic diagram of the modeling of the laser radar and compensation device of the present invention;

[0044] Figure 4 This is a schematic diagram of the synthesis result of the laser radar scanning field and the camera field of view of the present invention;

[0045] Figure 5 Schematic diagram of motion modeling of the compensation device of the present invention after projection onto the xOy plane;

[0046] Figure 6 This is a flow chart of the monocular camera surround view point cloud compensation method of the present invention;

[0047] Figure 7 It is a schematic diagram of the implementation of the present invention in a long and narrow environment.

[0048] List of Figure Symbols:

[0049] 1. LiDAR; 2. Point cloud compensation device; 3. Controller; 4. Wheeled chassis; 21. Monocular camera; 22. Transparent housing;

[0050] 23. Pan / tilt; 24. Upper base; 25. Through-support; 26. Sheet metal; 27. Lower base. DETAILED DESCRIPTION

[0051] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0052] like Figure 1 As shown, the main components of the unmanned wheeled vehicle system (compensation device) described in this embodiment are a laser radar 1 for collecting point cloud images within the laser radar scanning field; a point cloud compensation device 2 for performing point cloud compensation; a controller 3 for communication, control, and data acquisition and processing; and a wheeled chassis 4, which has internal components such as a power source, support plate, lower controller, and movement device to support the movement of the unmanned system. The controller 3 is electrically connected to the laser radar 1, the point cloud compensation device 2, and the wheeled chassis 4.

[0053] The installation locations of the laser radar 1, point cloud compensation device 2, and controller 3 used for demonstration in this embodiment are not limited to the type of vehicle, and can be applied to unmanned vehicles / ships / boats / aircraft, wheeled / legged robots, etc.

[0054] See also Figure 2 As shown, the compensation device of this embodiment includes a monocular camera 21, a transparent housing 22 for viewing through the lens, a pan-tilt platform 23 for rotating the monocular camera 21, upper and lower base plates 24 and 27 for mounting and supporting the monocular camera 21, and a through-pillar 25, and a sheet metal 26 for mounting the monocular camera 21. The pan-tilt platform 23 rotates the monocular camera 21, and the through-pillar 25 is fixedly connected to the pan-tilt platform 23 and the upper base plate 24. The upper and lower base plates 24 and 27 are fixedly connected to the transparent housing 22. The monocular camera 21 is mounted on the sheet metal 26, which is fixedly connected to the pan-tilt platform 23 and penetrated by the through-pillar 25. It should be noted that the rotation axis of the pan-tilt platform 23 is coaxial with the axis of the laser radar 1. The point cloud compensation device of the present invention can achieve the effect of monocular surround vision under electrical control.

[0055] See also Figure 3 and combined Figure 1 、 Figure 2 As shown, to facilitate formula derivation, the laser radar coordinate system, camera coordinate system, and world coordinate system are now defined. The laser radar coordinate system is defined as follows: the center of the laser radar 1 scanning field is the coordinate origin O, the vehicle's forward direction is the positive x-axis, and the top of the vehicle's shape is the positive z-axis. According to the right-hand rule, the y-axis direction is naturally determined. To simplify subsequent explanations, the world coordinate system is defined as the same as the laser radar coordinate system; the compensation device coordinate system (camera coordinate system) is defined as follows: the imaging center of the monocular camera 21 is set as the origin, the z-axis direction is the same as that of the laser radar coordinate system, and the camera coordinate system xOy plane is parallel to the laser radar xOy plane.

[0056] See also Figure 5 、 Figure 6 Combine Figure 2 As shown, the process of the monocular camera surround view point cloud compensation method of the present invention is as follows:

[0057] Step 1: Start the system and collect lidar point cloud data With monocular camera data Keep the position of the gimbal 23 unchanged and calibrate the laser radar 1 and the monocular camera 21. First, calibrate the monocular camera to obtain the camera intrinsic parameter matrix K and the extrinsic parameter rotation matrix R and translation vector Secondly, the laser radar and the monocular camera are jointly calibrated to obtain the rotation matrix φ of the laser radar coordinate system relative to the camera coordinate system and the translation vector of the laser radar coordinate system relative to the camera coordinate system. According to the laser radar coordinate system, the camera coordinate system and the world coordinate system, no additional calibration is required. The camera intrinsic parameter matrix K is used to correct the radial distortion of the monocular camera 21 imaging. Assume that the external parameter rotation matrix R and translation vector after the camera calibration are Used for camera coordinate system points To world coordinate point The mapping relationship is:

[0058]

[0059] The rotation matrix φ of the laser radar coordinate system relative to the camera coordinate system and the translation vector of the laser radar coordinate system relative to the camera coordinate system Combined with the camera calibration results, the point cloud points can be Projected into the camera coordinate system. Since we assume that the center of the world coordinate system is the same as the center of the lidar coordinate system, the projection relationship is:

[0060]

[0061] The monocular camera calibration and the joint calibration of the laser radar and the monocular camera are existing technologies and will not be described in detail here.

[0062] Step 2: After completing step 1, continue collecting lidar point cloud data With monocular camera data The program is used to control the rotation of the platform 23 to obtain the relative rotation angle θ of the platform.

[0063] Step 3: After completing step 2, see Figure 5 After projecting onto the xOy plane, the motion model is constructed. Based on the gimbal relative rotation angle θ and the calibration results, the result of mapping the points in the lidar coordinate system to the camera coordinate system is as follows:

[0064]

[0065] Obviously, under the model of Figure 3 The relative rotation angle of the holder does not change the attitude relative to the z-axis. Therefore, the part of the rotation angle containing the parameter θ is the sum of the initial rotation angle along the z-axis and the relative rotation angle θ, and this part has been included in R and φ. For Assuming the original displacement The original xOy plane camera rotation angle can be obtained And the rotation angle γ of the holder after rotation is θ0+θ, and finally the displacement is

[0066]

[0067] Where

[0068] Step four: after projecting the laser point into the camera coordinate system, sample the pixel points near the laser point to obtain

[0069] Project it back to the laser radar coordinate system to obtain the final projection result as follows:

[0070]

[0071] This embodiment only shows the compensation method of the present application in a single-axis holder. Some contents are abstracted and simplified for easy understanding and description. Those skilled in the art can easily extend it to the case of multi-axis holder movement through kinematics related knowledge.

[0072] The long scene implementation method of the present application mainly includes the following contents:

[0073] Step (1), install the device of the present application on the carrier system, and deploy the carrier system in the drivable area in the long environment.

[0074] Step (2), start the carrier system, calibrate according to step one of the monocular camera ring view point cloud compensation method; then rotate the monocular camera to the specified angle by dragging the holder, and compensate for the long part of the laser radar scanning field.

[0075] Specifically, determine the longitudinal range and obstacle target according to the range and algorithm of the point cloud scanning field, rotate the monocular camera to the angle where the longitudinal range is located by dragging the holder, and perform operations according to steps two, three and four of the point cloud compensation method to obtain the compensation result.

[0076] In order to better describe the contents of the long scene implementation method, see Figure 7 Combined with Figure 5 , Figure 6As shown, the embodiment of the narrow and long space of the present invention selects three positions of the modeling scene for detailed description:

[0077] At position P1, the vehicle heading is toward the left of the drawing. From the point cloud data, it can be known that the depth is toward the left. The gimbal drags the monocular camera to rotate until the field of view of the monocular camera is F11, and real-time point cloud compensation is performed.

[0078] At position P2, the vehicle is oriented upwards towards the drawing. An obstacle target is collected in the point cloud data through the algorithm. The gimbal drags the monocular camera to rotate until the field of view of the monocular camera is F21, and real-time point cloud compensation is performed. The gimbal stops after moving away from the obstacle target to a certain position.

[0079] At position P3, the vehicle is heading to the left of the drawing. The point cloud data indicates that the vehicle is facing to the left in depth. The algorithm collects an obstacle in the point cloud data. The gimbal drags the monocular camera to rotate, and performs real-time point cloud compensation when the monocular camera's fields of view are F31 and F32, respectively. After moving away from the obstacle to a certain position, the vehicle stops rotating and moves to the F32 position for compensation.

[0080] Some details in this embodiment, such as modeling, timestamp alignment, information collection and communication, are methods commonly used by those skilled in the art. Those skilled in the art can easily complete and implement these technical details through relevant knowledge.

Claims

1. A monocular camera surround view compensation device for a narrow and long environment laser radar, characterized by: A point cloud density compensation device for a narrow and long environment includes a monocular camera, a transparent housing for perspective viewing, a pan-tilt platform for rotating the monocular camera, upper and lower base plates for installation and support, a through-pillar, and a sheet metal for mounting the monocular camera, wherein the through-pillar is fixedly connected to the pan-tilt platform and the upper base plate, the upper and lower base plates are fixedly connected to the transparent housing, the monocular camera is mounted on the sheet metal, the sheet metal is fixedly connected to the pan-tilt platform and penetrated by the through-pillar, the pan-tilt platform rotates the monocular camera, and the pan-tilt platform rotation axis is coaxial with the axis of a laser radar. The method for achieving dense point cloud compensation in narrow and long environments is divided into two parts: the dense point cloud compensation method of the monocular camera surround view and the narrow and long environment compensation method; The monocular camera surround view point cloud dense compensation method comprises the following steps: After projecting the laser point to the camera coordinate system, the pixel points near the laser point are sampled and projected back to the lidar coordinate system to obtain the compensated point cloud; The narrow and long environment compensation method comprises the following steps: Start the vehicle system and calibrate it according to step 1 of the monocular camera surround point cloud dense compensation method. Then, the gimbal drags the monocular camera to rotate to a specified angle to compensate for the narrow and long part of the lidar scanning field.

2. A compensation method using the monocular camera surround view compensation device for a narrow and long environment laser radar according to claim 1, characterized in that: It is divided into two parts: the dense compensation method of the monocular camera surround view point cloud and the narrow and long environment compensation method; The monocular camera surround view point cloud dense compensation method comprises the following steps: Step 1: Start the system and calibrate the lidar and monocular camera; Step 2: After completing step 1, continue to collect lidar point cloud data and monocular camera data, and use the program to control the rotation of the gimbal to obtain the relative rotation angle θ of the gimbal; Step 3: After completing step 2, map the points in the lidar coordinate system to the camera coordinate system based on the gimbal relative rotation angle θ and the calibration results; Step 4: After projecting the laser point to the camera coordinate system, sample the pixel points near the laser point to obtain Projecting it back to the LiDAR coordinate system, the final projection result is as follows: The narrow and long environment compensation method comprises the following steps: Step (1), installing the compensation device on the carrier system, and deploying the carrier system in a narrow and long environment; Step (2): Start the vehicle system and perform calibration according to step 1 of the monocular camera surround view point cloud density compensation method; determine according to the range of the point cloud scanning field, define the part exceeding the threshold as the depth range of the scanning field, use the gimbal to drag the monocular camera to the angle where the depth range is located, and operate according to steps 2, 3 and 4 of the point cloud density compensation method to perform point cloud density compensation on the narrow and long part of the lidar scanning field to obtain the compensation result.

3. The compensation method of the monocular camera surround view compensation device for a narrow and long environment laser radar according to claim 2, characterized in that: Start the system as described in step 1 and calibrate the LiDAR and monocular camera as follows: First, calibrate the monocular camera to obtain the camera's intrinsic parameter matrix K and extrinsic parameter rotation matrix R and translation vector Secondly, calibrate the laser radar and the monocular camera to obtain the rotation matrix φ of the laser radar coordinate system relative to the camera coordinate system and the translation vector of the laser radar coordinate system relative to the camera coordinate system 4. The compensation method of the monocular camera surround view compensation device for a narrow and long environment laser radar according to claim 2, characterized in that: The relative rotation angle θ described in step 2 is calculated as follows: Continue to collect lidar point cloud data With monocular camera data And use the program to control the rotation of the pan-tilt head to obtain the relative rotation angle θ of the pan-tilt head.

5. The compensation method of the monocular camera surround view compensation device for a narrow and long environment laser radar according to claim 2, characterized in that: The result of mapping the points in the lidar coordinate system to the camera coordinate system as described in step 3 is as follows: Obviously, the relative rotation angle of the gimbal does not produce a change in attitude relative to the z-axis; therefore, the angle containing the parameter θ is the sum of the initial rotation angle along the z-axis and the relative rotation angle θ, and the angle containing the parameter θ is already included in R and φ; Assuming the original displacement Get the original xOy plane camera angle The angle after the gimbal rotates is γ=θ0+θ, and the final displacement is : in

Images