Target detection method based on laser radar and camera fusion

Abstract

The invention relates to a target detection method based on laser radar and camera fusion, belongs to the technical field of multi-sensor target detection, and solves the problem that detection cannot be accurately and comprehensively carried out when a single sensor is used for target detection in the prior art. The method comprises the steps of obtaining an image shot by a camera and a laser radar point cloud; based on the linear features of the image and the point cloud edge points of the laser radar point cloud, performing external parameter calibration on the laser radar and the camera; respectively extracting feature points of the image and the laser radar point cloud feature points; projecting the laser radar point cloud feature points to the image by using an external parameter calibration result, and carrying out feature fusion on the laser radar point cloud feature points and feature points of the image; and performing target detection on a feature fusion result to obtain a target detection result.

Description

Technical Field

[0001] This invention relates to the field of multi-sensor target detection technology, and in particular to a target detection method based on the fusion of lidar and camera. Background Technology

[0002] Object detection is a crucial component of environmental perception systems for intelligent robots, with wide applications in autonomous driving, home robots, drones, and AGVs. Currently, commonly used sensors for object detection modules include RGB cameras and LiDAR. Cameras offer fast detection speeds and can capture rich texture information of the target, but they struggle to directly measure the object's shape and position. Furthermore, as a passive sensor, they are susceptible to changes in ambient light intensity. In contrast, LiDAR detects the surrounding environment using laser light, accurately measuring object distance and shape, and exhibiting strong robustness to lighting variations. However, even high-resolution LiDAR generates relatively sparse point cloud data. Therefore, fusing RGB images and point cloud images to obtain semantically rich, accurate distance information, and data less affected by lighting changes can provide reliable source data for detection algorithms, thereby improving detection accuracy.

[0003] Since cameras and LiDAR are different sensors with independent coordinate systems, they need to be calibrated in order to effectively perform feature fusion. Only through accurate calibration can we ensure that the data obtained from different sensors can be reliably combined. Summary of the Invention

[0004] Based on the above analysis, the embodiments of the present invention aim to provide a target detection method based on the fusion of lidar and camera, in order to solve the problem that existing methods using a single sensor cannot perform accurate and comprehensive target detection.

[0005] The objective of this invention is mainly achieved through the following technical solutions:

[0006] This invention provides a target detection method based on the fusion of lidar and camera, characterized by the following steps:

[0007] Acquire images captured by the camera and point clouds from the lidar;

[0008] Based on the linear features of the image and the edge points of the point cloud of the lidar, the external parameters of the lidar and the camera are calibrated.

[0009] Extract feature points from the image and feature points from the lidar point cloud, respectively;

[0010] The feature points of the lidar point cloud are projected onto the image using the extrinsic parameter calibration results, and feature fusion is performed between the feature points of the image and the extrinsic parameter calibration results.

[0011] The result of feature fusion is used for target detection to obtain the target detection result.

[0012] Furthermore, the step of projecting the feature points of the lidar point cloud onto the image using the extrinsic parameter calibration results includes:

[0013] Based on the external parameter calibration results, each feature point in the lidar point cloud is transformed from the lidar coordinate system to the camera coordinate system;

[0014] The feature points of the lidar point cloud, transformed to the camera coordinate system, are projected onto the image to obtain the point cloud planar feature points.

[0015] Furthermore, the feature points of the point cloud plane and the feature points of the image are fused using multimodal features to obtain fused features.

[0016] Furthermore, the extrinsic parameter calibration of the lidar and camera based on the straight line features of the image and the point cloud edge points of the lidar point cloud includes:

[0017] Extract the straight line features of the image and the point cloud edge points of the lidar point cloud data, map the point cloud edge points to the image, and match them with the straight line features of the image to obtain a matched straight line edge point feature pair;

[0018] Based on the well-matched feature pairs of the straight line edge points, an extrinsic parameter optimization model is constructed to obtain the camera-LiDAR extrinsic parameter calibration results.

[0019] Furthermore, the step of mapping the point cloud edge points to the image and matching them with the straight line features of the image to obtain a matched straight line edge point feature pair includes:

[0020] Based on camera intrinsic parameters, the point cloud edge mapping points on the image plane are obtained for each point cloud edge point;

[0021] Based on the position of each point cloud edge mapping point, determine whether there are straight line features in the image within the preset feature matching range;

[0022] When a straight line feature exists in the image, the nearest straight line feature is selected and matched with the point cloud edge mapping point to form a straight line edge point feature pair;

[0023] When there are no straight line features in the image, the point cloud edge mapping point matching fails.

[0024] Furthermore, the step of constructing an extrinsic parameter optimization model based on the matched straight line edge point feature pairs to obtain the camera-LiDAR extrinsic parameter calibration results includes:

[0025] Based on the distance between the midpoint cloud edge mapping points and the straight line features of the matched edge feature pairs, an extrinsic parameter optimization model is constructed to minimize the distance between the midpoint cloud edge mapping points and the straight line features of each straight line edge point feature pair.

[0026] Solve the Jacobian matrix of the extrinsic parameter optimization model to obtain the camera-LiDAR extrinsic parameter calibration results.

[0027] Furthermore, the model structure of the extrinsic parameter optimization model is as follows:

[0028]

[0029] Where ξ represents the camera-LiDAR extrinsic parameter value to be optimized; M represents the number of matched line edge point feature pairs; d i ∑ represents the distance between the midpoint cloud edge mapping point of the i-th straight line edge point feature and the straight line feature; -1 Let represent the covariance matrix.

[0030] Furthermore, the Jacobian matrix of the extrinsic parameter optimization model is as follows:

[0031]

[0032] Among them, f x This indicates the camera's focal length in the X direction; f y This indicates the camera's focal length in the Y direction; (X) i ',Y i ',Z i ') represents the coordinates of the point cloud edge point corresponding to the point cloud edge mapping point in the i-th straight line edge point feature pair in the camera coordinate system.

[0033] Furthermore, extracting point cloud edge points based on the lidar point cloud data includes:

[0034] Based on the coordinates of each point in the lidar point cloud data, the distance information between each point and the lidar is calculated.

[0035] Based on the distance information of each point and the distance information between each point and its neighboring points within the preset edge point cloud recognition range and the lidar device, the edge value of each point is calculated.

[0036] When the edge value is greater than a preset value, the point is taken as the edge point of the point cloud.

[0037] Furthermore, the edge values ​​of each point in the point cloud data are calculated using the following formula:

[0038]

[0039] Among them, EdgeVal p Represents the edge value of point p; N represents the number of neighboring points within the preset edge point cloud recognition range of point p; range i This represents the distance information between the i-th neighboring point and the lidar; range p This indicates the distance information between point p and the lidar.

[0040] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0041] 1. This invention uses multi-sensor fusion for target detection. After calibrating the external parameters of multiple sensors, the features acquired by each sensor are fused to obtain data with rich semantic information, accurate distance information, and less susceptible to changes in light. This provides reliable source data for the detection algorithm, thereby improving the detection accuracy.

[0042] 2. This invention utilizes the straight line features in the scene to achieve alignment and matching between image information and point cloud information. Based on the matching straight line edge point features, an extrinsic parameter optimization model is constructed to obtain high-precision extrinsic parameter calibration results and improve the stability of the calibration results. At the same time, it reduces manual intervention and improves the automation of the calibration process.

[0043] 3. Straight line features are usually obvious and easy to detect in images. Using these features can improve the accuracy of the calibration process. At the same time, straight line features are common in a variety of scenarios, making the calibration method more versatile.

[0044] 4. This invention reduces the complexity of feature matching and extrinsic parameter calculation through specific algorithm optimization, thereby improving the efficiency of the calibration process.

[0045] 5. Compared with the traditional calibration plate method, this invention does not rely on a specific calibration plate, reducing the sensitivity to environmental factors such as changes in lighting, shadows, and reflections. At the same time, the straight line feature has good robustness to noise and occlusion in the image. Even when some features are occluded or interfered with by noise, effective calibration can still be performed. Therefore, it has good robustness to different scenes and can maintain high calibration accuracy under different environmental conditions.

[0046] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0047] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0048] Figure 1 This is a flowchart illustrating a target detection method based on the fusion of lidar and camera in an embodiment of the present invention.

[0049] Figure 2 This is a flowchart illustrating the extrinsic parameter calibration algorithm for the camera and lidar in an embodiment of the present invention.

[0050] Figure 3 This is a schematic diagram of the straight line feature extraction result of an image in an embodiment of the present invention;

[0051] Figure 4 This is a schematic diagram illustrating the matching of point cloud edge mapping points and line features in an embodiment of the present invention. Detailed Implementation

[0052] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0053] A specific embodiment of the present invention discloses a target detection method based on the fusion of lidar and camera, such as... Figure 1 As shown, it includes the following steps S1-S5:

[0054] Step S1: Obtain images captured by the camera and point cloud data from the LiDAR.

[0055] Specifically, the device uses a camera mounted on a mobile platform to capture real-time images of the scene, and simultaneously uses a lidar unit installed on the platform to acquire point cloud information of the same scene. The mobile platform can be a vehicle or a robot.

[0056] Step S2: Based on the straight line features of the image and the point cloud edge points of the lidar point cloud, perform extrinsic parameter calibration on the lidar and the camera.

[0057] Specifically, since the camera and LiDAR have their own coordinate systems, the external parameters of the camera and LiDAR need to be calibrated to unify the coordinate systems before their features are fused.

[0058] like Figure 2As shown, during the external parameter calibration of the camera and LiDAR, since the captured image information will only be used to extract straight line features, the image captured by the camera is preprocessed to obtain a grayscale image. The preprocessing includes converting the image from the original RGB color space to the grayscale space by weighted summation, and then using a filter to reduce image noise.

[0059] For the point cloud information obtained by lidar, in this embodiment, the spatial state of the point is determined and its attributes are determined by the spatial information within the local range of each point in the point cloud to obtain the edge point of the point cloud. Therefore, for the obtained point cloud information, the spatial position (x, y, z) corresponding to each point in the point cloud is retained.

[0060] It should be noted that, in order to ensure that the final acquired external parameters are not affected by factors such as the movement of the device carrier, it is necessary to ensure that the time between each set of acquired images and point cloud information is strictly aligned, and that they are in the same scene, so as to ensure that feature point matching is not affected by time and space factors.

[0061] Further, the line features of the image and the point cloud edge points of the lidar point cloud are extracted. The point cloud edge points are mapped onto the image and matched with the line features of the image to obtain a matched line edge point feature pair.

[0062] Furthermore, the LSD algorithm is used to extract features from the image to extract the straight line features of the image.

[0063] Specifically, the LSD algorithm is a computer vision algorithm used to detect straight line segments in an image. It analyzes continuous pixels by calculating the gradient magnitude and direction of the image to filter out significant straight line segments. The result of the straight line feature extraction is as follows: Figure 3 As shown.

[0064] Furthermore, the point cloud edge points are extracted based on the lidar point cloud, including steps S211-S213:

[0065] Step S211: Based on the coordinates of each point in the lidar point cloud, calculate the distance information between each point and the lidar.

[0066] Specifically, by calculating the distance between each point in the point cloud and the lidar device, features such as edges, corners, and planes can be extracted from the point cloud. Therefore, based on the coordinates of each point in the lidar point cloud, the distance information between each point and the lidar is calculated using the Euclidean distance formula, as follows:

[0067]

[0068] Among them, range nThis represents the distance of the nth point in the lidar point cloud relative to the lidar; (x n y n , z n ) represents the spatial position of the nth point in the lidar point cloud within the lidar coordinate system.

[0069] Step S212: Based on the distance information of each point and the distance information between each point and its neighboring points and the lidar device within the preset edge point cloud recognition range, calculate the edge value of each point.

[0070] Specifically, since edge points typically have higher local variability than surrounding points, determining whether a point in the lidar point cloud data is a meaningful edge point is done by judging the local smoothness of each point. Furthermore, since local smoothness depends on local spatial information rather than global information, the local smoothness analysis method is robust to noisy and incomplete lidar point cloud data.

[0071] Furthermore, the edge values ​​of each point in the point cloud data are calculated using the following formula:

[0072]

[0073] Among them, EdgeVal p Represents the edge value of point p; N represents the number of neighboring points within the preset edge point cloud recognition range of point p; range i This represents the distance information between the i-th neighboring point and the lidar; range p This indicates the distance information between point p and the lidar.

[0074] When the scene is indoors, the preset edge point cloud recognition range can be smaller, for example, 0.1 meters to 0.5 meters. When the scene is outdoors, the preset edge point cloud recognition range can be larger, for example, 1 meter to 3 meters.

[0075] Step S213: When the edge value is greater than the preset value, the point is taken as the edge point of the point cloud.

[0076] Specifically, in point cloud data, edge points are usually located where the surface changes significantly, such as transitioning from a plane to a corner or from one curved surface to another. Edge points have a spatially distinct distribution pattern from other points. If the local curvature of a point is significantly different from the surrounding area, it marks an edge.

[0077] When the scene is indoors, the preset edge threshold can be set to a smaller value, for example, 0.01 meters to 0.05 meters. When the scene is outdoors, the preset edge threshold can be set to a larger value, for example, 0.1 meters to 0.5 meters.

[0078] Furthermore, the step of mapping the point cloud edge points to the image and matching them with the straight line features of the image to obtain a matched straight line edge point feature pair includes steps S221-S223:

[0079] Step S221: Based on the camera intrinsic parameters, obtain the point cloud edge mapping points on the image plane for each point cloud edge point.

[0080] Specifically, since the edge feature points obtained in the image are in the form of 2D pixel coordinates, while the edge points of the point cloud are 3D spatial location information, in order to unify the feature dimensions for matching, it is necessary to map the spatial 3D point cloud edge points to the 2D image.

[0081] Furthermore, based on the pinhole imaging principle, the point cloud edge points mapped onto the image plane are obtained using the following formula:

[0082]

[0083] Among them, (u j v j (x) represents the coordinates of the j-th point cloud edge mapping point; (x) j y j , z j ) represents the spatial coordinates of the j-th point cloud edge point; f x This indicates the camera's focal length in the X direction; f y This indicates the camera's focal length in the Y direction; (c x c y ) represents the coordinates of the camera's principal point.

[0084] Step S222: Based on the position of each point cloud edge mapping point, determine whether there are straight line features of the image within the preset feature matching range.

[0085] Specifically, after obtaining the point cloud edge points mapped to the point cloud points on the 2D image plane, such as... Figure 4 As shown, the corresponding line features of the matching image are selected using the principle of local nearest matching.

[0086] The judgment steps are as follows: determine whether there is a straight line feature passing through the area of ​​a circle with a preset radius and the edge mapping point of each point cloud. The method for determining whether there is a straight line feature passing through is to use the distance from the point to the line to judge the distance between the center of the circle and the straight line feature. If the distance is less than or equal to the preset radius, it is determined that a straight line feature passes through. If the distance is greater than the preset radius, it is determined that no straight line feature passes through.

[0087] For the preset radius value, when the scene is indoors, it can be smaller, for example, 0.1 meters to 0.5 meters; when the scene is outdoors, it can be larger, for example, 1 meter to 3 meters.

[0088] Step S223: When there is a straight line feature in the image within the preset feature matching range, select the nearest straight line feature and form a matching straight line edge point feature pair with the point cloud edge mapping point.

[0089] Specifically, from Figure 4 It can be seen that there are multiple straight line features within the preset feature matching range of point cloud edge mapping point P1. Therefore, the closest straight line L2 is selected as the matching straight line edge point feature pair.

[0090] Furthermore, the vertical distance between the point cloud edge mapping point and the straight line feature of the image within its preset feature matching range is calculated, and the straight line feature of the image with the shortest vertical distance is taken as the matching straight line feature.

[0091] Specifically, the characteristics of a straight line are represented in the standard form: Ax + By + C = 0, where A, B, and C are the coefficients of the straight line. A represents the degree of inclination of the straight line relative to the x-axis, B represents the degree of inclination of the straight line relative to the y-axis, and C represents the vertical offset of the straight line in the xy plane.

[0092] Therefore, the vertical distance can be calculated using the following formula:

[0093]

[0094] Where, d mk This represents the distance from the m-th point cloud edge mapping point to the k-th straight line feature; The coefficients representing the characteristics of the k-th straight line; This represents the coordinates of the m-th point cloud edge mapping point.

[0095] When there are no straight line features in the image, the point cloud edge mapping point matching fails.

[0096] Specifically, from Figure 4 As can be seen, point cloud edge mapping point P2 does not have a straight line feature within the preset feature matching range. Therefore, point cloud edge mapping point P2 is considered a point cloud edge mapping point that failed to match and will not participate in subsequent optimization calculations as a feature point.

[0097] Furthermore, based on the well-matched feature pairs of the straight line edge points, an extrinsic parameter optimization model is constructed to obtain the camera-LiDAR extrinsic parameter calibration results.

[0098] Furthermore, based on the distance between the midpoint cloud edge mapping point and the straight line feature of the matched straight line edge feature pair, an extrinsic parameter optimization model is constructed to minimize the distance between the midpoint cloud edge mapping point and the straight line feature of each straight line edge feature pair.

[0099] Specifically, the purpose of extrinsic parameter calibration is to determine the spatial relationship between the camera and the LiDAR, so that the data from the two sensors can be accurately aligned. The optimization model usually aims to minimize the reprojection error. For the distance residual (i.e., the distance from the point to the line) between the matched point cloud edge mapping point and the line feature, when the result is 0, it means that the optimization result of the corresponding model is the final correct result, which is also the desired extrinsic parameter calibration result between the camera and the LiDAR.

[0100] Furthermore, the model structure of the extrinsic parameter optimization model is as follows:

[0101]

[0102] Where ξ represents the camera-LiDAR extrinsic parameter value to be optimized; M represents the number of matched edge feature pairs; d i ∑ represents the distance between the midpoint cloud edge mapping point of the i-th straight line edge point feature and the straight line feature; -1 Let represent the covariance matrix.

[0103] Furthermore, the distance between the midpoint cloud edge mapping point and the line feature is calculated using the following formula:

[0104]

[0105] Among them, A i B represents the first linear coefficient of the feature of the midline feature of the i-th straight line edge point; i C represents the second line coefficient of the feature of the i-th line edge point in the midline feature; i Represents the third linear coefficient of the feature of the midline feature of the i-th line edge point; (u i v i ) represents the coordinate value of the midpoint cloud edge mapping point of the feature pair of the i-th straight line edge point.

[0106] Furthermore, the Jacobian matrix of the extrinsic parameter optimization model is solved to obtain the camera-LiDAR extrinsic parameter calibration results.

[0107] Specifically, the extrinsic optimization model is optimized by solving the Jacobian matrix of the extrinsic optimization model, so that the distance residual continuously approaches 0 until the optimization terminates within the threshold range. The corresponding optimization result is the optimal extrinsic parameters of the camera-LiDAR.

[0108] Furthermore, the Jacobian matrix of the extrinsic parameter optimization model is as follows:

[0109]

[0110] Among them, f x This indicates the camera's focal length in the X direction; f y This indicates the camera's focal length in the Y direction; (X) i ',Y i ',Z i ') represents the coordinates of the point cloud edge point corresponding to the point cloud edge mapping point in the i-th straight line edge point feature pair in the camera coordinate system.

[0111] Step S3: Extract the feature points of the image and the feature points of the lidar point cloud, respectively.

[0112] Specifically, the feature points of the image are points in the image that have unique local properties, that is, "can represent the image or target in the same or at least very similar invariant form in other similar images containing the same scene or target". The feature point extraction methods include SURF feature detectors, SIFT feature detectors, FAST feature detectors, and ORB feature detectors, etc.

[0113] The lidar point cloud feature points are representative or significant points extracted from point cloud data. These points typically reflect important geometric characteristics of an object's surface, such as corners, edges, or planar regions. Methods for extracting lidar point cloud feature points include extracting features based on the curvature of points in the point cloud, the normal vector of points in the point cloud, or by detecting lines in the point cloud.

[0114] Step S4: Project the feature points of the lidar point cloud onto the image using the extrinsic parameter calibration results, and perform feature fusion with the features of the image.

[0115] Furthermore, based on the external parameter calibration results, the coordinates of each feature point in the lidar point cloud are transformed from the lidar coordinate system to the camera coordinate system.

[0116] Specifically, the extrinsic parameter calibration results include the rotation matrix R and translation vector T from the lidar coordinate system to the camera coordinate system; the coordinates of each feature point in the lidar point cloud in the camera coordinate system are obtained using the following formula:

[0117]

[0118] in, This represents the coordinates of a feature point in the nth lidar point cloud in the camera coordinate system. This represents the coordinates of a feature point in the nth lidar point cloud in the lidar coordinate system.

[0119] Furthermore, the feature points of the lidar point cloud, transformed to the camera coordinate system, are projected onto the image to obtain the point cloud planar feature points.

[0120] Specifically, based on the pinhole imaging principle, the feature points of the lidar point cloud are projected onto the plane using camera intrinsic parameters to obtain the feature points of the point cloud plane, as shown in the following formula:

[0121]

[0122] in, f represents the coordinates of the nth point cloud feature point in the image plane; x This indicates the camera's focal length in the X direction; f y This indicates the camera's focal length in the Y direction; (c x c y ) represents the coordinates of the camera's principal point.

[0123] Furthermore, the feature points of the point cloud plane and the feature points of the image are fused using multimodal features to obtain fused features.

[0124] Specifically, as mentioned above, the point cloud acquired by the lidar and the image captured by the camera are information from the same scene acquired at the same time. Therefore, multimodal feature fusion is performed using methods such as weighted average fusion or Kalman filter fusion.

[0125] The weighted average fusion involves assigning different weights to feature points in the LiDAR point cloud and camera image, then averaging the results to generate a comprehensive feature representation. These weights can be allocated based on the importance or reliability of the features.

[0126] Step S4: Perform target detection on the feature fusion result to obtain the target detection result.

[0127] Specifically, the fused features can be used to obtain the final target detection result using target detection methods such as Region Convolutional Neural Network (R-CNN) or Fast R-CNN.

[0128] In summary, the target detection method based on the fusion of lidar and camera according to the embodiments of the present invention has the following beneficial effects:

[0129] 1. This invention uses multi-sensor fusion for target detection. After calibrating the external parameters of multiple sensors, the features acquired by each sensor are fused to obtain data with rich semantic information, accurate distance information, and less susceptible to changes in light. This provides reliable source data for the detection algorithm, thereby improving the detection accuracy.

[0130] 2. This invention utilizes the straight line features in the scene to achieve alignment and matching between image information and point cloud information. Based on the matching straight line edge point features, an extrinsic parameter optimization model is constructed to obtain high-precision extrinsic parameter calibration results and improve the stability of the calibration results. At the same time, it reduces manual intervention and improves the automation of the calibration process.

[0131] 3. Straight line features are usually obvious and easy to detect in images. Using these features can improve the accuracy of the calibration process. At the same time, straight line features are common in a variety of scenarios, making the calibration method more versatile.

[0132] 4. This invention reduces the complexity of feature matching and extrinsic parameter calculation through specific algorithm optimization, thereby improving the efficiency of the calibration process.

[0133] 5. Compared with the traditional calibration plate method, this invention does not rely on a specific calibration plate, reducing the sensitivity to environmental factors such as changes in lighting, shadows, and reflections. At the same time, the straight line feature has good robustness to noise and occlusion in the image. Even when some features are occluded or interfered with by noise, effective calibration can still be performed. Therefore, it has good robustness to different scenes and can maintain high calibration accuracy under different environmental conditions.

[0134] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A target detection method based on laser radar and camera fusion, characterized in that, The method comprises the following steps: obtaining an image captured by a camera and a laser radar point cloud; performing extrinsic calibration of the laser radar and the camera based on straight line features of the image and point cloud edge points of the laser radar point cloud; extracting feature points of the image and feature points of the laser radar point cloud respectively; projecting the feature points of the laser radar point cloud to the image using the extrinsic calibration result, and performing feature fusion with the feature points of the image; performing target detection on the feature fusion result to obtain a target detection result.

2. The method of claim 1, wherein, The projecting of the feature points of the laser radar point cloud to the image using the extrinsic calibration result comprises: based on the extrinsic calibration result, converting each feature point in the laser radar point cloud from a laser radar coordinate system to a camera coordinate system; projecting each feature point of the laser radar point cloud converted to the camera coordinate system to the image to obtain point cloud plane feature points.

3. The method of claim 2, wherein, performing multi-modal feature fusion on the point cloud plane feature points and the feature points of the image to obtain fused features.

4. The method of claim 1, wherein, The extrinsic calibration of the laser radar and the camera based on the straight line features of the image and the point cloud edge points of the laser radar point cloud comprises: extracting straight line features of the image and point cloud edge points of the laser radar point cloud data, mapping the point cloud edge points to the image, and matching the mapped point cloud edge points with the straight line features of the image to obtain matched straight line edge point feature pairs; based on the matched straight line edge point feature pairs, constructing an extrinsic optimization model to obtain a camera-laser radar extrinsic calibration result.

5. The method of claim 4, wherein, The mapping of the point cloud edge points to the image and the matching of the mapped point cloud edge points with the straight line features of the image to obtain matched straight line edge point feature pairs comprises: based on camera intrinsic parameters, obtaining point cloud edge mapping points of each point cloud edge point on an image plane; based on the position of each point cloud edge mapping point, determining whether there is a straight line feature of the image within a preset feature matching range; when there is a straight line feature of the image, selecting the closest straight line feature to form a matched straight line edge point feature pair with the point cloud edge mapping point; when there is no straight line feature of the image, the point cloud edge mapping point fails to match.

6. The method of claim 4, wherein, The construction of the extrinsic optimization model based on the matched straight line edge point feature pairs to obtain the camera-laser radar extrinsic calibration result comprises: based on the distance between the point cloud edge mapping points and the straight line features in the matched edge feature pairs, constructing an extrinsic optimization model to minimize the distance between the point cloud edge mapping points and the straight line features in each straight line edge point feature pair; solving the Jacobian matrix of the extrinsic optimization model to obtain the camera-laser radar extrinsic calibration result.

7. The method of claim 6, wherein, The model structure of the extrinsic optimization model is as follows: wherein, ξ represents the camera-lidar extrinsic parameter value to be optimized; M represents the number of matched linear edge point feature pairs; d i represents the distance between the point cloud edge mapping point and the linear feature in the i-th linear edge point feature pair; ∑ -1 represents the covariance matrix.

8. The method of claim 7, wherein, The Jacobian matrix of the extrinsic optimization model is as follows: wherein f x represents the focal length of the camera in the X direction; f y represents the focal length of the camera in the Y direction; (X i , Y i , Z i ) represents the coordinates of the spatial position of the point cloud edge point corresponding to the point cloud edge mapping point in the i-th linear edge point feature pair in the camera coordinate system.

9. The method of claim 4, wherein, The extraction of point cloud edge points based on the laser radar point cloud data comprises: based on the coordinates of each point in the laser radar point cloud data, calculating distance information of each point from the laser radar; based on the distance information of each point and the distance information of neighboring points of each point from the laser radar device within a preset edge point cloud recognition range, calculating an edge value of each point; when the edge value is greater than a preset value, the point is regarded as a point cloud edge point.

10. The method of claim 9, wherein, The edge value of each point in the point cloud data is calculated using the following formula: wherein, EdgeVal p represents the edge value of point p; N represents the number of neighboring points within a preset edge point cloud recognition range of point p; range i represents the distance information of the i-th neighboring point from the lidar; range p represents the distance information of point p from the lidar.

Images