A 3D target detection method based on the fusion of multi-focal length camera and lidar

Abstract

The present invention seeks to protect a three-dimensional target detection method based on the fusion of a multi-focal length camera and a laser radar, which belongs to the field of autonomous driving and three-dimensional target detection. The method comprises the following steps: 1. The multi-focal length camera is composed of a long-focus and short-focus camera, and the data of the multi-focal length camera and the laser radar are processed synchronously in time and space. 2. The multi-focal length camera image is semantically segmented by a segmentation network, and the segmentation result is used to generate a viewing cone, and the laser radar point cloud is projected onto the viewing cone formed by the long-focus and short-focus images to perform three-dimensional target detection. 3. The multi-scale image pyramid is used to achieve accurate matching of feature points of long-focus and short-focus images, and the connection between the targets of the long-focus and short-focus segmented images is established through feature matching, and the data of the three-dimensional target is associated and fused through this connection. The present invention can be applied to autonomous driving or assisted driving vehicles to perceive the environment around the vehicle, and has the characteristics of accurate classification and high positioning accuracy, especially improving the detection accuracy of long-distance targets.

Description

Technical Field

[0001] The present invention belongs to the fields of autonomous driving, three-dimensional target detection, laser radar, etc., and specifically relates to a three-dimensional target detection method that integrates long-focus and short-focus cameras with laser radar. Background Art

[0002] With the rapid development of assisted and intelligent driving technologies, their safety is receiving increasing attention. Drivers demand advanced autonomous driving technology to relieve fatigue while also demanding improved safety. The safety of smart cars is inextricably linked to their ability to perceive their surroundings. Accurately identifying objects and obstacles on the road helps autonomous vehicles plan their routes and implement emergency control measures.

[0003] In recent years, deep neural networks for object detection based on images and radar point clouds have experienced rapid advancement. Image-based object detection detects the location and category of an object in an image and outlines it with a rectangular box. However, it cannot precisely determine the object's position in 3D world coordinates. LiDAR can accurately detect the location of all objects around a vehicle, and point cloud-based neural networks can also identify information such as the category and location of an object. However, point cloud data is sparse and lacks semantic information, meaning only a few location points can be detected for distant objects. Images contain rich semantic information but little location information. Combining image and LiDAR point cloud information for object detection is a promising approach.

[0004] Object detection is rapidly developing, and environmental perception is also driving demand for obtaining three-dimensional information and the center position of a target. Binocular cameras can simulate the human eye and obtain depth information in a scene through disparity estimation. However, this disparity estimation relies on precise calibration, resulting in poor depth estimation accuracy and inability to accurately determine the target's position.

[0005] In summary, three-dimensional target detection is the general trend in the field of target detection and is currently developing rapidly. However, current detection still has problems with inaccurate recognition and position estimation of distant targets.

[0006] After searching, the reference document 1, application number 202011288888.5, is a method for target detection and fusion in a long- and short-focus multi-camera vehicle environment. The method includes the following steps: 1. Using a convolutional neural network to detect targets in images captured by long- and short-focus binocular cameras, the target frame position in images captured by cameras with different focal lengths at the same time is obtained. 2. Based on the camera imaging principle and the internal and external parameters K, R, and T obtained by camera calibration, the mapping relationship f between the spatial target point P and the pixel coordinate system of the long- and short-focus cameras can be obtained. 3. The target frame position in the long-focus camera image is used to obtain the position of the corresponding target frame in the short-focus camera image through the mapping relationship f, and then fused with the target in the original short-focus camera image to achieve the target detection task under different distance conditions. Although this invention overcomes the limitation of a single focal length camera that cannot adapt to target detection at different distances, the depth information of the target obtained by visual technology alone is still not accurate enough, while radar sensors can provide accurate depth information. In addition, the fusion of long- and short-focus cameras with radar can obtain a larger environmental perception range.

[0007] Comparative Document 2, Application No. 202011520256.7 A target detection method based on millimeter-wave radar prior positioning and visual feature fusion, the method comprising: based on the calibrated millimeter-wave radar and the vehicle-mounted camera, simultaneously acquiring millimeter-wave radar point cloud data and camera images; performing spatial three-dimensional coordinate transformation on the millimeter-wave radar point cloud data to project it onto the camera plane; based on the projected millimeter-wave radar point cloud data, generating multiple anchor samples according to a pre-set anchor strategy, and obtaining the final anchor sample based on the speed distance weight of each candidate area; obtaining the features of the final sample by fusing the RGB information of the camera image and the scattering cross-section intensity information of the millimeter-wave radar point cloud data; inputting the features of the final anchor sample into the detection network to generate the category and position information of the target in the scene. Although this invention improves the target detection effect in all-weather scenes, it still cannot effectively detect distant targets in the scene. The present invention effectively improves the detection accuracy of distant targets in the scene by fusing the information of long and short focal length cameras, and by fusing radar data, our method can also adapt to all-weather scenes.

[0008] Comparative Document 3, Application No. 201910814532.1, a fusion SLAM method based on binocular camera and millimeter wave radar, characterized by: comprising the following steps: collecting images and radar data through binocular camera (100) and millimeter wave radar (200); processing the image and radar data respectively to obtain a depth map corresponding to the image, and mapping the radar data to the image; fusing the processing results of the image and radar data, and establishing a SLAM map based on the fusion results. This invention is mainly used for SLAM mapping, while our method focuses on improving the performance of target detection. In addition, our method uses data information from long and short focal length cameras for fusion, which enables our target detection method to detect targets at a longer distance. Summary of the Invention

[0009] The present invention aims to solve the above problems of the prior art. It proposes a three-dimensional target detection method based on the fusion of a multi-focal length camera and a lidar. The technical solution of the present invention is as follows:

[0010] A three-dimensional target detection method based on the fusion of a multi-focal length camera and a laser radar comprises the following steps:

[0011] Long- and short-focus cameras are used to capture images in front of the vehicle and perceive the environment in front of the vehicle. LiDAR perceives the environment around the vehicle. The multi-focal-length camera and LiDAR are jointly calibrated, and the image data and the point cloud data collected by the LiDAR are synchronized in time and space through a spatiotemporal synchronization algorithm.

[0012] Use a neural network to semantically segment the long and short focus images respectively, use the segmentation results to generate the cones of the long and short focus images, project the point clouds onto the cones formed by the segmented images of the long and short focus cameras respectively, use the point clouds in the cones as the target point clouds, calculate the centroid of the target point cloud, set a threshold, and directly input the point cloud network to predict the target's three-dimensional information if the centroid depth is less than the threshold. If the centroid depth is greater than the threshold, the point cloud is pre-processed and the target's three-dimensional information is predicted;

[0013] The multi-scale image pyramid is used to match the feature points of long- and short-focus images. The connection between the long- and short-focus segmented image targets is established through feature matching. The data association and data fusion of the three-dimensional targets are achieved through this connection.

[0014] Furthermore, the vehicle perceives the surrounding environment and obtains the size, position and posture of the vehicles around it, specifically the relative position of the target vehicle, the length, width, height and yaw angle of the vehicle.

[0015] Furthermore, the use of long and short focal length cameras to capture images in front of the vehicle and perceive the environment in front of the vehicle, and the lidar to perceive the environment around the vehicle, specifically includes:

[0016] The multi-focal length camera consists of two cameras with different focal lengths. It is used to observe the front of the vehicle. The multi-focal length camera is placed horizontally, with the short-focus camera on the left and the long-focus camera on the right. The two cameras are close to each other, have a common field of view, and collect images simultaneously.

[0017] The lidar is placed on the roof to sense the environment around the vehicle and select the image data collected by the multi-focal camera at the timestamp when the lidar scans the area directly in front of the vehicle. The data collection interval between the two sensors is less than 0.01 seconds.

[0018] Furthermore, the joint calibration of the multi-focal length camera and the lidar performs spatiotemporal synchronization on the image data and the point cloud data collected by the lidar through a spatiotemporal synchronization algorithm, specifically including:

[0019] Establish a world coordinate system, calibrate the long and short focus cameras, obtain their respective intrinsic and extrinsic parameters, and obtain the rotation and translation matrices of the world coordinate system and the short focus camera as R1 and T1, and the rotation and translation matrices of the long focus camera as R2 and T2; the non-homogeneous linear coordinates of a point P in the world coordinate system, the long focus camera coordinate system, and the short focus camera coordinate system are but:

[0020]

[0021]

[0022] Combining the two equations, we can get

[0023]

[0024]

[0025] R 12 、T 12 The rotation and translation matrix between the long and short focus cameras, the rotation and translation matrix between the short focus camera and the lidar R3, T3, the rotation and translation matrix between the long focus camera and the lidar R 23 、T 23 .

[0026] Furthermore, the image data and the point cloud data collected by the lidar are synchronized in time and space by using a time and space synchronization algorithm, specifically including:

[0027] The data collected by the lidar and the multi-focal length camera are selected with the closest timestamps to obtain time-synchronized point cloud data and multi-focal length camera image data.

[0028] Based on the rotation and translation matrices of the LiDAR and the long- and short-focus cameras, as well as the intrinsic parameter matrices of the long- and short-focus cameras, combined with time-synchronized point cloud data and multi-focal length camera data, the pixel coordinates of any point in the point cloud in the image can be obtained. This achieves spatial synchronization, completing the spatiotemporal synchronization of image data and point cloud data.

[0029] Furthermore, the semantic segmentation of the long and short focus images using a neural network and generating the visual cones of the long and short focus images using the segmentation results specifically includes:

[0030] Use a semantic segmentation network to semantically segment the long-focus and short-focus camera images respectively to obtain the target category c i and semantic segmentation regions;

[0031] Semantic segmentation networks classify each pixel in an image into a category, with objects in the same category assigned the same category label. Convolutional and pooling layers are typically used to extract image features, reducing the image resolution while adding feature layers. Deconvolutional layers then restore the image resolution to its original size, ultimately determining the category and pixel location of objects in the image.

[0032] The obtained semantic segmentation area is combined with the calibrated long- and short-focus camera intrinsic parameters and projection matrix to generate a viewing cone. The RT matrix, a transformation relationship between the lidar and the camera, is used to project the point cloud data obtained by the lidar into the viewing cone, filter out the point cloud outside the viewing cone, and retain the point cloud inside the viewing cone.

[0033] Vision cone: Figure 2 As shown, a pyramid shape is generated by the optical axis and this region in an image. All planar objects from the phase plane to distant objects are imaged in this region. Semantic segmentation is used to detect the edges of the object, and a viewing cone is generated from these edges. Based on the camera's intrinsic parameters and projection matrix, the 3D coordinates of the camera's optical center and a point in the image plane are obtained. A straight line in 3D space can be drawn from these two points. Any point on this line, away from the image plane in the direction of the optical center, has the same pixel coordinates in the image. A viewing cone is established between all pixels along the edge of the semantically segmented target mask and the optical center.

[0034] Furthermore, the centroid of the target point cloud is obtained, a threshold is set, and if the centroid depth is less than the threshold, the three-dimensional information of the target is directly input into the point cloud network, and if the centroid depth is greater than the threshold, the three-dimensional information of the target is predicted after the point cloud is pre-processed; specifically, the following steps are included:

[0035] The random consistency algorithm is used to calculate the center point of each target’s point cloud as the center of mass of the target (x i ,y i ,z i );

[0036] Set a distance threshold z t, when the depth information z of the target center of mass in the viewing cone i When the value is less than the threshold, the point cloud in the viewing cone is input into the point cloud detection network to obtain the target category, center point position, length, width, height and yaw angle;

[0037] When the depth information z of the target center of mass in the viewing cone i When the threshold is greater than the target point cloud, the right-hand coordinate system is established with the center of mass of the target point cloud as the origin, the optical center direction of the camera coordinate system as the z-axis, and the right as the x-axis, and a three-dimensional box is established with the spatial range of the target point cloud. The point cloud center of mass, the three-dimensional box and the target category c obtained by segmentation are combined. i Input into the neural network to regress the center of mass and 3D box of the target.

[0038] Furthermore, the random consistency algorithm is used to calculate the center point of each target point cloud as the center of mass of the target (x i ,y i ,z i );

[0039] The random sampling consensus algorithm is an iterative method for estimating the parameters of a mathematical model from a set of observation data containing outliers. Randomly select a point (x, y, z) in a point cloud in a cone and calculate its Euclidean distance and distance to all points in the cone. (N is the total number of point clouds in the viewing cone, j is a point cloud in the viewing cone), iterate g times, each selected point cannot be the same as the previous one, and take the point (x i ,y i ,z i ) as the center of mass of the target.

[0040] Furthermore, the method of using a multi-scale image pyramid to achieve matching of feature points of long- and short-focus images and establishing a connection between objects in the long- and short-focus segmented images through feature matching specifically includes:

[0041] According to the focal length ratio of the long-focus and short-focus cameras, multi-scale feature pyramids of the long-focus and short-focus images at the same moment are established to extract the feature points of the long-focus and short-focus images;

[0042] The image is scaled to obtain a multi-scale feature pyramid of the image, so that the target in the long-focus image has the same scale as the target in the short-focus image. The specific algorithm for matching feature points in the long- and short-focus multi-scale feature pyramid is as follows;

[0043] According to the focal length ratio s of the long-focus and short-focus cameras and the projection matrix of the long-focus and short-focus cameras, the feature points of the nth layer of the long-focus multi-scale image pyramid are matched with the feature points of the 0th layer of the short-focus multi-scale image pyramid. Each matching of the long-focus and short-focus images is performed every n-1 layers.

[0044] The feature points of the long and short focus image matching are retained, and the feature points are mapped back to the long and short focus segmentation image according to the scale scaling factor. For the case where both the long and short focus image segmentation masks have feature points, the connection between the long and short focus segmentation image objects is established.

[0045] Furthermore, the data association and fusion of the three-dimensional target detection results include the following steps:

[0046] For targets that are connected in the long- and short-focus segmentation images, the target category, center position, length, width, height, and yaw angle are queried based on the association of the visual cone. Since distant targets appear in a larger pixel range in the long-focus image, edge detection is more accurate. The category, center position, length, width, height, yaw angle, and other information detected in the long-focus image are used for the target; targets that are not connected in the long- and short-focus target segmentation are treated as new targets, and the three-dimensional information of the fused target and the new target are mapped to the short-focus image and point cloud image for easy observation.

[0047] The advantages and beneficial effects of the present invention are as follows:

[0048] 1) The present invention utilizes a multi-focal length camera combination. The short-focus camera detects targets within a larger field of view, and the long-focus camera detects targets at farther locations. The short-focus image presents fewer pixels for distant targets and has poor recognition accuracy. The long-focus field of view is relatively narrow and cannot present nearby targets. This method combines the long-focus and short-focus cameras, taking advantage of each to solve the problem of detecting targets with a larger field of view and farther away in front of the vehicle.

[0049] 2) The present invention uses a semantic segmentation network to segment targets on an image, and uses a segmentation mask to generate a cone of view method, making full use of the semantic information of the image to retrieve the target, avoiding target omission and improving the hit rate of target detection.

[0050] 3) The present invention projects the point cloud onto the viewing cone, obtains the center point of the target based on the Euclidean distance, and classifies the predicted structure according to the depth information of the center point. First, the search range of three-dimensional target detection is reduced, the amount of calculation is reduced, and the calculation time is saved. Secondly, the fastest point cloud detection method is directly used for targets with depth less than a threshold, which speeds up the prediction speed. Finally, regression is performed on targets greater than the threshold, which can detect the three-dimensional information of distant targets.

[0051] 4) This paper proposes a new fusion strategy. It utilizes a multi-scale image pyramid feature point matching method to correlate long- and short-focus image features, establish connections between long- and short-focus image segmentation targets, and thereby establish connections between three-dimensional targets, thereby associating and fusing target data. This data association is precise, avoiding mixing and reuse of targets, and achieving target-level fusion, with wide target coverage and minimal omissions.

[0052] Conventional methods use a rectangular frame generated by target detection in an image to form a viewing cone. This method uses an image semantic segmentation mask to form a viewing cone. The viewing cone fits the edge of the object and can more accurately filter out non-target point clouds in the viewing cone, thereby more accurately obtaining point cloud information of the vehicle target.

[0053] Projecting the point cloud onto the viewing cone filters out the background point cloud and only retains the foreground point cloud containing the target. During prediction, only the point cloud containing the target is used for prediction, which reduces the computational complexity of point cloud prediction and speeds up the 3D target detection.

[0054] The novelty of using center point depth information to predict the structure lies in the fact that images are rich in semantic information but lack depth information. Currently, image-based 3D object detection cannot predict the 3D information of distant targets, and the accuracy of close-range 3D object detection is lower than that of lidar-based methods. LiDAR provides precise distance information but lacks semantic information. The point cloud of distant targets is sparse, and it cannot obtain good semantic information. For close-range 3D object detection, lidar-based 3D object detection methods are generally faster than image-based methods. Common methods use single-sensor data to detect 3D target information, which can only predict 3D information for close-range targets. For close-range targets with sufficient point cloud information, lidar-based 3D object detection can accelerate 3D object detection. For distant targets, using image semantic segmentation categories as input, 3D bounding box regression is performed on the point cloud, resulting in more accurate 3D target category prediction. Initializing the 3D object box regression to the target 3D bounding box reduces feature extraction and computational complexity. Conventional methods based on monocular image feature point detection or depth estimation cannot predict distant objects, and even the accuracy of close-range 3D object detection is poor. LiDAR-based 3D object detection methods, due to a lack of semantic information, suffer from poor category prediction and are unable to predict the category of distant 3D objects, often missing distant objects. Therefore, this method uses object depth to classify 3D object detection methods, reducing the computational complexity of 3D object detection and accelerating inference. It is capable of predicting the category of small distant objects and regressing their 3D bounding boxes.

[0055] The innovation of the new fusion strategy lies in utilizing a multi-scale image pyramid feature point matching method to associate long- and short-focus image features, establish connections between long- and short-focus image segmentation targets, and thereby establish connections between three-dimensional targets, thereby associating and fusing target data. This is not available in conventional methods. Conventional methods use projection to project point cloud information onto images, and the three-dimensional target detection results are highly dependent on the image detection results. The feature points of an image can accurately match similar information between two images. Even if there are similar categories, the feature points are different. The feature points of the two images are matched and their positions can also correspond to each other. The size ratios of the same targets in long- and short-focus images are different, which is a difficult problem for feature point matching. The use of a multi-scale image feature pyramid can effectively solve this problem. Therefore, the semantic segmentation targets of long- and short-focus images can correspond well, and the long-focus three-dimensional target detection results can correspond well with the short-focus three-dimensional target detection results, making data association more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a flowchart of a specific implementation of a preferred embodiment of the present invention;

[0057] Figure 2 View frustum generation and point cloud projection;

[0058] Figure 3 Matching diagram of long and short focal length cameras;

[0059] Figure 4 Feature pyramid matching diagram for long and short focal length cameras. DETAILED DESCRIPTION

[0060] The following will describe the technical solutions in the embodiments of the present invention in detail with reference to the accompanying drawings. The described embodiments are only a part of the embodiments of the present invention.

[0061] The technical solution of the present invention to solve the above technical problems is:

[0062] In response to the above technical problems, the present invention proposes a three-dimensional target detection method based on the fusion of a multi-focal length camera and a lidar.

[0063] A 3D target detection method based on the fusion of a multi-focal-length camera and a laser radar is used for a vehicle to perceive surrounding vehicles. The multi-focal-length camera is a pair of cameras with different focal lengths. The 3D target detection method includes the types of vehicles around the vehicle, the positions of the centers of the surrounding vehicles relative to the vehicle, and the length, width, height, and yaw angle of the surrounding vehicles. The method includes the following steps:

[0064] 1. The multi-focal-length camera consists of cameras with different focal lengths. The long- and short-focus cameras are jointly calibrated, and the long- and short-focus cameras and lidar are synchronized in time and space. The long- and short-focus cameras primarily capture images in front of the vehicle and perceive the environment in front of the vehicle, while the radar perceives the environment around the vehicle. The multi-focal-length cameras are jointly calibrated, and a spatiotemporal synchronization algorithm is used to synchronize the image data with the point cloud data collected by the lidar.

[0065] 11) The images captured by the telephoto and short-focus cameras are RGB three-channel color images, and the data collected by the lidar is four-dimensional data (x, y, z, a), where (x, y, z) represents the three-dimensional position of the lidar reflection point and a represents the reflection intensity.

[0066] 12) A multi-focal camera consists of two cameras with different focal lengths. It is used to observe the front of the vehicle. The optical centers of the multi-focal cameras are parallel and located in the same plane. The multi-focal cameras are 30 cm apart and have a common field of view. The multi-focal cameras collect images simultaneously.

[0067] 13) The laser radar is placed on the roof to sense the environment around the vehicle. The image data collected by the multi-focal length camera at the timestamp of the laser radar scanning directly in front of the vehicle is selected. The data collection interval between the two sensors is less than 0.01 seconds.

[0068] 14) Place a calibration plate in front of the vehicle. The calibration plate is located in the common field of view of the multi-focal length camera. The calibration plate is placed in different postures. The multi-focal length camera collects images and calibrates the internal and external parameters of the multi-focal length camera.

[0069] 15) Establish a world coordinate system, calibrate the telephoto and short focal length cameras, and obtain their respective intrinsic and extrinsic parameters. The rotation and translation matrices of the world coordinate system and the short focal length camera are R1 and T1, and the rotation and translation matrices of the telephoto camera are R2 and T2. The non-homogeneous linear coordinates of a point P in the world coordinate system, the telephoto camera coordinate system, and the short focal length camera coordinate system are: but:

[0070]

[0071]

[0072] Combining the two equations, we can get

[0073]

[0074]

[0075] 16)R 12 、T 12 is the rotation and translation matrix between the long and short focus cameras, and the rotation and translation matrix between the short focus camera and the laser radar R3 and T3. The rotation and translation matrix between the long focus camera and the laser radar R23 、T 23 .

[0076] 2. Use a neural network to semantically segment the long-focus and short-focus images respectively, use the segmentation results to generate a viewing cone, project the point cloud onto the viewing cone formed by the long-focus and short-focus camera segmentation images respectively, obtain the center of mass of the target point cloud, set a threshold, and directly input the point cloud network to predict the target's three-dimensional information if the center of mass depth is less than the threshold. If the center of mass depth is greater than the threshold, the point cloud is preprocessed and the target's three-dimensional information is predicted.

[0077] 21) Use an image segmentation network to semantically segment the target in the long and short focal length camera images and obtain the target segmentation score c i and segmentation mask M i , which is the edge of the target pixel in the image.

[0078] 22) The obtained semantic segmentation area is used in combination with the calibrated long- and short-focus camera intrinsic parameters and projection matrix to generate a viewing cone. The point cloud data obtained by the lidar is projected into the viewing cone using the conversion relationship R_T matrix between the lidar and the camera. The point cloud outside the viewing cone is filtered out, and the point cloud inside the viewing cone is retained.

[0079] 23) Using the random consistency algorithm and the Euclidean distance of the point cloud, the center point of each target’s point cloud is calculated as the center of mass of the target (x i ,y i ,z i ).

[0080] 24) Set a distance threshold z t , when the depth information z of the target center of mass in the viewing cone i When it is less than the threshold, z is set to 40 meters, and the point cloud 3D target detection network is used to detect the 3D information of the target.

[0081] 25) When the depth information z of the target center of mass in the viewing cone i When it is greater than the threshold, a right-handed coordinate system is established with the center of mass of the target point cloud as the origin, the optical center direction of the camera coordinate system as the z-axis, and the right as the x-axis, and a three-dimensional frame is established with the spatial range of the target point cloud.

[0082] 26) Input the segmented categories and point cloud obtained in step 25) into the neural network to predict the position, length, width, height and yaw angle of the target relative to the vehicle.

[0083] 3. Use multi-scale image pyramid to achieve accurate matching of feature points of long- and short-focus images, establish the connection between long- and short-focus segmented image targets through feature matching, and use this connection to associate and fuse data of three-dimensional targets.

[0084] 31) According to the focal length ratio of the long-focus camera and the short-focus camera, a multi-scale feature pyramid is established for the long-focus and short-focus images at the same moment.

[0085] 32) Extract feature points of long and short focus images.

[0086] 33) Due to the inconsistency in the size and shooting position of the long-focus and short-focus images, it is not convenient to directly match the feature points of the long-focus and short-focus images. The images are scaled to obtain a multi-scale feature pyramid of the image so that the objects in the long-focus image and the short-focus image have the same scale.

[0087] 34) According to the focal length ratio s of the long-focus and short-focus cameras and the projection matrix of the long-focus and short-focus cameras, the feature points of the nth layer of the long-focus multi-scale image pyramid are matched with the feature points of the 0th layer of the short-focus multi-scale image pyramid. Each matching of the long-focus and short-focus images is performed every n-1 layers.

[0088] 35) Retain the feature points of the long and short focus image matches, and map the feature points back to the long and short focus segmented images according to the scale scaling factor.

[0089] 36) For the case where both the long and short focus image segmentation masks have feature points, establish the connection between the long and short focus segmentation image targets.

[0090] 37) For targets that are connected by the long- and short-focus segmented images, the target category, center position, length, width, height, and yaw angle are queried based on the association of the visual cone. Since distant targets appear in a larger pixel range in the long-focus image, edge detection is more accurate. The category, center position, length, width, height, yaw angle, and other information detected by the long-focus image are used for the target.

[0091] 38) Segment the long-focus and short-focus targets and unconnected targets as new targets.

[0092] 39) Mapping the fused three-dimensional information of the target and the new target onto the short-focus image and the point cloud image for easy observation.

[0093] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0094] The above embodiments should be understood as merely illustrating the present invention and not as limiting the scope of protection of the present invention. After reading the contents of the present invention, technicians may make various changes or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A three-dimensional target detection method based on the fusion of multi-focal length camera and laser radar, characterized in that: The following steps are involved: Long- and short-focus cameras are used to capture images in front of the vehicle and perceive the environment in front of the vehicle. LiDAR perceives the environment around the vehicle. The multi-focal-length camera and LiDAR are jointly calibrated, and the image data and the point cloud data collected by the LiDAR are synchronized in time and space through a spatiotemporal synchronization algorithm. Use a neural network to semantically segment the long and short focus images respectively, use the segmentation results to generate the cones of the long and short focus images, project the point clouds onto the cones formed by the segmented images of the long and short focus cameras respectively, use the point clouds in the cones as the target point clouds, calculate the centroid of the target point cloud, set a threshold, and directly input the point cloud network to predict the target's three-dimensional information if the centroid depth is less than the threshold. If the centroid depth is greater than the threshold, the point cloud is pre-processed and the target's three-dimensional information is predicted; A multi-scale image pyramid is used to match the feature points of long- and short-focus images. The connection between the long- and short-focus segmented image objects is established through feature matching. Through this connection, data association and data fusion are performed on the three-dimensional objects. The method of using a neural network to semantically segment the long-focus and short-focus images respectively and using the segmentation results to generate the visual cones of the long-focus and short-focus images specifically includes: Use a semantic segmentation network to semantically segment the long-focus and short-focus camera images respectively to obtain the target category c i and semantic segmentation areas; semantic segmentation networks refer to classifying each pixel in an image into a category, with objects of the same category having the same category label; generally, image features are extracted by convolutional layers and pooling layers, the image resolution is reduced, the feature layer is increased, and the deconvolution layer restores the image resolution to the original image size to obtain the category and pixel position of the object in the image; The obtained semantic segmentation area is combined with the calibrated long and short focus camera intrinsic parameters and projection matrix to generate a viewing cone. The point cloud data obtained by the lidar is projected into the viewing cone using the RT matrix, a transformation relationship between the lidar and the camera. The point cloud outside the viewing cone is filtered out, and the point cloud inside the viewing cone is retained. A pyramid shape is generated in an area of ​​the cone image by the optical axis and the area. All objects from the phase plane to distant planes are imaged in this area. Semantic segmentation is used to obtain the edge of the detected object, and the cone is generated from this edge. Based on the camera's intrinsic parameters and projection matrix, the three-dimensional coordinates of the camera's optical center and the three-dimensional coordinates of a point in the image plane are obtained. A straight line in three-dimensional space can be drawn from the two points. The cone surface can be established by connecting all pixel points along the mask edge of the target obtained by semantic segmentation and the optical center. The method of obtaining the centroid of the target point cloud, setting a threshold, directly inputting the target three-dimensional information into the point cloud network if the centroid depth is less than the threshold, and predicting the target three-dimensional information after the point cloud is pre-processed if the centroid depth is greater than the threshold; specifically includes: The random consistency algorithm is used to calculate the center point of each target’s point cloud as the center of mass of the target (x i ,y i ,z i ); Set a distance threshold z t , when the depth information z of the target center of mass in the viewing cone i When the value is less than the threshold, the point cloud in the viewing cone is input into the point cloud detection network to obtain the target category, center point position, length, width, height and yaw angle; When the depth information z of the target center of mass in the viewing cone i When the threshold is greater than the target point cloud, a right-handed coordinate system is established with the center of mass of the target point cloud as the origin, the optical center direction of the camera coordinate system as the z-axis, and the right as the x-axis. A three-dimensional frame is established with the spatial range of the target point cloud. The point cloud center of mass, the three-dimensional frame and the target category c obtained by segmentation are combined. i Input into the neural network to regress the center of mass and 3D box of the target; The random sampling consistency algorithm is used to calculate the center point of each target's point cloud as the center of mass of the target (x i ,y i ,z i ); The random sampling consensus algorithm is an iterative method for estimating the parameters of a mathematical model from a set of observation data containing outliers. A point (x, y, z) in a point cloud in a cone is randomly selected and its Euclidean distance to all points in the cone is calculated. N is the total number of point clouds in the viewing cone, j is a point cloud in the viewing cone, iterate g times, and take the point (x i ,y i ,z i ) as the center of mass of the target.

2. The three-dimensional target detection method based on the fusion of multi-focal length camera and laser radar according to claim 1, characterized in that: The vehicle perceives the surrounding environment and obtains the size, position and posture of the vehicles around it, specifically the relative position of the target vehicle, the length, width, height and yaw angle of the vehicle.

3. The three-dimensional target detection method based on the fusion of multi-focal length camera and laser radar according to claim 1, characterized in that: The long-focus and short-focus cameras are used to capture images in front of the vehicle and perceive the environment in front of the vehicle, and the laser radar perceives the environment around the vehicle, specifically including: The multi-focal length camera consists of two cameras with different focal lengths. It is used to observe the front of the vehicle. The multi-focal length camera is placed horizontally, with the short-focus camera on the left and the long-focus camera on the right. The two cameras are close to each other, have a common field of view, and collect images simultaneously. The lidar is placed on the roof to sense the environment around the vehicle and select the image data collected by the multi-focal camera at the timestamp when the lidar scans the area directly in front of the vehicle. The data collection interval between the two sensors is less than 0.01 seconds.

4. The three-dimensional target detection method based on the fusion of multi-focal length camera and laser radar according to claim 3, characterized in that: The joint calibration of the multi-focal length camera and the lidar uses a spatiotemporal synchronization algorithm to synchronize the image data and the point cloud data collected by the lidar in time and space, specifically including: Establish a world coordinate system, calibrate the long and short focus cameras, obtain their respective intrinsic and extrinsic parameters, and obtain the rotation and translation matrices of the world coordinate system and the short focus camera as R1 and T1, and the rotation and translation matrices of the long focus camera as R2 and T2; the non-homogeneous linear coordinates of a point P in the world coordinate system, the long focus camera coordinate system, and the short focus camera coordinate system are but: Combining the two equations, we can get R 12 、T 12 The rotation and translation matrix between the long and short focus cameras, the rotation and translation matrix between the short focus camera and the lidar R3, T3, the rotation and translation matrix between the long focus camera and the lidar R 23 、T 23 .

5. The three-dimensional target detection method based on the fusion of multi-focal length camera and laser radar according to claim 4, characterized in that: The image data and the point cloud data collected by the lidar are synchronized in time and space by using a time and space synchronization algorithm, specifically including: Select the data with the closest timestamp among the data collected by the lidar and the multi-focal length camera to obtain time-synchronized point cloud data and multi-focal length camera image data; Based on the rotation and translation matrices of the lidar and long- and short-focus cameras and the intrinsic parameter matrices of the long- and short-focus cameras, combined with the time-synchronized point cloud data and multi-focal length camera data, the pixel coordinates of any point in the point cloud in the image can be obtained; spatial synchronization is achieved, and the spatiotemporal synchronization of image data and point cloud data is completed.

6. The three-dimensional target detection method based on the fusion of multi-focal length camera and laser radar according to claim 1, characterized in that: The method of using a multi-scale image pyramid to achieve matching of feature points of long- and short-focus images and establishing a connection between objects in the long- and short-focus segmented images through feature matching specifically includes: According to the focal length ratio of the long-focus and short-focus cameras, multi-scale feature pyramids of the long-focus and short-focus images at the same moment are established to extract the feature points of the long-focus and short-focus images; The image is scaled to obtain a multi-scale feature pyramid of the image, so that the target in the long-focus image has the same scale as the target in the short-focus image. The specific algorithm for matching feature points in the long- and short-focus multi-scale feature pyramid is as follows; According to the focal length ratio s of the long-focus and short-focus cameras and the projection matrix of the long-focus and short-focus cameras, the feature points of the nth layer of the long-focus multi-scale image pyramid are matched with the feature points of the 0th layer of the short-focus multi-scale image pyramid. Each matching of the long-focus and short-focus images is performed every n-1 layers. The feature points of the long and short focus image matching are retained, and the feature points are mapped back to the long and short focus segmentation image according to the scale scaling factor. For the case where both the long and short focus image segmentation masks have feature points, the connection between the long and short focus segmentation image objects is established.

7. The three-dimensional target detection method based on the fusion of multi-focal length camera and laser radar according to claim 6, characterized in that: The data association and fusion of the results of the three-dimensional target detection include the following steps: For targets that are connected by the long- and short-focus segmented images, the target's category, center position, length, width, height, and yaw angle are queried based on the association of the viewing cone. Because distant targets appear in a larger pixel range in the long-focus image, edge detection is more accurate. The target's category, center position, length, width, height, and yaw angle detected in the long-focus image are used. The long-focus and short-focus targets are segmented and the unconnected targets are treated as new targets. The three-dimensional information of the fused targets and the new targets are mapped to the short-focus image and point cloud image for easy observation.

Images