Multi-modal three-dimensional object detection method and system combining with a view cone and geometric constraints

Abstract

The application provides a multi-modal three-dimensional target detection method and system combining a frustum with geometric constraints, comprising: generating a two-dimensional perception result on an image to obtain a two-dimensional bounding box of a target to be detected; projecting a laser radar point cloud to an image plane to extract a point cloud projected into the two-dimensional bounding box; performing ground detection on the point cloud in the frustum, and combining the position of the ground with the lower boundary of the frustum to preliminarily filter the point cloud in the frustum; clustering the filtered point cloud to obtain each candidate point cloud cluster, merging the point cloud clusters, setting a scoring standard for the point cloud clusters to score each point cloud cluster, and selecting the point cloud cluster with the highest score as the result of a first stage; voxelizing the result point cloud of the first stage, performing template matching convolution calculation with a refined template kernel to obtain a matching response map, selecting a peak value position in the response map, inversely deducing a corresponding three-dimensional coordinate position and orientation according to a coordinate relationship, and obtaining a detection result of a three-dimensional target; and correcting the height and lateral direction in the detection result of the three-dimensional target by using the consistency of the three-dimensional projection result and the two-dimensional detection result to obtain a final three-dimensional target detection result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional target detection, in particular to a multi-modal three-dimensional target detection method and system combining a frustum with geometric constraints, and relates to a frustum and geometric rules, which can be applied to a perception module of an autonomous driving system. BACKGROUND

[0002] Autonomous driving technology is a hot research direction at present, which can be applied to unmanned transportation, unmanned sanitation, assisted driving and other scenarios, and can effectively liberate labor. The perception module is an important part of autonomous driving technology, which can help the autonomous driving system to obtain the current environmental information to help the vehicle to avoid obstacles and plan the path. Three-dimensional target detection is an important task in perception, which is used to obtain the position, size and orientation of obstacles in the scene, therefore, how to fuse the laser radar point cloud data and image data captured by the sensor, and design an efficient, accurate, robust and easy-to-deploy three-dimensional target detector is a big challenge.

[0003] After searching the existing technical literature, it is found that the current effective three-dimensional target detection model is based on deep learning method, which sends the laser radar point cloud and image data into the already constructed deep network model, and realizes end-to-end training according to the labeled three-dimensional label value. This method can get high effect on the data set, but since most of the models involve sparse convolution (Sparse Convolution) operators, which cannot be well deployed on the vehicle chip at present. A small part of methods such as PointPillars directly compresses the point cloud to the top view, and then uses a two-dimensional backbone network to regress the three-dimensional position of the target, but this method cannot well combine the information of the image. Some works such as F-PointNets, F-convnet, Frustum-PointPillars, etc. first use a two-dimensional target detector and geometric projection to construct a frustum, and then use a deep network module to detect the three-dimensional target in the frustum. This method can effectively combine the mature two-dimensional target detector, and has no complex module, and is easier to deploy; but the two stages (two-dimensional detection and three-dimensional detection) of this method need to use deep network training, which needs a large amount of data to fit, resulting in poor generalization ability of this method, and depends on the amount and quality of the training data. The present application makes changes on the basis of the frustum method, and combines geometric constraints in the three-dimensional detection step to realize the final detection. This method does not depend on a large amount of training data, and is easy to deploy, has good generalization and real-time performance.

[0004] Patent document CN110909623B (application number: 201911052349.9) discloses a three-dimensional object detection method and a three-dimensional object detector. The three-dimensional object detection method mainly includes the following steps: semantic segmentation of image data in a three-dimensional object detection dataset to obtain semantic predictions; projecting the semantic predictions into a point cloud space and selecting points of specific categories to form view frustums; constructing a three-dimensional object detection network and using the view frustums as input to the three-dimensional object detection network; enhancing the sensitivity of the three-dimensional object detection network to targets through a loss function; and optimizing the three-dimensional object detection network. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the purpose of this invention is to provide a multimodal three-dimensional target detection method and system that combines view frustum and geometric constraints.

[0006] A multimodal three-dimensional target detection method combining view frustum and geometric constraints provided by the present invention includes:

[0007] Step S1: Use a two-dimensional target detector to generate a two-dimensional perception result on the camera image to obtain the two-dimensional bounding box of the target to be detected in the image; according to the extrinsic parameters between the sensors and the intrinsic parameters of the camera, project the LiDAR point cloud onto the image plane, extract the point cloud projected into the two-dimensional bounding box, and obtain the corresponding in-frust point cloud.

[0008] Step S2: Perform ground detection on the point cloud within the view frustum, and perform preliminary filtering on the point cloud within the view frustum based on the position of the intersection of the ground and the lower boundary of the view frustum. Use DBSCAN clustering on the filtered point cloud to obtain each candidate point cloud cluster, and set a distance threshold to merge the point cloud clusters. Set a scoring standard for the point cloud clusters, score each point cloud cluster, and finally select the point cloud cluster with the highest score as the result of the first stage.

[0009] Step S3: Voxelize the point cloud results from the first stage and perform template matching convolution calculation with the refined template kernel to obtain the matching response map. Select the peak position in the response map and deduce its corresponding three-dimensional coordinate position and orientation based on the coordinate relationship to obtain the detection result of the three-dimensional target.

[0010] Step S4: Use the consistency between the 3D projection results and the 2D detection results to correct the height and lateral direction in the 3D target detection results to obtain the final 3D target detection results.

[0011] Specifically, step S1 employs the following:

[0012] Step S1.1: Use a two-dimensional object detector to detect the camera image and obtain the two-dimensional detection result (u1, v1, u2, v2, cls, conf) of the target to be detected, where (u1, v1, u2, v2) represent the left boundary coordinates u1, the upper boundary coordinates v1, the right boundary coordinates u2, and the lower boundary coordinates v2 of the two-dimensional detection box in the image, cls represents the category attribute of the target, and conf represents the confidence score of the detection box;

[0013] Step S1.2: Using the extrinsic parameters between the camera and the LiDAR, the LiDAR point cloud is transformed into the camera coordinate system. Then, based on the camera's intrinsic parameters, the point cloud data in the camera coordinate system is projected onto the image plane. The point set with positive depth and whose projection is located within the two-dimensional detection box is selected and mapped to the original LiDAR point cloud to form an in-view frustum point cloud.

[0014] Specifically, step S2 employs the following:

[0015] Step S2.1: Sort all point clouds within the view frustum by height value from smallest to largest, and select the top N. LPR The average height of each point is taken as the initial height H of the ground. init Set a distance threshold Δh1, and ensure that all heights satisfy z < H. init The point +Δh1 is considered a ground point. The least squares method is used to fit the plane equation of the ground based on the current ground point. The intersection position of the lower boundary of the view frustum and the view frustum is calculated based on the ground equation. The coordinates (x, y, y) of the intersection position are then used to determine the intersection position. inter y inter and distance threshold d inter Screening out points in the cloud that are far from the boundary with the view frustum, satisfying the following conditions: The points are selected, while points belonging to the ground are filtered out;

[0016] Step S2.2: Use DBSCAN clustering to cluster the point cloud, obtaining several candidate point cloud sets set1, set2, ..., set... n For the i-th point cloud set set i With the j-th point cloud set set j Calculate the farthest distance between two point sets. If the current farthest distance is less than the distance merging threshold d, then... merge Then, merge the two current point sets; after merging all candidate point sets, a new point cloud set set1, set2, ..., set is obtained. m ;

[0017] Step S2.3: Each candidate point set is scored according to the preset scoring rules, and the point set with the highest score is selected as the first-stage candidate point;

[0018] The preset scoring rules include: the more points included, the higher the score; the more ground and background points included, the lower the score; or the closer the distance to the intersection of the ground and the lower boundary of the view frustum, the higher the score.

[0019] Preferably, step S3 employs the following methods:

[0020] Step S3.1: Obtain the ground truth point cloud for each category from the training set of the autonomous driving dataset based on the 3D label values, and calculate the average 3D size of each category (l m w m h m The center coordinates of the ground truth point cloud for each category are normalized to (0, 0, 0), the 3D size is normalized to the average value, and the orientation angle is normalized to 0°. The point sets are then stitched together to obtain a preliminary template point set. Each template point set is then flipped along the x-axis to enhance symmetry for each category. Finally, a VoxelGrid filter is used to downsample the templates, resulting in a filtered template. K is set... θ The number of angle intervals is used to divide the interval from -180° to 180° into K equal intervals. θ Each angle value is used to rotate the template, resulting in K with orientation information. θ A template, with a voxel size (x grid ,y grid ,z grid The current template is voxelized to obtain the template kernel for each category, with a dimension of [K]. θ L m W m H m ];

[0021] Step S3.2: Using voxel size (x grid y grid , z grid Voxelization is performed on the candidate first-stage point cloud to obtain the size [L]. p W p H p The three-dimensional tensor of ];

[0022] Step S3.3: Apply the template kernel to the input voxel tensor, set the convolution stride to 1, and the edge padding method to zero-padding, to obtain a tensor of size [K]. θ L p W p H p The matching response map shows that positions with high response values ​​represent positions where the distribution of the point cloud to be matched at the current position is highly similar to the distribution of the template.

[0023] Step S3.4: Obtain the peak position in the response graph and get the angle value θ corresponding to the current peak position.dt and three-dimensional coordinate position (x dt y dt , z dt The output is the detection result of the three-dimensional target (x). dt y dt , z dt , l m w m h m θ dt ).

[0024] Preferably, step S4 employs the following methods:

[0025] Step S4.1: Detection results of 3D targets (x dt y dt , z dt , l m w m h m θ dt The detection result in the camera coordinate system (x) is obtained by transforming the camera extrinsic parameters to the camera coordinate system. cam ,y cam ,z cam ,l cam ,w cam ,h cam ,θ cam Based on the definition of the 3D detection box, the coordinates of the 8 corner points of the 3D detection box in the camera coordinate system are calculated:

[0026]

[0027]

[0028]

[0029]

[0030]

[0031]

[0032]

[0033]

[0034] Then, based on the camera's intrinsic focal length and offset parameter f u f v c u c v The projected coordinates of the eight corner points on the image plane are obtained as follows:

[0035]

[0036]

[0037]

[0038]

[0039]

[0040]

[0041]

[0042]

[0043] These coordinates are labeled as (m i n i The value of i ranges from 1 to 8;

[0044] Step S4.2: Input the projected pixel coordinates (m) i n i Given i = 1, ..., 8 and the two-dimensional detection results (u1, v1, u2, v2), construct an error function in the height direction:

[0045]

[0046] Substituting the formula for the eight corner points into the above formula, the variable that needs optimization is y. cam The corrected height value was obtained by optimizing it using gradient descent.

[0047] Step S4.3: Input the projected pixel coordinates (m) i n i Given i = 1, ..., 8 and the two-dimensional detection results (u1, v1, u2, v2), construct the error function in the lateral direction:

[0048]

[0049] Substituting the formula for the eight corner points into the above formula, the variable that needs optimization is x. cam The corrected lateral value was obtained by optimizing it using gradient descent. It then outputs the final 3D target detection results.

[0050] A multimodal three-dimensional target detection system combining view frustum and geometric constraints, provided by the present invention, includes:

[0051] Module M1: Uses a two-dimensional target detector to generate two-dimensional perception results on the camera image, and obtains the two-dimensional bounding box of the target to be detected in the image; based on the extrinsic parameters between the sensors and the intrinsic parameters of the camera, projects the LiDAR point cloud onto the image plane, extracts the point cloud projected into the two-dimensional bounding box, and obtains the corresponding in-frust point cloud;

[0052] Module M2: Performs ground detection on the point cloud within the view frustum, and performs preliminary filtering of the point cloud within the view frustum based on the location of the intersection between the ground and the lower boundary of the view frustum. The filtered point cloud is clustered using DBSCAN to obtain each candidate point cloud cluster, and the point cloud clusters are merged by setting a distance threshold. The scoring criteria for the point cloud clusters are set, and each point cloud cluster is scored. Finally, the point cloud cluster with the highest score is selected as the result of the first stage.

[0053] Module M3: Voxelizes the point cloud results from the first stage and performs convolutional calculations with the refined template kernel to obtain a matching response map. The peak positions in the response map are selected, and their corresponding 3D coordinate positions and orientations are deduced based on the coordinate relationships to obtain the detection results of the 3D target.

[0054] Module M4: Utilizes the consistency between the 3D projection results and the 2D detection results to correct the height and lateral dimensions in the 3D detection results, thus obtaining the final 3D detection results.

[0055] Preferably, module M1 adopts:

[0056] Module M1.1: Uses a two-dimensional object detector to detect the camera image and obtains the two-dimensional detection result (u1, v1, u2, v2, cls, conf) of the target to be detected, where (u1, v1, u2, v2) represent the left boundary coordinates u1, upper boundary coordinates v1, right boundary coordinates u2, and lower boundary coordinates v2 of the two-dimensional detection box in the image, cls represents the target's category attribute, and conf represents the confidence score of the detection box;

[0057] Module M1.2: Using the extrinsic parameters between the camera and the LiDAR, the LiDAR point cloud is transformed into the camera coordinate system. Then, based on the camera's intrinsic parameters, the point cloud data in the camera coordinate system is projected onto the image plane. The point set with positive depth and whose projection is located within the two-dimensional detection box is selected and mapped to the original LiDAR point cloud to form an in-frust point cloud.

[0058] Preferably, the module M2 adopts:

[0059] Module M2.1: Sort all point clouds within the view frustum by height values ​​from smallest to largest, and select the top N. LPR The average height of each point is taken as the initial height H of the ground. init Set a distance threshold Δh1, and ensure that all heights satisfy z < H. initThe point +Δh1 is considered a ground point. The least squares method is used to fit the plane equation of the ground based on the current ground point. The intersection position of the lower boundary of the view frustum and the view frustum is calculated based on the ground equation. The coordinates (x, y, y) of the intersection position are then used to determine the intersection position. inter y inter and distance threshold d inter Screening out points in the cloud that are far from the boundary with the view frustum, satisfying the following conditions: The points are selected, while points belonging to the ground are filtered out;

[0060] Module M2.2: Uses DBSCAN clustering to cluster point clouds, obtaining several candidate point cloud sets set1, set2, ..., set n For the i-th point cloud set set i With the j-th point cloud set set j Calculate the farthest distance between two point sets. If the current farthest distance is less than the distance merging threshold d, then... merge Then, merge the two current point sets; after merging all candidate point sets, a new point cloud set set1, set2, ..., set is obtained. m ;

[0061] Module M2.3: Each candidate point set is scored according to a preset scoring rule, and the point set with the highest score is selected as the first-stage candidate point;

[0062] The preset scoring rules include: the more points included, the higher the score; the more ground and background points included, the lower the score; or, the closer the distance to the intersection of the ground and the lower boundary of the view frustum, the higher the score.

[0063] Preferably, the module M3 adopts:

[0064] Module M3.1: Obtains ground truth point clouds for each category from the training set of the autonomous driving dataset based on the 3D labels, and calculates the average 3D size of each category (l m w m h m Then, the center coordinates of the ground truth point cloud for each category are normalized to (0, 0, 0), the 3D size is normalized to the average value, and the orientation angle is normalized to 0°. The point sets are then stitched together to obtain a preliminary template point set. Each template point set is then flipped along the x-axis to enhance symmetry for each category. Finally, a VoxelGrid filter is used to downsample the templates, resulting in a filtered template. K is then set... θ The number of angle intervals is used to divide the interval from -180° to 180° into K equal intervals. θ Each angle value is used to rotate the template, resulting in K with orientation information. θ A template, with a voxel size (x grid ygrid , z grid The current template is voxelized to obtain the template kernel for each category, with a dimension of [K]. θ L m W m H m ];

[0065] Module M3.2: in voxel size (x grid ,y grid ,z grid Voxelization is performed on the candidate first-stage point cloud to obtain the size [L]. p W p H p The three-dimensional tensor of ];

[0066] Module M3.3: Applies the template kernel to the input voxel tensor, sets the convolution stride to 1, and uses zero-padding to fill the edges, resulting in a tensor of size [K]. θ L p W p H p The matching response map shows that positions with high response values ​​represent positions where the distribution of the point cloud to be matched at the current position is highly similar to the distribution of the template.

[0067] Module M3.4: Obtain the peak position in the response graph and get the angle value θ corresponding to the current peak position. dt and three-dimensional coordinate position (x dt y dt , z dt The output is the detection result of the three-dimensional target (x). dt y dt , z dt , l m w m h m θ dt ).

[0068] Preferably, the module M4 adopts:

[0069] Module M4.1: Detection results of 3D targets (x dt y dt , z dt , l m w m h m θ dt The detection result in the camera coordinate system (x) is obtained by transforming the camera extrinsic parameters to the camera coordinate system. cam y cam , z cam , l cam w cam h cam θcam Based on the definition of the 3D detection box, the coordinates of the 8 corner points of the 3D detection box in the camera coordinate system are calculated:

[0070]

[0071]

[0072]

[0073]

[0074]

[0075]

[0076]

[0077]

[0078] Then, based on the camera's intrinsic focal length and offset parameter f u f v c u c v The projected coordinates of the eight corner points on the image plane are obtained as follows:

[0079]

[0080]

[0081]

[0082]

[0083]

[0084]

[0085]

[0086]

[0087] These coordinates are labeled as (m i n i The value of i ranges from 1 to 8;

[0088] Module M4.2: Input projected pixel coordinates (m) i n i Given i = 1, ..., 8 and the two-dimensional detection results (u1, v1, u2, v2), construct an error function in the height direction:

[0089]

[0090] Substituting the formula for the eight corner points into the above formula, the variable that needs optimization is y. cam The corrected height value was obtained by optimizing it using gradient descent.

[0091] Module M4.3: Input projected pixel coordinates (m) i n i Given i = 1, ..., 8 and the two-dimensional detection results (u1, v1, u2, v2), construct the error function in the lateral direction:

[0092]

[0093] Substituting the formula for the eight corner points into the above formula, the variable that needs optimization is x. cam The corrected lateral value was obtained by optimizing it using gradient descent. It then outputs the final 3D target detection results.

[0094] Compared with the prior art, the present invention has the following beneficial effects:

[0095] 1. This invention does not rely on massive amounts of data for training, which can reduce the cost of data collection, model training and model inference;

[0096] 2. This invention has good generalization ability, and geometric constraints provide a guarantee of generalization for target detection in general scenarios;

[0097] 3. This invention does not use complex deep networks and relies on traditional geometric methods, which has the advantages of simple algorithm and fast running speed.

[0098] 4. This invention can be easily deployed in autonomous driving systems and simultaneously possesses accuracy, real-time performance, and robustness. Attached Figure Description

[0099] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0100] Figure 1 This is a flowchart of a multimodal 3D target detection method that combines view frustum and geometric constraints. Detailed Implementation

[0101] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0102] This invention specifically relates to a 3D target detection method combining view frustum and geometric constraints, applicable to the perception module of autonomous driving systems, and belonging to the target detection problem in computer vision. The purpose of this invention is to efficiently fuse LiDAR point cloud data and camera image data captured by sensors, proposing a 3D target detection method based on view frustum and geometric constraints. This method can be easily deployed in autonomous driving systems and simultaneously possesses accuracy, real-time performance, and robustness. Furthermore, this invention has advantages such as simple algorithm, fast computation speed, and strong generalization ability, providing a new and reliable method for 3D target detection.

[0103] To achieve the above objectives, this invention first uses a two-dimensional target detector to obtain the detection results of targets in the image, and obtains a point cloud of the view frustum projected within the detection frame based on the extrinsic parameters between the LiDAR and the camera and the intrinsic parameters of the camera. Then, ground detection is performed on the point cloud within the view frustum, and preliminary filtering is performed based on the position of the intersection of the ground and the lower boundary of the view frustum. Next, DBSCAN clustering is used to obtain candidate point cloud clusters, and a distance threshold is set to merge the point cloud clusters. A scoring standard is set for each point cloud cluster, and each cluster is scored. Finally, the point cloud cluster with the highest score is selected as the first-stage result. Next, the point cloud of the first-stage result is voxelized and convolved with a refined template kernel (in the form of a convolution kernel) to obtain a matching response map. The peak position in the response map is selected, and its corresponding three-dimensional coordinate position and orientation are deduced based on the coordinate relationship. Finally, the height and lateral dimensions in the three-dimensional detection result are corrected using the consistency between the three-dimensional projection result and the two-dimensional detection result to obtain the final three-dimensional detection result.

[0104] Example 1

[0105] According to the present invention, a multimodal three-dimensional target detection method combining view frustum and geometric constraints is provided, such as... Figure 1 As shown, it includes:

[0106] Step S1: 2D Target Detection and View Frustum Generation. The inputs to this step are time-stamped aligned camera image data and LiDAR point cloud data. The extrinsic parameters between the sensors and the intrinsic parameters of the camera are known. The outputs are the 2D target detection results and the point cloud data falling within the view frustum. A 2D target detector generates a 2D perception result on the camera image, obtaining the 2D bounding box of the target to be detected (car, pedestrian, rider, etc.) in the image. Based on the extrinsic parameters between the sensors and the intrinsic parameters of the camera, the LiDAR point cloud is projected onto the image plane. The point cloud data projected within the 2D detection box is extracted. Since the 2D detection box corresponds to a cone-shaped region in 3D space (called the view frustum), a cone-shaped point cloud data can be obtained.

[0107] Step S2: Clustering and Target Point Selection. The input for this step is the point cloud data within the view frustum, and the output is the points contained within the target (hereinafter referred to as the first-stage result). First, a ground detection algorithm is used to fit the ground within the view frustum, which yields the intersection of the fitted ground and the lower boundary of the view frustum, thus filtering out some point clouds that do not belong to the target, achieving point cloud filtering within the view frustum. Second, DBSCAN clustering is used to cluster the filtered point cloud, resulting in several clusters. Different distance thresholds are set for each type of target, and cluster integration is performed, merging the point sets of several classes that are spatially close. Finally, target point selection rules are set, a score is calculated for each cluster, and the cluster with the highest score is selected as the output result of this stage.

[0108] Step S3: Template Matching and Detection Result Generation. The input to this step is the first-stage result, and the output is the 3D detection result of the target. First, based on the ground truth, the point cloud sets of each target category are obtained, templates for each category are generated, and the templates are voxelized to obtain the probability of a point falling into each voxel in the template, represented by a 3D convolution kernel. Second, the first-stage result is voxelized again to obtain the number of points in each voxel. Third, the template is convolved on the voxelized first-stage result to obtain the matching response map. Finally, the peak position of the response map is selected, and based on the coordinate correspondence, the 3D coordinates and orientation of the corresponding position are deduced. The average 3D size of each category's template is selected as the average value of the detection result, and the 3D detection result is output.

[0109] Step S4: Post-processing correction based on reprojection. The input to this step is the 3D detection result and the 2D template detection result, and the output is the corrected 3D detection result. The eight corner points of the 3D detection result are projected onto the image plane, resulting in eight points on the image. The minimum bounding rectangle is taken for these points. Based on the consistency between this minimum bounding rectangle and the 2D detection result, an error function is constructed regarding the target's 3D height and horizontal direction. The correction of the target's 3D detection result is achieved by minimizing this error function, and the final corrected result is output.

[0110] Specifically, step S1 employs the following:

[0111] Step S1.1: Use a mature 2D object detector to detect the input camera image and obtain the 2D detection result of the target to be detected. Each result contains 6 data (u1, v1, u2, v2, cls, conf), where (u1, v1, u2, v2) represent the left boundary coordinates u1, the upper boundary coordinates v1, the right boundary coordinates u2, and the lower boundary coordinates v2 of the 2D detection box in the image, cls represents the category attribute of the target (e.g., 0 represents pedestrian, 1 represents rider, etc.), and conf represents the confidence of the detection box (the value ranges from 0 to 1).

[0112] Step S1.2: Using the extrinsic parameters between the camera and the LiDAR, the LiDAR point cloud is transformed into the camera coordinate system. Then, based on the camera's intrinsic parameters, the point cloud data in the camera coordinate system is projected onto the image plane. The point set with positive depth and whose projection is located within the two-dimensional detection box is selected and mapped to the original LiDAR point cloud to form an in-view frustum point cloud.

[0113] Specifically, step S2 employs the following:

[0114] Step S2.1: Perform ground detection on the LiDAR point cloud to obtain points located on the ground, and then filter the point cloud within the view frustum. First, sort all points within the view frustum by height value from smallest to largest, and select the top N. LPR The average height of each point is taken as the initial height H of the ground. init Set a distance threshold Δh1, and ensure that all heights satisfy z < H. init Points with +Δh1 are considered ground points. The least squares method is used to fit the plane equation of the ground based on these points. According to the ground equation F... g (x, y, z) = 0 and the error threshold Δh2, all satisfying F g Points where (x, y, z) < Δh2 are considered ground points. A new ground equation is fitted using the least squares method based on these points. This process is repeated until convergence. The intersection of the lower boundary of the view frustum and the view frustum can be calculated from the ground equation. The coordinates (x, y, z) of the intersection point are then used to determine the ground equation. inter y inter and distance threshold d inter Screening out points in the cloud that are far from the boundary, i.e. The points are selected, while points belonging to the ground are filtered out.

[0115] Step S2.2: Use DBSCAN to cluster the point cloud to obtain several candidate point cloud sets set1, set2, ..., set n For the i-th point cloud set set i With the j-th point cloud set set jCalculate the farthest distance between two point sets. If this farthest distance is less than the distance merging threshold d, then... merge Then, these two point sets are merged. After merging all candidate point sets, a new point cloud set set1, set2, ..., set is obtained. m .

[0116] Step S2.3: Set rules, score each candidate point set, and select the point set with the highest score as the first-stage candidate point. The scoring rules include: ① the number of points included (the more, the higher the score); ② the number of ground background points included (the more, the lower the score); ③ the distance from the intersection of the ground and the lower boundary of the view frustum (the closer, the higher the score).

[0117] Specifically, step S3 employs the following:

[0118] Step S3.1: Template Extraction and Filtering. From the training set of the autonomous driving dataset, obtain the ground truth point clouds for each category based on the 3D label values, and calculate the average 3D size of each category (l). m w m h m Then, the center coordinates of the ground truth point cloud for each category are normalized to (0, 0, 0), the 3D size is normalized to the average value, and the orientation angle is normalized to 0°. The point sets are then stitched together to obtain a preliminary template point set. Next, each template point set is flipped along the x-axis to enhance symmetry, ensuring that the templates for each category satisfy the required symmetry. Finally, V... o x e lGrid filt e r downsamples the template to obtain the filtered template. Let K be... θ The number of angle intervals is used to divide the interval from -180° to 180° into K equal intervals. θ Each angle value is used to rotate the template, resulting in K with orientation information. θ A template, with a voxel size (x grid y grid , z grid These templates are voxelized to obtain the template kernel for each category, with dimensions [Kθ, L]. m W m H m ].

[0119] Step S3.2: Using voxel size (x grid y grid , z grid Voxelization is performed on the candidate first-stage point cloud to obtain the size [L]. p W p H p The three-dimensional tensor of ];

[0120] Step S3.3: Apply the template kernel to the input voxel tensor, set the convolution stride to 1, and the edge padding method to zero-padding, to obtain a tensor of size [K]. θ L p W p H p The matching response map shows that positions with high response values ​​represent positions where the distribution of the point cloud to be matched at the current position is highly similar to the distribution of the template.

[0121] Step S3.4: Obtain the peak position in the response graph and get the angle value θ corresponding to the current peak position. dt and three-dimensional coordinate position (x dt y dt , z dt The output is the detection result of the three-dimensional target (x). dt y dt , z dt , l m w m h m θ dt ).

[0122] Specifically, step S4 employs the following:

[0123] Step S4.1: Detection results of 3D targets (x dt y dt , z dt , l m w m h m θ dt The detection result in the camera coordinate system (x) is obtained by transforming the camera extrinsic parameters to the camera coordinate system. cam y cam , z cam , l cam w cam h cam θ cam Based on the definition of the 3D detection box, the coordinates of the 8 corner points of the 3D detection box in the camera coordinate system are calculated:

[0124]

[0125]

[0126]

[0127]

[0128]

[0129]

[0130]

[0131]

[0132] Then, based on the camera's intrinsic focal length and offset parameter f u f v c u c v The projected coordinates of the eight corner points on the image plane are obtained as follows:

[0133]

[0134]

[0135]

[0136]

[0137]

[0138]

[0139]

[0140]

[0141] These coordinates are labeled as (m i n i The value of i ranges from 1 to 8;

[0142] Step S4.2: Input the projected pixel coordinates (m) i n i Given i = 1, ..., 8 and the two-dimensional detection results (u1, v1, u2, v2), construct an error function in the height direction:

[0143]

[0144] Substituting the formula for the eight corner points into the above formula, the variable that needs optimization is y. cam The corrected height value was obtained by optimizing it using gradient descent.

[0145] Step S4.3: Input the projected pixel coordinates (m) i n i Given i = 1, ..., 8 and the two-dimensional detection results (u1, v1, u2, v2), construct the error function in the lateral direction:

[0146]

[0147] Substituting the formula for the eight corner points into the above formula, the variable that needs optimization is x. cam The corrected lateral value was obtained by optimizing it using gradient descent. It then outputs the final 3D target detection results.

[0148] A multimodal three-dimensional target detection system combining view frustum and geometric constraints, provided by the present invention, includes:

[0149] Module M1: 2D Object Detection and View Frustum Generation. The inputs to this step are time-stamped aligned camera image data and LiDAR point cloud data, with known extrinsic parameters between the sensors and intrinsic parameters of the camera. The outputs are the 2D object detection results and the point cloud data falling within the view frustum. A 2D object detector generates a 2D perception result on the camera image, obtaining the 2D bounding box of the target (car, pedestrian, rider, etc.) in the image. Based on the extrinsic parameters between the sensors and the intrinsic parameters of the camera, the LiDAR point cloud is projected onto the image plane. The point cloud data projected within the 2D detection box is extracted. Since the 2D detection box corresponds to a cone-shaped region in 3D space (called the view frustum), a cone-shaped point cloud data is obtained.

[0150] Module M2: Clustering and Target Point Selection. The input to this step is the point cloud data within the view frustum, and the output is the points contained within the target (hereinafter referred to as the first-stage result). First, a ground detection algorithm is used to fit the ground within the view frustum, obtaining the boundary position between the fitted ground and the lower boundary of the view frustum, thus filtering out some point clouds that do not belong to the target, achieving point cloud filtering within the view frustum. Second, DBSCAN clustering is used to cluster the filtered point cloud, obtaining several clusters. Different distance thresholds are set for each category of target, and cluster integration is performed, merging the point sets of several classes that are spatially close. Finally, target point selection rules are set, a score is calculated for each cluster, and the cluster with the highest score is selected as the output result of this stage.

[0151] Module M3: Template Matching and Detection Result Generation. This step takes the first-stage result as input and outputs the 3D detection result of the target. First, based on the ground truth, point cloud sets for each target category are obtained, templates for each category are generated, and the templates are voxelized to obtain the probability of a point falling on each voxel in the template, represented as a 3D convolution kernel. Second, the first-stage result is voxelized again to obtain the number of points in each voxel. Third, the template is convolved on the voxelized first-stage result to obtain the matching response map. Finally, the peak position of the response map is selected, and based on the coordinate correspondence, the 3D coordinates and orientation of the corresponding position are deduced. The average 3D size of each category's template is selected as the average value of the detection result, and the 3D detection result is output.

[0152] Module M4: Post-processing correction based on reprojection. This step takes the 3D detection result and the 2D template detection result as input and outputs the corrected 3D detection result. The eight corner points of the 3D detection result are projected onto the image plane, resulting in eight points on the image. The minimum bounding rectangle is taken for these points. Based on the consistency between this minimum bounding rectangle and the 2D detection result, an error function is constructed regarding the target's 3D height and horizontal direction. Minimizing this error function corrects the target's 3D detection result, and the corrected result is finally output.

[0153] Specifically, module M1 adopts:

[0154] Module M1.1: Uses a mature 2D object detector to detect the input camera image and obtains the 2D detection result of the target to be detected. Each result contains 6 data points (u1, v1, u2, v2, cls, conf), where (u1, v1, u2, v2) represent the left boundary coordinates u1, the upper boundary coordinates v1, the right boundary coordinates u2, and the lower boundary coordinates v2 of the 2D detection box in the image, cls represents the category attribute of the target (e.g., 0 represents pedestrian, 1 represents rider, etc.), and conf represents the confidence of the detection box (with a value between 0 and 1).

[0155] Module M1.2: Using the extrinsic parameters between the camera and the LiDAR, the LiDAR point cloud is transformed into the camera coordinate system. Then, based on the camera's intrinsic parameters, the point cloud data in the camera coordinate system is projected onto the image plane. The point set with positive depth and whose projection is located within the two-dimensional detection box is selected and mapped to the original LiDAR point cloud to form an in-frust point cloud.

[0156] Specifically, module M2 adopts:

[0157] Module M2.1: Performs ground detection on the lidar point cloud to obtain points located on the ground, and filters the point cloud within the view frustum. First, all points within the view frustum are sorted by height value from smallest to largest, and the top N are selected. LPR The average height of each point is taken as the initial height H of the ground. init Set a distance threshold Δh1, and ensure that all heights satisfy z < H. init Points with +Δh1 are considered ground points. The least squares method is used to fit the plane equation of the ground based on these points. According to the ground equation F... g (x, y, z) = 0 and the error threshold Δh2, all satisfying F g (x, y Points where z < Δh2 are considered ground points. A new ground equation is fitted using the least squares method based on these points. This process is repeated until convergence. The intersection of the lower boundary of the view frustum and the view frustum can be calculated from the ground equation. The coordinates (x, z) of the intersection point are then used to determine the ground equation. inter yinter and distance threshold d inter Screening out points in the cloud that are far from the boundary, i.e. The points are selected, while points belonging to the ground are filtered out.

[0158] Module M2.2: Uses DBSCAN to cluster point clouds, obtaining several candidate point cloud sets set1, set2, ..., set n For the i-th point cloud set set i With the j-th point cloud set set j Calculate the farthest distance between two point sets. If this farthest distance is less than the distance merging threshold dmer, then... g If e, then merge these two point sets. After merging all candidate point sets, a new point cloud set set1, set2, ..., set is obtained. m .

[0159] Module M2.3: Defines rules for scoring each candidate point set, selecting the set with the highest score as the first-stage candidate point. Scoring rules include: ① the number of points included (more points, higher score); ② the number of ground / background points included (more ground / background points, lower score); ③ the distance from the intersection of the ground and the lower boundary of the view frustum (closer distance, higher score).

[0160] Specifically, module M3 adopts:

[0161] Module M3.1: Template Extraction and Filtering. From the training set of the autonomous driving dataset, ground truth point clouds for each category are obtained based on 3D label values, and the average 3D size of each category is calculated (l). m w m h m Then, the center coordinates of the ground truth point cloud for each category are normalized to (0, 0, 0), the 3D size is normalized to the average value, and the orientation angle is normalized to 0°. The point sets are then stitched together to obtain a preliminary template point set. Next, each template point set is flipped along the x-axis to enhance symmetry between categories. Finally, a VoxelGrid filter is used to downsample the templates, resulting in a filtered template. K is set... θ The number of angle intervals is used to divide the interval from -180° to 180° into K equal intervals. θ Each angle value is used to rotate the template, resulting in K with orientation information. θ A template, with a voxel size (x grid y grid , z grid These templates are voxelized to obtain the template kernel for each category, with a dimension of [K]. θ L m W mH m ].

[0162] Module M3.2: in voxel size (x grid y grid , z grid Voxelization is performed on the candidate first-stage point cloud to obtain the size [L]. p W p H p The three-dimensional tensor of ];

[0163] Module M3.3: Applies the template kernel to the input voxel tensor, sets the convolution stride to 1, and uses zero-padding to fill the edges, resulting in a tensor of size [K]. θ L p W p H p The matching response map shows that positions with high response values ​​represent positions where the distribution of the point cloud to be matched at the current position is highly similar to the distribution of the template.

[0164] Module M3.4: Obtain the peak position in the response graph and get the angle value θ corresponding to the current peak position. dt and three-dimensional coordinate position (x dt y at , z dt The output is the detection result of the three-dimensional target (x). dt y at , z dt , l m w m h m θ dt ).

[0165] Specifically, module M4 adopts:

[0166] Module M4.1: Detection results of 3D targets (x dt y dt , z dt , l m w m h m θ at The detection result in the camera coordinate system (x) is obtained by transforming the camera extrinsic parameters to the camera coordinate system. cam， y cam , z cam , l cam w cam h cam θ cam Based on the definition of the 3D detection box, the coordinates of the 8 corner points of the 3D detection box in the camera coordinate system are calculated:

[0167]

[0168]

[0169]

[0170]

[0171]

[0172]

[0173]

[0174]

[0175] Then, based on the camera's intrinsic focal length and offset parameters fu, fv, c u ,cv, obtain the projected coordinates of the 8 corner points on the image plane, which are as follows:

[0176]

[0177]

[0178]

[0179]

[0180]

[0181]

[0182]

[0183]

[0184] These coordinates are labeled as (m i n i The value of i ranges from 1 to 8;

[0185] Module M4.2: Input projected pixel coordinates (m) i n i Given i = 1, ..., 8 and the two-dimensional detection results (u1, v1, u2, v2), construct an error function in the height direction:

[0186]

[0187] Substituting the formula for the eight corner points into the above formula, the variable that needs optimization is y. cam The corrected height value was obtained by optimizing it using gradient descent.

[0188] Module M4.3: Input projected pixel coordinates (m) i n i Given i = 1, ..., 8 and the two-dimensional detection results (u1, v1, u2, v2), construct the error function in the lateral direction:

[0189]

[0190] Substituting the formula for the eight corner points into the above formula, the variable that needs optimization is x. cam The corrected lateral value was obtained by optimizing it using gradient descent. It then outputs the final 3D target detection results.

[0191] Example 2

[0192] Example 2 is a preferred example of Example 1.

[0193] In one embodiment of the present invention, the process of the method is as follows: Figure 1 As shown, the specific implementation steps of this embodiment are as follows:

[0194] Step 1: First, use the YOLOv5 2D object detector, which has been trained and converged, to perform 2D object detection on the input image, obtaining 2D detection results. Each detection result includes the coordinates of the detection box, the object category, and the detection box confidence score, totaling six values: (u1, v1, u2, v2, cls, conf). Then, based on the extrinsic parameters between the camera and the LiDAR, the LiDAR point cloud is set... Transform to the camera coordinate system to obtain the point cloud set in that coordinate system. Then, based on the camera's intrinsic parameters, the point cloud set is transformed to pixel coordinates to obtain... Extract the points whose pixel coordinates are located within the i-th detection box to obtain a cone-shaped point cloud set.

[0195] Step 2: Use a ground detection algorithm to obtain ground points and calculate the intersection of the ground and the lower boundary of the view frustum. Set a distance threshold to filter the point cloud within the view frustum. Use the DBSCAN algorithm to cluster the filtered point cloud to obtain several clusters. Set scoring rules for each category and obtain a score for each category. Finally, select the cluster with the highest score as the first-stage candidate points. The scoring rules include: ① the number of points contained (the more, the higher the score); ② the number of ground background points contained (the more, the lower the score); ③ the distance from the intersection of the ground and the lower boundary of the view frustum (the closer, the higher the score).

[0196] Step 3: Template Matching and Result Generation. First, ground truth point clouds for each category are obtained from the training set of the autonomous driving dataset based on 3D label values. After normalizing the point set, it is stitched together and flipped along the x-axis for enhancement, resulting in a preliminary template point set. Then, a VoxelGrid filter is used to downsample the template, obtaining the filtered template. 72 angle intervals are set, dividing the -180° to 180° interval into 72 angle values. The template is then rotated based on these angle values, resulting in 72 templates with orientation information. These templates are voxelized at voxel sizes (0.1, 0.1, 0.1) to obtain the template kernel module_conv for each category. Finally, the first-stage candidate points are voxelized at voxel sizes (0.1, 0.1, 0.1) to obtain the input voxel tensor input_voxel. Applying the template kernel to the voxel tensor in a convolutional manner yields the matching response map response_map = conv(input_voxel, module_conv). Calculating the peak positions x, y, z of the response map and the corresponding angle θ yields the 3D target detection result.

[0197] Step 4: Reprojection Correction. Based on the 3D target detection results, generate the coordinates of the eight corner points, including the bounding box. Projecting these 8 corner points onto the image plane yields the pixel coordinates (u i v i ), i = 1, ..., 8. Based on these pixel coordinates and the 2D detection results (u1, v1, u2, v2), the error functions in the height and horizontal directions are constructed as follows:

[0198]

[0199]

[0200] The corrected height value was obtained by optimizing it using gradient descent. With the corrected horizontal value It then outputs the final 3D target detection results.

[0201] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0202] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A multimodal three-dimensional target detection method combining view frustum and geometric constraints, characterized in that, include: Step S1: Use a two-dimensional target detector to generate a two-dimensional perception result on the camera image to obtain the two-dimensional bounding box of the target to be detected in the image; according to the extrinsic parameters between the sensors and the intrinsic parameters of the camera, project the LiDAR point cloud onto the image plane, extract the point cloud projected into the two-dimensional bounding box, and obtain the corresponding in-frust point cloud. Step S2: Perform ground detection on the point cloud within the view frustum, and perform preliminary filtering on the point cloud within the view frustum based on the position of the intersection of the ground and the lower boundary of the view frustum. Use DBSCAN clustering on the filtered point cloud to obtain each candidate point cloud cluster, and set a distance threshold to merge the point cloud clusters. Set a scoring standard for the point cloud clusters, score each point cloud cluster, and finally select the point cloud cluster with the highest score as the result of the first stage. Step S3: Voxelize the point cloud results from the first stage and perform template matching convolution calculation with the refined template kernel to obtain the matching response map. Select the peak position in the response map and deduce its corresponding three-dimensional coordinate position and orientation based on the coordinate relationship to obtain the detection result of the three-dimensional target. Step S4: Use the consistency between the 3D projection results and the 2D detection results to correct the height and lateral direction in the 3D target detection results to obtain the final 3D target detection results.

2. The multimodal three-dimensional target detection method combining view frustum and geometric constraints according to claim 1, characterized in that, Step S1 adopts the following: Step S1.1: Use a two-dimensional object detector to detect the camera image and obtain the two-dimensional detection result (u1, v1, u2, v2, cls, conf) of the target to be detected, where (u1, v1, u2, v2) represent the left boundary coordinates u1, the upper boundary coordinates v1, the right boundary coordinates u2, and the lower boundary coordinates v2 of the two-dimensional detection box in the image, cls represents the category attribute of the target, and conf represents the confidence score of the detection box; Step S1.2: Using the extrinsic parameters between the camera and the LiDAR, the LiDAR point cloud is transformed into the camera coordinate system. Then, based on the camera's intrinsic parameters, the point cloud data in the camera coordinate system is projected onto the image plane. The point set with positive depth and whose projection is located within the two-dimensional detection box is selected and mapped to the original LiDAR point cloud to form an in-view frustum point cloud.

3. The multimodal three-dimensional target detection method combining view frustum and geometric constraints according to claim 1, characterized in that, Step S2 employs the following: Step S2.1: Sort all point clouds within the view frustum by height value from smallest to largest, and select the top N. LPR The average height of each point is taken as the initial height H of the ground. init Set a distance threshold Δh1, and ensure that all heights satisfy z <H init The point +Δh1 is considered a ground point. The least squares method is used to fit the plane equation of the ground based on the current ground point. The intersection position of the lower boundary of the view frustum and the view frustum is calculated based on the ground equation. The coordinates (x, y, y) of the intersection position are then used to determine the intersection position. inter ,y inter and distance threshold d inter Screening out points in the cloud that are far from the boundary with the view frustum, satisfying the following conditions: The points are selected, while points belonging to the ground are filtered out; Step S2.2: Use DBSCAN clustering to cluster the point cloud, obtaining several candidate point cloud sets set1, set2, ..., set n For the i-th point cloud set set i With the j-th point cloud set set j Calculate the farthest distance between two point sets. If the current farthest distance is less than the distance merging threshold d, then... merge Then, merge the two current point sets; after merging all candidate point sets, a new point cloud set set1, set2, ..., set is obtained. m ; Step S2.3: Each candidate point set is scored according to the preset scoring rules, and the point set with the highest score is selected as the first-stage candidate point; The preset scoring rules include: the more points included, the higher the score; the more ground and background points included, the lower the score; or the closer the distance to the intersection of the ground and the lower boundary of the view frustum, the higher the score.

4. The multimodal three-dimensional target detection method combining view frustum and geometric constraints according to claim 1, characterized in that, Step S3 employs the following: Step S3.1: Obtain the ground truth point cloud for each category from the training set of the autonomous driving dataset based on the 3D label values, and calculate the average 3D size of each category (l m ,w m ,h m The center coordinates of the ground truth point cloud for each category are normalized to (0,0,0), the 3D size is normalized to the average value, and the orientation angle is normalized to 0°. The point sets are then stitched together to obtain a preliminary template point set. Each template point set is then flipped along the x-axis to enhance symmetry for each category. Finally, the template is downsampled using a VoxelGridfilter to obtain a filtered template. K is set... θ The number of angle intervals is used to divide the interval from -180° to 180° into K equal intervals. θ Each angle value is used to rotate the template, resulting in K with orientation information. θ A template, with a voxel size (x grid ,y grid ,z grid The current template is voxelized to obtain the template kernel for each category, with a dimension of [K]. θ ,L m W m H m ]; Step S3.2: Using voxel size (x grid ,y grid , z grid Voxelization is performed on the candidate first-stage point cloud to obtain the size [L]. p W p H p The three-dimensional tensor of ]; Step S3.3: Apply the template kernel to the input voxel tensor, set the convolution stride to 1, and the edge padding method to zero-padding, to obtain a tensor of size [K]. θ ,L p W p H p The matching response map shows that positions with high response values ​​represent positions where the distribution of the point cloud to be matched at the current position is highly similar to the distribution of the template. Step S3.4: Obtain the peak position in the response graph and get the angle value θ corresponding to the current peak position. dt and three-dimensional coordinate position (x dt ,y dt ,z dt The output is the detection result of the three-dimensional target (x). dt ,y dt ,z dt ,l m ,w m ,h m ,θ dt ).

5. The multimodal three-dimensional target detection method combining view frustum and geometric constraints according to claim 1, characterized in that, Step S4 employs the following: Step S4.1: Detection results of 3D targets (x dt ,y dt ,z dt ,l m ,w m ,h m ,θ dt The detection result in the camera coordinate system (x) is obtained by transforming the camera extrinsic parameters to the camera coordinate system. cam ,y cam ,z cam ,l cam ,w cam ,h cam ,θ cam Based on the definition of the 3D detection box, the coordinates of the 8 corner points of the 3D detection box in the camera coordinate system are calculated: Then, based on the camera's intrinsic focal length and offset parameter f u ,f v ,c u ,c v The projected coordinates of the eight corner points on the image plane are obtained as follows: These coordinates are labeled as (m i ,n i The value of i ranges from 1 to 8; Step S4.2: Input the projected pixel coordinates (m) i ,n i Based on the two-dimensional detection results (u1, v1, u2, v2), construct the error function in the height direction: Substituting the formula for the eight corner points into the above formula, the variable that needs optimization is y. cam The corrected height value was obtained by optimizing it using gradient descent. Step S4.3: Input the projected pixel coordinates (m) i ,n i Using the two-dimensional detection results (u1, v1, u2, v2), i = 1, ..., 8, construct the error function in the lateral direction: Substituting the formula for the eight corner points into the above formula, the variable that needs optimization is x. cam The corrected lateral value was obtained by optimizing it using gradient descent. It then outputs the final 3D target detection results.

6. A multimodal three-dimensional target detection system combining view frustum and geometric constraints, characterized in that, include: Module M1: Uses a two-dimensional target detector to generate a two-dimensional perception result on the camera image, and obtains the two-dimensional bounding box of the target to be detected in the image; based on the extrinsic parameters between the sensors and the intrinsic parameters of the camera, the LiDAR point cloud is projected onto the image plane, and the point cloud projected into the two-dimensional bounding box is extracted to obtain the corresponding in-frust point cloud; Module M2: Performs ground detection on the point cloud within the view frustum, and performs preliminary filtering of the point cloud within the view frustum based on the location of the intersection between the ground and the lower boundary of the view frustum. The filtered point cloud is clustered using DBSCAN to obtain each candidate point cloud cluster, and the point cloud clusters are merged by setting a distance threshold. The scoring criteria for the point cloud clusters are set, and each point cloud cluster is scored. Finally, the point cloud cluster with the highest score is selected as the result of the first stage. Module M3: Voxelizes the point cloud results from the first stage and performs template matching convolution calculation with the refined template kernel to obtain the matching response map. The peak position in the response map is selected, and its corresponding 3D coordinate position and orientation are deduced based on the coordinate relationship to obtain the detection result of the 3D target. Module M4: Utilizes the consistency between the 3D projection results and the 2D detection results to correct the height and lateral dimensions in the 3D detection results, thus obtaining the final 3D detection results.

7. The multimodal three-dimensional target detection system combining view frustum and geometric constraints according to claim 6, characterized in that, The module M1 adopts: Module M1.1: Uses a two-dimensional object detector to detect the camera image and obtains the two-dimensional detection result (u1, v1, u2, v2, cls, conf) of the target to be detected, where (u1, v1, u2, v2) represent the left boundary coordinates u1, the upper boundary coordinates v1, the right boundary coordinates u2, and the lower boundary coordinates v2 of the two-dimensional detection box in the image, cls represents the target's category attribute, and conf represents the confidence score of the detection box; Module M1.2: Using the extrinsic parameters between the camera and the LiDAR, the LiDAR point cloud is transformed into the camera coordinate system. Then, based on the camera's intrinsic parameters, the point cloud data in the camera coordinate system is projected onto the image plane. The point set with positive depth and whose projection is located within the two-dimensional detection box is selected and mapped to the original LiDAR point cloud to form an in-frust point cloud.

8. The multimodal three-dimensional target detection system combining view frustum and geometric constraints according to claim 6, characterized in that, The module M2 adopts: Module M2.1: Sort all point clouds within the view frustum by height values ​​from smallest to largest, and select the top N. LPR The average height of each point is taken as the initial height H of the ground. init Set a distance threshold Δh1, and ensure that all heights satisfy z <H init The point +Δh1 is considered a ground point. The least squares method is used to fit the plane equation of the ground based on the current ground point. The intersection position of the lower boundary of the view frustum and the view frustum is calculated based on the ground equation. The coordinates (x, y, y) of the intersection position are then used to determine the intersection position. inter ,y inter and distance threshold d inter Screening out points in the cloud that are far from the boundary with the view frustum, satisfying the following conditions: The points are selected, while points belonging to the ground are filtered out; Module M2.2: Uses DBSCAN clustering to cluster point clouds, obtaining several candidate point cloud sets set1, set2, ..., set n For the i-th point cloud set set i With the j-th point cloud set set j Calculate the farthest distance between two point sets. If the current farthest distance is less than the distance merging threshold d, then... merge Then, merge the two current point sets; after merging all candidate point sets, a new point cloud set set1, set2, ..., set is obtained. m ; Module M2.3: Each candidate point set is scored according to a preset scoring rule, and the point set with the highest score is selected as the first-stage candidate point; The preset scoring rules include: the more points included, the higher the score; the more ground and background points included, the lower the score; or, the closer the distance to the intersection of the ground and the lower boundary of the view frustum, the higher the score.

9. The multimodal three-dimensional target detection system combining view frustum and geometric constraints according to claim 6, characterized in that, The module M3 adopts: Module M3.1: Obtains ground truth point clouds for each category from the training set of the autonomous driving dataset based on the 3D labels, and calculates the average 3D size of each category (l m ,w m ,h m Then, the center coordinates of the ground truth point cloud for each category are normalized to (0,0,0), the 3D size is normalized to the average value, and the orientation angle is normalized to 0°. The point sets are then stitched together to obtain a preliminary template point set. Each template point set is then flipped along the x-axis to enhance symmetry, and a VoxelGrid filter is used to downsample the templates to obtain the filtered templates. K is then set... θ The number of angle intervals is used to divide the interval from -180° to 180° into K equal intervals. θ Each angle value is used to rotate the template, resulting in K with orientation information. θ A template, with a voxel size (x grid ,y grid ,z grid The current template is voxelized to obtain the template kernel for each category, with a dimension of [K]. θ ,L m W m H m ]; Module M3.2: in voxel size (x grid ,y grid ,z grid Voxelization is performed on the candidate first-stage point cloud to obtain the size [L]. p W p H p The three-dimensional tensor of ]; Module M3.3: Applies the template kernel to the input voxel tensor, sets the convolution stride to 1, and uses zero-padding to fill the edges, resulting in a tensor of size [K]. θ ,L p W p H p The matching response map shows that positions with high response values ​​represent positions where the distribution of the point cloud to be matched at the current position is highly similar to the distribution of the template. Module M3.4: Obtain the peak position in the response graph and get the angle value θ corresponding to the current peak position. dt and three-dimensional coordinate position (x dt ,y dt ,z dt The output is the detection result of the three-dimensional target (x). dt ,y dt ,z dt ,l m ,w m ,h m ,θ dt ).

10. The multimodal three-dimensional target detection system combining view frustum and geometric constraints according to claim 6, characterized in that, The module M4 adopts: Module M4.1: Detection results of 3D targets (x dt ,y dt ,z dt ,l m ,w m ,h m ,θ dt The detection result in the camera coordinate system (x) is obtained by transforming the camera extrinsic parameters to the camera coordinate system. cam ,y cam ,z cam ,l cam ,w cam ,h cam ,θ cam Based on the definition of the 3D detection box, the coordinates of the 8 corner points of the 3D detection box in the camera coordinate system are calculated: Then, based on the camera's intrinsic focal length and offset parameter f u ,f v ,c u ,c v The projected coordinates of the eight corner points on the image plane are obtained as follows: These coordinates are labeled as (m i ,n i The value of i ranges from 1 to 8; Module M4.2: Input projected pixel coordinates (m) i ,n i Based on the two-dimensional detection results (u1, v1, u2, v2), construct the error function in the height direction: Substituting the formula for the eight corner points into the above formula, the variable that needs optimization is y. cam The corrected height value was obtained by optimizing it using gradient descent. Module M4.3: Input projected pixel coordinates (m) i ,n i Using the two-dimensional detection results (u1, v1, u2, v2), i = 1, ..., 8, construct the error function in the lateral direction: Substituting the formula for the eight corner points into the above formula, the variable that needs optimization is x. cam The corrected lateral value was obtained by optimizing it using gradient descent. It then outputs the final 3D target detection results.

Images