Lidar point cloud target detection method and system based on camera image driving

Abstract

The application discloses a camera image driving-based laser radar point cloud target detection method and system, and a two-stage three-dimensional target detection network can efficiently and accurately detect three-dimensional targets from image and laser point cloud data obtained from indoor and outdoor scenes. The first-stage network can locate three-dimensional targets in the frustum point cloud obtained by back-projection based on the image detection result, and the second-stage network optimizes the detection frame based on the result of the first-stage network, improves the detection effect on the hidden or long-distance objects, the context foreground point extraction module considers the target context information to enhance the detection result, extracts the background points in a certain range of the foreground points as the context foreground points, significantly improves the target detection result, and through the fusion of the semantic features learned from the two-dimensional image and the information based on the target and the target context in the three-dimensional space, the accuracy of three-dimensional bounding box prediction can be improved.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving, specifically relating to a method and system for target detection of LiDAR point clouds based on camera image driving. Background Technology

[0002] When autonomous vehicles navigate on roads, they need to detect the surrounding 3D scene, obtain the category of objects in 3D space (e.g., cars, pedestrians, and cyclists), and return their geometric spatial position, orientation, and semantic instance labels. This information is crucial for subsequent risk assessment, path planning, and decision-making control. Scene data can be obtained through sensors such as cameras or LiDAR. Based on sensor type, 3D object detection methods are mainly divided into three types: LiDAR point cloud-based 3D object detection, view-based 3D object detection, and multi-sensor fusion-based 3D object detection. LiDAR point cloud-based methods can directly detect 3D objects from point clouds, but they struggle to accurately and efficiently search for objects in large-scale, sparse, and unstructured point clouds. View-based methods convert 3D point clouds into 2D views and utilize mature image detectors to detect objects. However, the projection from 3D space to a 2D view may lose some geometrically relevant spatial information in 3D space. Multi-sensor fusion schemes leverage the advantages of both LiDAR point cloud-based and view-based methods to significantly improve 3D object detection performance.

[0003] Existing methods first assume the availability of 2D candidate boxes in the image, which can be obtained from off-the-shelf object detectors. The detected 2D candidate boxes are then back-projected into 3D space to obtain 3D frustum point cloud regions. A series of (potentially overlapping) frustums are proposed for each region by sliding along the frustum axis. These acquired frustums define a local point cloud set. Given a sequence of frustum and point associations, PointNet is used to aggregate point-oriented features into frustum-level feature vectors.

[0004] Then, in its early stages, these feature vectors are used as 2D feature maps and a subsequent fully convolutional network (FCN) is used to downsample and upsample the view frustum so that its features are fully fused along the view frustum axis at a higher view frustum resolution. Together with the final detection head, this method supports end-to-end continuous estimation of oriented 3D boxes, while also proposing a fully convolutional network variant for extracting multi-resolution view frustum features.

[0005] The shortcomings of existing technologies are: they cannot accurately detect targets from view frustum points with background and clutter interference, and the background information of the object is not utilized during the detection process, resulting in suboptimal detection results; inaccurate two-dimensional candidate boxes lead to inaccurate detection results, and target boundary information is easily lost in back projection. Summary of the Invention

[0006] In order to solve the technical problems existing in the background art, the present invention aims to provide a method and system for target detection of LiDAR point clouds based on camera image driving.

[0007] To solve the technical problem, the technical solution of the present invention is as follows:

[0008] A camera image-driven LiDAR point cloud target detection method, the method comprising:

[0009] Preprocessing is performed on camera image data and lidar point cloud data to obtain view frustum point cloud data;

[0010] A one-stage detection box prediction network is constructed, and a loss function is designed to optimize the network. The optimized one-stage detection box prediction network is then used to process the frustum point cloud data to obtain the target 3D detection box.

[0011] A two-stage detection box optimization network is constructed, and a loss function is designed to optimize the two-stage detection box optimization network. The optimized two-stage detection box optimization network is used to process the point cloud data within the magnified target 3D detection box to obtain an accurate target 3D detection box, thus realizing target detection in the driving scene of autonomous vehicles.

[0012] Furthermore, the preprocessing specifically includes:

[0013] Acquire camera image data and LiDAR point cloud data;

[0014] The image data is subjected to target detection processing to obtain a two-dimensional detection box of the target object;

[0015] Based on the two-dimensional detection bounding box of the target object, the view frustum point cloud data is extracted from the lidar point cloud data by utilizing the projection relationship between the camera coordinate system and the lidar coordinate system.

[0016] Furthermore, camera image data and lidar point cloud data are collected by using cameras and lidar mounted on the autonomous vehicle, respectively.

[0017] Furthermore, the processing of the view frustum point cloud data specifically includes:

[0018] The view frustum point cloud data is processed using a context foreground point segmentation network to obtain the target context foreground points;

[0019] The target center point and bounding box prediction network are used to process the target context foreground points to obtain the target 3D detection box.

[0020] Furthermore, the view frustum point cloud data is processed using a context-foreground segmentation network, specifically including:

[0021] The point cloud segmentation network is used to process the view frustum point cloud data to obtain the foreground and background points of the target.

[0022] The target context points are collected from the background points using a neighbor search algorithm, and the target foreground points are fused with the collected target context points to obtain the target context foreground points.

[0023] Furthermore, the network uses the target center point and bounding box prediction to process the target context foreground points, specifically including:

[0024] The target center point prediction network is used to process the target context foreground points to obtain the estimated target center point coordinates;

[0025] Based on the estimated center point coordinates of the target, the target context foreground points are transformed to the target coordinate system;

[0026] A bounding box prediction network is used to process the contextual foreground points after coordinate transformation to obtain the target's 3D detection box.

[0027] Furthermore, the processing of the target 3D detection bounding box specifically includes:

[0028] The point cloud data within the magnified 3D detection box of the target is used as the input to the two-stage detection box optimization network;

[0029] After processing with the same point cloud segmentation network, target center point prediction network, and bounding box prediction network as in the first stage, an accurate 3D target detection box is obtained.

[0030] Furthermore, a one-stage detection box prediction network is constructed, and a multi-task loss function is designed that includes point cloud segmentation, center point prediction, detection box prediction, and object classification, specifically as follows:

[0031] L multi-task =L seg +L objectness +L center-reg +L box ;

[0032] Among them, L seg For semantic segmentation loss, L center-reg L is the regression loss for the center point of the detection box. obiectness Loss of the target score;

[0033] L box =L center-reg_box +L ang-cls +20L angle-reg +L size-cls +20L size-reg +10L corner ;

[0034] Among them, L center-reg-boxTo predict the residual loss at the center point, L ang-cls and L angle-reg L represents the angular classification loss and regression loss, respectively. size-cls and L size-reg These represent the classification loss and regression loss of the detection boxes, respectively, and the corner loss L. corner It represents the minimum distance between the corner points of the prediction box and the corner points of the truth box.

[0035] Furthermore, a two-stage detection box optimization network is constructed, and the same multi-task loss function as the one used in the first stage is designed, which includes point cloud segmentation, center point prediction, detection box prediction, and object classification. Specifically:

[0036] L multi-task =L seg +L objectness +L center-reg +L box ;

[0037] Among them, L seg For semantic segmentation loss, L center-reg The regression loss for the center point of the detection box, and L obiectness Loss of the target score;

[0038] L box =L center-reg_box +L ang-cls +20L angle-reg +L size-cls +20L size-reg +10L corner ;

[0039] Among them, L center-reg-box To predict the residual loss at the center point, L ang-cls and L angle-reg L represents the angular classification loss and regression loss, respectively. size-cls and L size-reg These represent the classification loss and regression loss of the detection boxes, respectively, and the corner loss L. corner This represents the minimum distance between the corner points of the prediction box and the corner points of the truth box.

[0040] A camera image-driven lidar point cloud target detection system, the system comprising:

[0041] One or more processors;

[0042] Memory, used to store one or more programs;

[0043] When the one or more programs are executed by the one or more processors, the one or more processors perform the camera image-driven LiDAR point cloud target detection method as described above.

[0044] Compared with the prior art, the advantages of the present invention are as follows:

[0045] The proposed two-stage 3D target detection network can efficiently and accurately detect 3D targets from images and laser point cloud data acquired from indoor and outdoor scenes. The first stage network can locate 3D targets in the frustum point cloud obtained by backprojection based on the image detection results, while the second stage network optimizes the detection box based on the results of the first stage network, improving the detection effect of occluded or distant objects.

[0046] The proposed context foreground point extraction module enhances the detection results by considering the target's context information. By extracting background points within a certain range of the foreground point as context foreground points, it significantly improves the target detection results.

[0047] By fusing semantic features learned from two-dimensional images with information based on the target and its context in three-dimensional space, the accuracy of three-dimensional bounding box prediction can be improved. Attached Figure Description

[0048] Figure 1 A camera image-driven LiDAR point cloud target detection framework;

[0049] Figure 2 Contextual foreground point extraction;

[0050] Figure 3 Results of front view target detection and lidar point cloud target detection on the KITTI dataset. Detailed Implementation

[0051] The specific implementation of the present invention is described below with reference to embodiments:

[0052] It should be noted that the structures, proportions, sizes, etc. shown in this specification are only used to complement the content disclosed in the specification for those skilled in the art to understand and read, and are not intended to limit the conditions under which the present invention can be implemented. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0053] Furthermore, the terms such as "upper," "lower," "left," "right," "middle," and "one" used in this specification are merely for clarity of description and are not intended to limit the scope of the invention. Any changes or adjustments to their relative relationships, without substantially altering the technical content, should also be considered within the scope of the invention.

[0054] Example 1:

[0055] A camera image-driven LiDAR point cloud target detection method, the method comprising:

[0056] Preprocessing is performed on camera image data and lidar point cloud data to obtain view frustum point cloud data;

[0057] A one-stage detection box prediction network is constructed, and a loss function is designed to optimize the network. The optimized one-stage detection box prediction network is then used to process the frustum point cloud data to obtain the target 3D detection box.

[0058] A two-stage detection box optimization network is constructed, and a loss function is designed to optimize the two-stage detection box optimization network. The optimized two-stage detection box optimization network is used to process the point cloud data within the magnified target 3D detection box to obtain an accurate target 3D detection box, thus realizing target detection in the driving scene of autonomous vehicles.

[0059] It can be understood that the proposed two-stage 3D target detection network can efficiently and accurately detect 3D targets from images and laser point cloud data acquired from indoor and outdoor scenes. The first stage network can locate 3D targets in the frustum point cloud obtained by backprojection based on the image detection results, while the second stage network optimizes the detection box based on the results of the first stage network, improving the detection effect of occluded or distant objects.

[0060] The proposed context foreground point extraction module enhances the detection results by considering the target's context information. By extracting background points within a certain range of the foreground point as context foreground points, it significantly improves the target detection results.

[0061] By fusing semantic features learned from two-dimensional images with information based on the target and its context in three-dimensional space, the accuracy of three-dimensional bounding box prediction can be improved.

[0062] Furthermore, the preprocessing specifically includes:

[0063] Acquire camera image data and LiDAR point cloud data;

[0064] The image data is subjected to target detection processing to obtain a two-dimensional detection box of the target object;

[0065] Based on the two-dimensional detection bounding box of the target object, the view frustum point cloud data is extracted from the lidar point cloud data by utilizing the projection relationship between the camera coordinate system and the lidar coordinate system.

[0066] Furthermore, camera image data and lidar point cloud data are collected by using cameras and lidar mounted on the autonomous vehicle, respectively.

[0067] Furthermore, the processing of the view frustum point cloud data specifically includes:

[0068] The view frustum point cloud data is processed using a context foreground point segmentation network to obtain the target context foreground points;

[0069] The target center point and bounding box prediction network are used to process the target context foreground points to obtain the target 3D detection box.

[0070] Furthermore, the view frustum point cloud data is processed using a context-foreground segmentation network, specifically including:

[0071] The point cloud segmentation network is used to process the view frustum point cloud data to obtain the foreground and background points of the target.

[0072] The target context points are collected from the background points using a neighbor search algorithm, and the target foreground points are fused with the collected target context points to obtain the target context foreground points.

[0073] Furthermore, the network uses the target center point and bounding box prediction to process the target context foreground points, specifically including:

[0074] The target center point prediction network is used to process the target context foreground points to obtain the estimated target center point coordinates;

[0075] Based on the estimated center point coordinates of the target, the target context foreground points are transformed to the target coordinate system;

[0076] A bounding box prediction network is used to process the contextual foreground points after coordinate transformation to obtain the target's 3D detection box.

[0077] Furthermore, the processing of the target 3D detection bounding box specifically includes:

[0078] The point cloud data within the magnified 3D detection box of the target is used as the input to the two-stage detection box optimization network;

[0079] After processing with the same point cloud segmentation network, target center point prediction network, and bounding box prediction network as in the first stage, an accurate 3D target detection box is obtained.

[0080] Furthermore, a one-stage detection box prediction network is constructed, and a multi-task loss function is designed that includes point cloud segmentation, center point prediction, detection box prediction, and object classification, specifically as follows:

[0081] L multi-task =L seg +L objectness +L center-reg +L box ;

[0082] Among them, L seg For semantic segmentation loss, L center-reg L is the regression loss for the center point of the detection box. objectnessLoss of the target score;

[0083] L box =L center-reg_box +L ang-cls +20L angle-reg +L size-cls +20L size-reg +10L corner ;

[0084] Among them, L center-reg_box To predict the residual loss at the center point, L ang-cls and L angle-reg L represents the angular classification loss and regression loss, respectively. size-cls and L size-reg These represent the classification loss and regression loss of the detection boxes, respectively, and the corner loss L. corner It represents the minimum distance between the corner points of the prediction box and the corner points of the truth box.

[0085] Furthermore, a two-stage detection box optimization network is constructed, and the same multi-task loss function as the one used in the first stage is designed, which includes point cloud segmentation, center point prediction, detection box prediction, and object classification. Specifically:

[0086] L multi-task =L seg +L objectness +L center-reg +L box ;

[0087] Among them, L seg For semantic segmentation loss, L center-reg The regression loss for the center point of the detection box, and L obiectness Loss of the target score;

[0088] L box =L center-reg_box +L ang-cls +20L angle-reg +L size-cls +20L size-reg +10L corner ;

[0089] Among them, L center-reg_box To predict the residual loss at the center point, L ang-cls and L angle-reg L represents the angular classification loss and regression loss, respectively. size-cls and L size-reg These represent the classification loss and regression loss of the detection boxes, respectively, and the corner loss L. corner This represents the minimum distance between the corner points of the prediction box and the corner points of the truth box.

[0090] A camera image-driven lidar point cloud target detection system, the system comprising:

[0091] One or more processors;

[0092] Memory, used to store one or more programs;

[0093] When the one or more programs are executed by the one or more processors, the one or more processors perform the camera image-driven LiDAR point cloud target detection method as described above.

[0094] Example 2:

[0095] like Figure 1 As shown, the proposed method consists of two stages: the first stage is for the bounding box prediction network, and the second stage is for the bounding box optimization network. Both networks include point cloud segmentation, center prediction, and bounding box prediction modules. Figure 1 This demonstrates the framework of the object detection method proposed in this paper.

[0096] In our method, we first detect 2D bounding boxes using a 2D object detector, and then project these bounding boxes into 3D frustums using a known camera projection matrix. These frustums define the 3D search region for object detection in the point cloud. Points in camera coordinates are collected to form the frustum point cloud. This mechanism leverages mature 2D detectors and significantly reduces the computational cost of point-based 3D object detection frameworks. To improve the rotation invariance of the proposed method, these frustums are normalized so that their central axes are orthogonal to the image plane. In the proposed two-stage point-based detection framework, the stage one network predicts bounding boxes within the frustum point cloud as input, while the stage two network uses points from the magnified predicted bounding boxes to optimize the predictions and compensate for incorrect 2D detection results.

[0097] A. Bounding box prediction

[0098] 1) Context-based foreground point segmentation

[0099] Within normalized view frustums, there are two methods for object detection: (1) detecting objects directly from the point cloud; and (2) first extracting foreground points and then using these points to predict bounding boxes. Although view frustums reduce the least relevant background and clutter, the remaining points and overlapping objects can still interfere with the accurate localization of the object.

[0100] Foreground point segmentation can accurately locate associated targets based on the foreground context. To utilize the geometric features of each foreground point, we apply a multi-scale GeoConv[2] with an encoder-decoder structure to the input view frustum points. Since GeoConv can only extract intra-target features, PointNet[3] is used in our backbone to extract inter-target features as the downsampling scale increases. Semantic cues learned from the 2D image can also be used for segmentation. This information is encoded as a one-hot encoded class vector and concatenated with the learned global features, and then backpropagated to the features of the point-by-point class labels. This segmentation network is a binary classifier that segments the background and foreground points. Figure 2 The process of the proposed context-ahead point collection method is shown.

[0101] Contextual information around the target can improve the accuracy of bounding box reasoning[4].

[0102] Therefore, this paper proposes a method for collecting contextual foreground points to gather contextual points from background points (see [link to paper]). Figure 2 For each background point, we collect its 16 nearest neighbors. If there is at least one foreground point, the background point is marked as a context point. Query ball search and nearest neighbor search are commonly used as neighbor search methods. Nearest neighbor search searches for the nearest point without considering distance. Therefore, background points far from the foreground point are likely to be selected as context points. These points contribute little to object detection. To avoid this contamination, the experiment chooses query ball search with a radius of 0.9m as the neighbor search method. This method not only selects nearby background points but also preserves the geometric properties of the target. All context points and foreground points are combined into a context foreground point for bounding box prediction. Experimental results demonstrate the effectiveness of the proposed method.

[0103] 2) Residual center estimation and bounding box prediction

[0104] Coordinate transformations are crucial for improving object detection performance. These transformations align points within a set of constrained and canonical frames. Specifically, center-oriented transformations help 3D detectors leverage target geometry properties such as symmetry and planarity. In the obtained contextual foreground points, we normalize these points to local coordinates by subtracting their average coordinates to improve translation invariance. These points are then fed into a T-Net network to predict residual box centers. Contextual points are disregarded during computation to ensure the predicted centers are closer to the target portion. The normalized points are then transformed into predicted target centers for bounding box prediction with canonical coordinates.

[0105] To predict accurate bounding boxes, the bounding box prediction network should consider both context-based and object-based features of the target. Object-based features encode target information, while context-based features provide information about the target's surroundings. Therefore, features extracted from the context foreground points represent object-based attributes. Features learned from view frustums are better suited to represent context-based features. In this paper, context-based features extracted from the foreground segmentation network are concatenated with object-based features learned from the canonical context foreground points to predict bounding box parameters. PointNet was selected as the bounding box prediction network. Furthermore, the reflection attributes of each point and semantic features learned from the 2D detection boxes are also encoded for bounding box prediction. Experimental results demonstrate the effectiveness of this network.

[0106] In this algorithm, each 3D bounding box is parameterized as (x, y, z, h, w, l, θ, score), where (x, y, z) represents the target center point, (h, w, l) represents the target dimensions (length, width, height), θ represents the target orientation, and score represents the target score. For angle prediction, we predefine N. a and N s The angles and sizes were then divided into equal bins, and the angles and sizes were categorized into different bins. Regression was performed on the residuals relative to the bin values. N a Set to 12, N s Set to 8. The bounding box prediction network output is 3 + 4 × N. s +2×N a +2.

[0107] B. Bounding box optimization

[0108] While existing mature 2D measuring instruments detect 2D regions with sufficient accuracy, they cannot accurately enclose the target instance. Larger 2D boxes contain the entire target instance but also include more irrelevant background and clutter, while smaller 2D boxes contain less background noise but do not provide a complete 3D target instance. To compensate for this, during the optimization phase, we collect points from magnified predicted bounding boxes as input. Specifically, we magnify each predicted box by a selected factor, set to 1.2 in this work.

[0109] The points within this magnified bounding box are normalized. To further improve 3D detection performance, point-by-point, object-based, and context-based features are considered to extract fine-grained bounding box information. The input points contain the object and a limited set of context points, which can be considered as context foreground points. Therefore, the context foreground point extraction module is removed at this stage, while other modules are the same as in the first-stage network. All input points are used to learn bounding box information. The point-by-point features learned in the segmentation network are concatenated with the point-precise features learned in the residual center prediction network and the bounding box prediction network to generate object-based features, respectively. Similarly, the context-based features obtained in the segmentation stage are concatenated with the one-hot semantic 2D cues and object-based features, respectively, for residual center prediction and bounding box prediction. The bounding box prediction network at this stage also outputs 3+4×N. s +2×N a +2.

[0110] Both object detection loss and semantic segmentation loss are two types of cross-entropy losses. We employ similar column-based classification and regression losses for bounding box optimization.

[0111] The bounding box loss is composed of sub-losses such as center regression, heading estimation, and size estimation, which are derived from Huber loss.

[0112]

[0113] Where L center-reg_box Residual loss at the predicted center point, L ang-cls and L angle-reg These represent the angular classification loss and regression loss, respectively. size-cls and L size-reg These represent the classification loss and regression loss of the detection boxes, respectively. Corner loss L corner It is the minimum distance between the corner points of the prediction box and the corner points of the truth box (including rotation by 90°).

[0114] The ground truth value of the target score can be labeled using the distance from the center of the predicted bounding box to the center of the ground truth bounding box. If the distance is greater than 0.3 meters, the ground truth value is marked as 1; otherwise, it is marked as 0. This establishes a geometric correlation between the predicted score and the predicted bounding box. Therefore, the loss of the entire network can be defined as:

[0115] L multi-task =L seg +L objectness +L center-reg +L box (2)

[0116] Among them, L seg It is the semantic segmentation loss, L center-reg The regression loss of the detection box center point, and L obiectness It is the target score loss.

[0117] References:

[0118] [1] Z. Wang, and K. Jia, "Frustum ConvNet: sliding frustums to aggregate local point-wise features for amodal 3D object detection," IEEE / RSJ IROS, pp. 1742-1749, 2019.

[0119] [2] CRQi, W. Liu, C. Wu, H. Su, and LJ Guibas, "Frustum pointnets for 3D object detection from RGB-D data," in Proc. IEEE CVPR, p. 918-927, 2018.

[0120] [3] Y.Li, L.Ma, W.Tan, C.Sun, D.Cao, and J.Li, "GRNet: Geometric relationnetwork for 3D object detection from point clouds," ISPRS J.Photogramm.RemoteSens., vol.165, pp.43-53, 2020.

[0121] [4] CRQi, H.Su, K.Mo, and LJGuibas, "Pointnet: deep learning on pointsets for 3D classification and segmentation," in Proc.IEEE CVPR, pp.652-660, 2017.

[0122] Example 3:

[0123] In this embodiment, the image-based deep learning object detection method can be replaced with other rule-based object detection algorithms or other deep learning algorithms. Furthermore, the 3D object detection method in this embodiment can also be replaced with a voxel-based or graph convolution-based object detection algorithm; the final output will be the size and center position of the object's 3D bounding box and the object's category.

[0124] like Figure 3 As shown, Figure 3The results show the front view target detection and LiDAR point cloud target detection on the KITTI dataset.

[0125] Table 1 shows a comparison of AP values ​​for LiDAR point cloud target detection, bird's-eye view target detection, and front view target detection on the KITTI dataset.

[0126] Table 1

[0127]

[0128] Table 2 shows the model size, training time, and model training parameters on the KITTI dataset.

[0129] Table 2

[0130]

[0131]

[0132] The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

[0133] Many other changes and modifications can be made without departing from the concept and scope of this invention. It should be understood that this invention is not limited to the specific embodiments, and the scope of this invention is defined by the appended claims.

Claims

1. A method for driving a laser radar point cloud target detection based on a camera image, characterized in that, The method includes: Two-dimensional bounding boxes are generated using a two-dimensional object detector, and these two-dimensional bounding boxes are projected into three-dimensional space using a known camera projection matrix to form a frustum point cloud, defining the three-dimensional search area for object detection. Extract the view frustum point cloud data to form a contextual foreground point; A one-stage detection box prediction network is constructed, and a loss function is designed to optimize the network. The optimized one-stage detection box prediction network is then used to process the frustum point cloud data to obtain the target 3D detection box. The target 3D detection box is enlarged to form an enlarged detection box, and point cloud data is collected from the enlarged detection box. This point cloud data is used as input to the two-stage detection box optimization network for processing to obtain an accurate target 3D detection box. In the above process, the context foreground points are collected through segmentation methods and combined with target and background points to provide richer geometric and semantic information for subsequent detection box optimization; The processing of the view frustum point cloud data specifically includes: The view frustum point cloud data is processed using a context foreground point segmentation network to obtain the target context foreground points; The target center point and bounding box prediction network are used to process the target context foreground points to obtain the target 3D detection box; Processing view frustum point cloud data using a context-foreground segmentation network specifically includes: The point cloud segmentation network is used to process the view frustum point cloud data to obtain the foreground and background points of the target. The target context points are collected from the background points using a neighbor search algorithm, and the target foreground points are fused with the collected target context points to obtain the target context foreground points; The network uses the target center point and bounding box prediction to process the target context foreground points, specifically including: The target center point prediction network is used to process the target context foreground points to obtain the estimated target center point coordinates; Based on the estimated center point coordinates of the target, the target context foreground points are transformed to the target coordinate system; A bounding box prediction network is used to process the contextual foreground points after coordinate transformation to obtain the target's 3D detection box; The target 3D detection bounding box is processed, specifically including: The point cloud data within the magnified 3D detection box of the target is used as the input to the two-stage detection box optimization network; After processing with the same point cloud segmentation network, target center point prediction network and bounding box prediction network as in the first stage, an accurate 3D target detection box is obtained. A one-stage bounding box prediction network is constructed, and a multi-task loss function is designed that includes point cloud segmentation, center point prediction, bounding box prediction, and object classification. Specifically: ， wherein, is a semantic segmentation loss, is a regression loss of the center point of the detection frame, is a target score loss; ， wherein, is the residual loss of the center point, respectively represent the angle classification loss and the regression loss, respectively represent the classification loss and the regression loss of the detection frame, and the angle point loss is the minimum distance between the predicted angle point of the frame and the angle point of the true value frame. A two-stage detection box optimization network is constructed, and the same multi-task loss function as the one used in the first stage is designed, which includes point cloud segmentation, center point prediction, detection box prediction, and object classification. Specifically: ， wherein, is a semantic segmentation loss, is a regression loss for the center point of the detection box, and is a target score loss. ， in, To predict the residual loss at the center point, These represent the classification loss and regression loss, respectively. These represent the classification loss and regression loss of the detection boxes, and the corner loss, respectively. This represents the minimum distance between the corner points of the prediction box and the corner points of the truth box.

2. The camera image-driven lidar point cloud target detection method according to claim 1, characterized in that, Camera image data and lidar point cloud data are collected by using cameras and lidar mounted on autonomous vehicles, respectively.

3. A camera image-driven lidar point cloud target detection system, characterized in that, The system includes: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors perform the camera image-driven lidar point cloud target detection method as described in any one of claims 1-2.

Images