Three-dimensional object detection method based on foreground information down-sampling and point-level feature fusion
By introducing hybrid downsampling of image foreground information and feature fusion of attention mechanism into the 3D point cloud target detection model, the problem of insufficient target point retention in traditional methods is solved, and higher accuracy 3D target detection is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HARBIN ENG UNIV
- Filing Date
- 2024-05-21
- Publication Date
- 2026-05-29
Smart Images
Figure CN118505972B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of 3D target detection technology, and particularly relates to a 3D target detection method based on point-level feature fusion using foreground information downsampling. Background Technology
[0002] In recent years, object detection, as an important research direction in computer vision, has been widely applied in many fields. However, most researchers have focused on two-dimensional object detection based on planar images, which can only provide the relative position of objects in the image. Three-dimensional object detection, with its depth information, can accurately determine the spatial position of targets and plays a crucial role in complex tasks such as autonomous driving and obstacle avoidance in aircraft. Therefore, research on three-dimensional object detection has significant theoretical and practical value. However, three-dimensional data lacks color and texture information, making traditional single-modal detection methods prone to missed or false detections. With the development of multi-source sensors and information processing technologies, information fusion-based three-dimensional object detection methods aim to overcome the limitations of single-modal approaches, improve the accuracy and robustness of detection results, and achieve better performance in practical applications. This has become an important research area, and the academic community is still actively exploring information fusion strategies in three-dimensional object detection, with many researchers proposing various fusion methods.
[0003] However, traditional point-based single-modal 3D target detection methods suffer from insufficient target point retention after downsampling, and the difficulty in distinguishing and identifying objects with similar point cloud structures or incomplete point clouds when using only single-modal features of point clouds for 3D target detection, which easily leads to false detections and missed detections. Summary of the Invention
[0004] To address the aforementioned technical problems, this invention proposes a point-level feature fusion method for 3D target detection based on foreground information downsampling, thereby resolving the issues existing in the prior art.
[0005] To achieve the above objectives, this invention provides a point-level feature fusion method for 3D target detection based on foreground information downsampling, comprising the following steps:
[0006] A baseline 3D point cloud target detection model is constructed, and a hybrid downsampling method incorporating image foreground information is introduced into the baseline 3D point cloud target detection model to construct a 3D point cloud target detection model based on foreground information downsampling.
[0007] Based on the foreground information downsampling three-dimensional point cloud target detection model, combined with the attention mechanism-based point-level interpolation pooling feature fusion module, a point-level feature fusion three-dimensional point cloud target detection model based on foreground information downsampling is constructed.
[0008] The point cloud and its corresponding image are input into the point-level feature fusion 3D point cloud target detection model based on foreground information downsampling for detection, and a 3D target bounding box is obtained.
[0009] Preferably, the method for constructing a baseline 3D point cloud target detection model includes: constructing a point cloud feature extraction network and constructing a point cloud voting detection network, and constructing a baseline 3D point cloud target detection model based on the point cloud feature extraction network and the point cloud voting detection network;
[0010] The point cloud feature extraction network includes three SA modules, each of which includes a downsampling operation module and a point cloud feature extraction module.
[0011] The point cloud feature extraction network obtains downsampled points of attached features through downsampling operations and point cloud feature extraction.
[0012] Preferably, the point cloud voting detection network includes a voting point generation module, a clustering and grouping module, and a target bounding box generation module;
[0013] The workflow of the point cloud voting detection network is as follows: the downsampled points with attached features are used as seed points, voting prediction and center offset calculation are performed on the seed points to obtain voting points, the voting points are clustered by point-by-point ball query to obtain clusters, the clusters are predicted to obtain predicted target boxes, and the predicted target boxes are subjected to non-maximum suppression to obtain the final target boxes.
[0014] The overall loss function of the point cloud voting detection network is:
[0015] L = L vote-reg +L obj-cls +L dir-cls +L dir-reg +
[0016] L corner-reg +L size-reg +L center-reg +L sem-cls
[0017] In the formula, L vote-reg For the loss of votes, L obj-cls For the target foreground classification loss, L sem-cls For the target category classification loss, L size-reg To predict the regression loss for the deviation between the target bounding box and the anchor box dimensions, L center-reg To predict the regression loss for the center of the target bounding box, L dir-cls To predict the angular classification loss of the bounding box, L dir-reg To predict the regression loss of the target bounding box, L corner-reg The regression loss is used to predict the target bounding box and the eight corner points of the true target bounding box.
[0018] Preferably, the method for constructing a 3D point cloud target detection model based on foreground information downsampling includes:
[0019] A hybrid downsampling method incorporating foreground information is adopted to replace the farthest point sampling method in the downsampling operation module. A residual connection block is introduced into the point cloud feature extraction module to obtain a new downsampling feature extraction module. The new downsampling feature extraction module replaces the SA module. Then, a three-dimensional centrality loss is introduced into the point cloud voting detection network to obtain the three-dimensional point cloud target detection model based on foreground information downsampling.
[0020] Preferably, the hybrid downsampling method incorporating foreground information of the image includes:
[0021] The corresponding image is detected by a two-dimensional target detection model to obtain a two-dimensional detection result. The two-dimensional detection result is then segmented by perspective projection relationship to obtain a foreground point set and a background point set.
[0022] The foreground point set is randomly downsampled to obtain the foreground point sampling result; the background point set is sampled based on the farthest point of the composite distance to obtain the background point sampling result; the downsampling result is obtained by combining the foreground point sampling result and the background point sampling result.
[0023] Preferably, the three-dimensional centrality loss is expressed as:
[0024] L ctr-cls =-centerness * logcenterness-(1-centerness * log(1-centerness)
[0025] In the formula, L ctr-cls For the 3D centrality loss, centerness * is the true centrality corresponding to the voting point, and centerness is the predicted centrality corresponding to the voting point;
[0026] The true centrality can be expressed as:
[0027]
[0028] Among them, l * r * t * d * f * b * These represent the distances from the voting point to the left, right, top, bottom, front, and back boundaries of the actual 3D target bounding box, respectively.
[0029] Preferably, the method for constructing a 3D point cloud target detection model based on foreground information downsampling and point-level feature fusion includes:
[0030] In the 3D point cloud target detection model based on foreground information downsampling, a point-level interpolation pooling feature fusion module based on attention mechanism is introduced. The downsampling points of the attached features and the downsampling results detected by the 2D target detection model are respectively input into the point-level interpolation pooling feature fusion module based on attention mechanism to obtain the downsampling points of the attached fusion features. The downsampling points of the attached fusion features are used to replace the downsampling points of the attached features to construct the 3D point cloud target detection model based on foreground information downsampling point-level feature fusion.
[0031] Preferably, the method for constructing the attention-based point-level interpolation pooling feature fusion module includes:
[0032] The downsampling points of the attached features and the single-scale image features in the downsampling result are used as inputs. The coordinates of the downsampling points of the attached features in the corresponding image pixel plane are obtained through perspective projection relationship, and then converted into the coordinates of the feature map plane composed of the single-scale image features.
[0033] Interpolation pooling is performed on the feature map plane based on the coordinates to obtain the image features corresponding to the downsampled points of the attached features; the attached point cloud features of the downsampled points of the attached features and the image features corresponding to the downsampled points of the attached features are fed into a single-point weight learning module to learn weights and obtain fused features; the attached point cloud features of the downsampled points of the attached features are replaced with the fused features to obtain the downsampled points of the attached fused features as output; the fused feature calculation expression is:
[0034] f F (P)=f P (P)|wf I (P)
[0035] Where | represents the concatenation operation, f I (P) represents the image features corresponding to the downsampling points with attached features, f P (P) represents the attached point cloud features of the downsampled points with attached features, and w represents the weight.
[0036] Preferably, the interpolation pooling operation includes:
[0037] In the feature map plane, bilinear interpolation is used to obtain the corresponding interpolated features of the pixel plane coordinates. A 3×3 max pooling operation is used to obtain the corresponding pooled features of the pixel plane coordinates. The interpolated features and the pooled features are added channel by channel to obtain the image features corresponding to the downsampling points of the attached features. The expression for the image features corresponding to the downsampling points of the attached features is:
[0038] f I (P)=f N (P)+f B (P)
[0039] In the formula, f B (P) represents the corresponding interpolation feature of the pixel plane coordinate point, f N (P) represents the corresponding pooling feature of the coordinate point of the pixel plane.
[0040] Preferably, the method for constructing the single-point weight learning module includes:
[0041] The attached point cloud features of the downsampled points of the attached features and the corresponding image features are further encoded with an MLP layer to obtain encoded features. The encoded features are added by channel to obtain summed features. The summed features are compressed to the interval [-1, 1] using the Tanh activation function to obtain compressed features. The compressed features are reduced to 1 channel using an MLP layer to obtain preliminary weights. The preliminary weights are compressed to the interval using the Sigmoid activation function to obtain the final weights. The learning expression for the weights is as follows:
[0042] w = sigmoid(w1 tanh(w2f)) P (P)+w3f I (P)))
[0043] Where w1 is the weight learned by the first MLP layer, w2 is the weight learned by the second MLP layer, and w3 is the weight learned by the third MLP layer.
[0044] Compared with the prior art, the present invention has the following advantages and technical effects:
[0045] To preserve more target points after downsampling, this invention proposes a hybrid downsampling method incorporating foreground information. It uses image detection results to distinguish between foreground and background point sets, randomly downsampling the foreground point set and sampling the farthest point of the background point set using composite distance, thus better preserving target points. To improve the model's ability to extract point cloud structural features, this invention deepens the point cloud feature extraction network using residual connection blocks while avoiding the gradient vanishing problem, enhancing the model's feature extraction capability. To improve the quality of the predicted bounding boxes, this invention replaces the original target loss with a 3D centrality loss, suppressing predicted bounding boxes far from the object center voting points, allowing high-quality bounding boxes proposed by voting points closer to the center to have higher confidence, thereby improving detection accuracy. To address the model's false positives and false negatives, this invention proposes introducing a point-level interpolation pooling feature fusion module based on an attention mechanism to construct a multi-scale feature fusion network. This fully utilizes image semantic and texture features to supplement the single-modal features of the point cloud, further improving the model's detection accuracy.
[0046] The point-level feature fusion 3D target detection method based on foreground information downsampling proposed in this invention has a good 3D target detection effect. The model can make good use of the complementary relationship between image information and 3D point cloud information, with a lower frequency of false detection and false negative, and higher accuracy of 3D target detection, thus having a certain degree of effectiveness. Attached Figure Description
[0047] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:
[0048] Figure 1 This is a flowchart of a point-level feature fusion 3D target detection method based on foreground information downsampling, according to an embodiment of the present invention.
[0049] Figure 2 This is a schematic diagram of the three-dimensional point cloud target detection baseline model structure according to an embodiment of the present invention;
[0050] Figure 3 This is a detailed structural diagram of the point cloud feature extraction module in the three-dimensional point cloud target detection baseline model according to an embodiment of the present invention;
[0051] Figure 4 This is a detailed structural diagram of the point cloud voting detection network in the three-dimensional point cloud target detection baseline model according to an embodiment of the present invention;
[0052] Figure 5 This is a schematic diagram of the structure of a 3D point cloud target detection model based on foreground information downsampling according to an embodiment of the present invention;
[0053] Figure 6This is a schematic diagram of the perspective projection operation in the hybrid downsampling method of this invention.
[0054] Figure 7 This is a detailed structural diagram of the improved point cloud feature extraction module that introduces residual connections in the three-dimensional point cloud target detection model based on foreground information downsampling, according to an embodiment of the present invention.
[0055] Figure 8 The image shows the downsampling results of the downsampling method and the farthest point sampling method in the outdoor KITTI dataset according to embodiments of the present invention.
[0056] Figure 9 This is a training loss curve comparing the 3D point cloud target detection model based on foreground information downsampling in this embodiment of the invention with the baseline 3D point cloud target detection model. Figure 9 (a) is a graph of the training loss in the KITTI dataset. Figure 9 (b) is a training loss curve for the SUN RGB-D dataset;
[0057] Figure 10 This is a schematic diagram of the structure of a 3D point cloud target detection model based on foreground information downsampling and point-level feature fusion according to an embodiment of the present invention;
[0058] Figure 11 This is a schematic diagram of the point-level interpolation pooling feature fusion module in the point-level feature fusion 3D point cloud target detection model based on foreground information downsampling according to an embodiment of the present invention;
[0059] Figure 12 This is a schematic diagram of the bilinear interpolation operation used in the point-level interpolation pooling feature fusion module of the point-level feature fusion 3D point cloud target detection model based on foreground information downsampling in an embodiment of the present invention.
[0060] Figure 13 This is a schematic diagram of the weight learning module structure of the point-level interpolation pooling feature fusion module in the point-level feature fusion 3D point cloud target detection model based on foreground information downsampling in this embodiment of the invention.
[0061] Figure 14 This is a schematic diagram of the image feature extraction network structure in the point-level feature fusion 3D point cloud target detection model based on foreground information downsampling, as described in an embodiment of the present invention.
[0062] Figure 15 The training loss curves for the baseline 3D point cloud target detection model, the 3D point cloud target detection model based on foreground information downsampling, and the 3D point cloud target detection model based on point-level feature fusion based on foreground information downsampling, as shown in the embodiments of the present invention, are as follows: Figure 15 (a) is a graph of the training loss in the KITTI dataset. Figure 15(b) is a training loss curve in the SUN RGB-D dataset. Detailed Implementation
[0063] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.
[0064] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.
[0065] like Figure 1 As shown, this invention proposes a point-level feature fusion method for 3D target detection based on foreground information downsampling, comprising the following steps:
[0066] A baseline 3D point cloud target detection model is constructed, and a hybrid downsampling method incorporating image foreground information is introduced into the baseline 3D point cloud target detection model to construct a 3D point cloud target detection model based on foreground information downsampling.
[0067] Based on the foreground information downsampling three-dimensional point cloud target detection model, combined with the attention mechanism-based point-level interpolation pooling feature fusion module, a point-level feature fusion three-dimensional point cloud target detection model based on foreground information downsampling is constructed.
[0068] The point cloud and its corresponding image are input into the point-level feature fusion 3D point cloud target detection model based on foreground information downsampling for detection, and a 3D target bounding box is obtained.
[0069] The above steps are explained in detail below:
[0070] S1. Construct a baseline 3D target detection model and use point cloud data to complete 3D target detection.
[0071] Using a multimodal 3D point cloud dataset, the part of the 3D point cloud that corresponds to the image is sampled up to 20,000 points and used as the input of the model.
[0072] The baseline 3D target detection model structure is as follows: Figure 2 As shown, the specific method for constructing the baseline 3D target detection model is as follows:
[0073] Construct a point cloud feature extraction network, such as Figure 2As shown, the point cloud feature extraction network consists of three SA modules. Each SA module comprises a downsampling operation and a point cloud feature extraction module. The downsampling operation uses the Farthest Point Sampling (FPS) algorithm. The FPS algorithm flow is as follows:
[0074] (1) Randomly select a point P0 from all points as the initial point. S And add it to the sampling point set S, at which point S = {P0} S}
[0075] (2) Calculate the relationship between all points and P0 respectively. S The distances are used to form an N-dimensional array L, and the index of the maximum value in the array is taken as point P1. S And add it to S to get S = {P0} S P1 S}
[0076] (3) Calculate the relationship between all points and P1 respectively. S The distance from a point P, if a certain point P i With P1 S Distance d(P) i P1 S If P < L[i], then let L[i] = d(P) i P1 S Update the distances of all points that meet the conditions, and at this point, L is the distance from all points to the sampling point set S.
[0077] (4) Take the index of the maximum value in the array as point P2. S And add it to S to get S = {P0} S P1 S P2 S}
[0078] (5) Perform steps (3) and (4) on the newly added points in S until the number of points in S reaches the sampling requirement N1.
[0079] The downsampled point cloud and the input point cloud from the SA module are fed into the point cloud feature extraction module. The structure of the point cloud feature extraction module is as follows: Figure 3 As shown, firstly, spherical query clustering of the input point cloud of the SA module is performed at different scales, centered on the downsampled point cloud. That is, given a cluster center A, a cluster radius R, and a number of clusters M, all points M located in the spherical neighborhood of radius R of A are selected. i , i.e. d(M iFor points A ≤ R, sort the selected points by their distance from A, and take the top M points to obtain a cluster. If the number of points in the neighborhood is less than M, randomly select a point and repeat the selection until the number reaches M. Multiple clusters are obtained at each scale. These clusters at different scales are then fed into a weighted network layer for feature extraction and aggregation. This operation can be represented as:
[0080]
[0081] In the formula, g(·) is the max pooling function, which is a symmetric function; h(·) is a multilayer perceptron (MLP), which is equivalent to a 1×1 convolutional layer and is used for point-by-point feature extraction; r is also an MLP and is used to process the pooled features. The three functions combine to form the final symmetric function f(·). Using a symmetric function ensures that the output is independent of the input order. The aggregated features at different scales are then concatenated to reduce the dimensionality, and this feature is used as the output of the point cloud feature extraction module.
[0082] Construct a point cloud voting detection network. The detailed structure of the point cloud voting detection network is as follows: Figure 4 As shown, it is mainly divided into the following parts: (1) voting point generation module; (2) clustering and grouping module; (3) target box generation module.
[0083] In this embodiment, the 256 downsampled points output by the point cloud feature extraction network are used as seed points for the voting detection network, which can be represented as seeds=[x,f], where x∈R M×3 , f∈R M×C M represents the number of seed points. M×(3+C) seed points are input into the voting point generation module to generate voting points for their respective target centers. This module consists of a multi-layer MLP network with shared weights, predicting the voting point for each point individually, resulting in an output of size M×3. This output can be represented as Δx, which is the offset of the seed point's coordinates relative to the target center. Therefore, the final voting points are votes = [x + Δx, f]. The loss function of the voting point generation module is called the voting loss, and its expression is:
[0084]
[0085] In the formula, M pos M represents the number of target points among the seed points; pos In S i The value is 1 if it is the target point, otherwise it is 0; Δx i and These represent the predicted coordinate offset and the actual coordinate offset, respectively, from the voting point generation module; ||*|| represents the smoothL1 loss function.
[0086] After the voting process, points belonging to the same target will cluster near the target center. The clustering module uses ball query to cluster k voting point clusters. Finally, the voting point clusters are fed into the weight-sharing target bounding box generation module to aggregate features and generate target bounding boxes. The target bounding box generation module applies multi-layer MLP and max pooling layers to aggregate features for each voting cluster. Weights are shared between the network layers of each cluster, and the output size is k×C. out C out This refers to the number of predicted channels. In this embodiment, 256 voting point clusters are generated from 256 points. Each point cluster predicts one target box, resulting in a total of 256 target boxes. These are then subjected to 3D maximum suppression to generate the final predicted target boxes. The overall loss function of the voting detection network is as follows:
[0087] L = L vote-reg +L obj-cls +L dir-cls +L dir-reg +
[0088] L corner-reg +L size-reg +L center-reg +L sem-cls
[0089] In the formula, L vote-reg For the loss of votes, L obj-cls For the target foreground classification loss, L sem-cls Classification loss for the target category.
[0090] The voting detection network sets a 3D-sized anchor box for each category and assigns 12 evenly spaced directions from 0 to 360 degrees to the predicted target box. size-reg This refers to the regression loss for predicting the size deviation between the target bounding box and the anchor box, L. center-reg To predict the regression loss for the center of the target bounding box, L dir-cls With L dir-reg L represents the angle classification and regression loss for predicting the bounding box. corner-reg The regression loss is used to predict the target bounding box and the eight corner points of the true target bounding box.
[0091] Except for the target loss, all other losses are calculated only for the target point; background points are not included in the calculation. All regression losses in the total loss function are smoothL1 loss functions.
[0092] In the overall loss function, except for L obj-cls All classification losses are multi-class cross-entropy loss functions, and the expression for the cross-entropy loss function is:
[0093]
[0094] In the formula, C represents the number of sample categories in the classification task; p represents the true probability that a sample belongs to the i-th class, taking the value 1 when the sample belongs to the i-th class and 0 when it does not; i Let be the predicted probability that the sample belongs to the i-th class.
[0095] L obj-cls The cross-entropy loss function for binary classification is calculated using the following formula:
[0096] L obj-cls =-Y t logY-(1-Y t log(1-Y)
[0097] In the formula, Y t Y represents the true centrality of the point cloud, which is 0 when the point cloud is not within the true target bounding box.
[0098] All regression losses in the loss function are smoothL1 loss functions, and their expression is:
[0099]
[0100] In the formula, Let x be the value of the i-th dimension of the regressed vector. i Let be the value of the i-th dimension of the prediction vector.
[0101] S2. Introduce a hybrid downsampling method that incorporates image foreground information into the baseline 3D point cloud target detection model, construct a 3D point cloud target detection model based on foreground information downsampling, and improve the model's ability to detect small targets and overall detection accuracy.
[0102] A 3D point cloud dataset is used, and the multimodal data consisting of the sampled point cloud and the corresponding image is used as the input to the model.
[0103] The structure of the 3D point cloud target detection model based on foreground information downsampling is as follows: Figure 5 As shown, the specific method for constructing a 3D point cloud target detection model based on foreground information downsampling is as follows:
[0104] A hybrid downsampling method incorporating foreground information is introduced to replace the original farthest point sampling method. First, the YOLOX 2D object detection model is used to obtain the 2D object detection results in the input image. The relationship between the corresponding positions in the point cloud and the camera image can be simplified to be obtained from a perspective projection model, the basic representation of which is shown in the figure below. Figure 6 As shown, it consists of a viewpoint, a view plane, and a real point. B is the viewpoint, i.e., the camera's optical center, and A is a point in 3D space with coordinates (x, y, z) in the real-world coordinate system, which can be transformed into point A in the camera coordinate system using the camera extrinsic parameter matrix. C (x C ,y C ,zC This process can be represented as:
[0105]
[0106] In the formula, E is the camera extrinsic parameter matrix, which is obtained by combining the rotation matrix R and the offset matrix t, and its expression is:
[0107]
[0108] By A C Through perspective projection, we obtain the projected coordinates of the original spatial point A onto the viewing plane H, which is the intersection point A of the line of sight AB and H. H (x H ,y H Then convert these physical coordinates into the corresponding pixel coordinates A. P (u,v), this calculation process can be expressed as:
[0109]
[0110] In the formula, K is the intrinsic parameter matrix of the camera, and its expression is:
[0111]
[0112] In the formula, the unknown elements of matrix K are the camera's intrinsic parameters, which are fixed values and do not change with different scenes, while the extrinsic parameter matrix is calculated based on the camera's real-time orientation and other states. The conversion from point cloud world coordinates to corresponding pixel coordinates in the image can be completed using these intrinsic and extrinsic parameter matrices.
[0113] To retain more target points while maintaining a balance between positive and negative samples after downsampling the point cloud, this paper proposes a hybrid downsampling method, MS, assuming that the set of all points in the point cloud is P. total The target bounding box B output by the 2D target detection module i For (x) i ,y i ,w i ,h i Point P in the point cloud j The pixel coordinates are (u j ,v j If P j The point is located within the target box, that is:
[0114]
[0115] Then point P j Add to B i point set In this context, assuming a 2D object detection network obtains n predicted bounding boxes, this operation is performed on each bounding box to obtain the set of points within that box. It is permissible for the same point to exist in different sets. Then we can obtain... With P neg =P total -P obj These are the predicted foreground and background point sets, respectively. Each sampling uses different sampling methods for the points in these two sets. The foreground elements are relatively large and concentrated, while the background elements are more scattered. Therefore, P... obj The points in the frame are randomly downsampled frame by frame. For P... neg In downsampling, although some points are identified as background points, target points may still exist in this set due to the inevitable missed detections of 2D target detectors. However, missed targets are usually distant or small targets, with a relatively small number of target points, making it difficult to retain them using direct FPS sampling. To make it easier to select sparse target points during downsampling, we consider introducing feature distances from the point cloud for downsampling. Unlike most background points, target points have complex and diverse structural features; for example, the features of different parts of a car are also different, and the structural features of different categories of target points are very different. In contrast, many background points, such as walls and roads, have simple and uniform structures. Therefore, the feature distances between different categories of target points and between target points and background points are large, while the feature distances between background points are small. We use the feature vector distance d. f Distance d from the actual distance t The composite distance d ft FPS downsampling can retain more target points; this distance can be expressed as:
[0116] d ft =λd f +d t
[0117] In the formula, λ is the feature distance weight coefficient, which is set to 1 in this embodiment. This downsampling method is called FT-FPS. Assuming the number of downsampling points is N, the proposed hybrid downsampling process is as follows:
[0118] (1)Judgment Does the number of midpoints exceed If it exceeds S Bi The largest number of point clouds in China Perform random downsampling to the original number of points Then, loop through the process and perform this operation; if the quantity is less than or equal to... When P ends, this loop terminates. obj The number of midpoints is
[0119] (2) P neg Point downsampling in
[0120] (3) Obtain P total =P neg ∪P obj This will be output as the final set of downsampled points.
[0121] The above hybrid downsampling method can still preserve most targets and retain a certain number of target points even when the number of downsampled point clouds is small.
[0122] The point cloud feature extraction module is improved by introducing residual connections. The baseline model's point cloud feature extraction network is relatively wide but lacks depth. To enhance the network's feature extraction capability without introducing too many parameters, this embodiment adds three MLP layers after the original point cloud feature extraction module's concatenation and dimensionality reduction operation to further learn the point cloud neighborhood aggregation features. However, due to the vanishing gradient problem, directly adding more MLP layers does not improve model performance. Therefore, a residual module is used to alleviate the vanishing gradient problem while deepening the feature extraction network, enabling the network to more fully learn the neighborhood structure features of each point. The output feature of the residual block is H(x) = F(x) + x, where F(x) is the feature learned by the MLP layer, and x is the input feature directly fed in by the residual connection. The improved point cloud feature extraction module results are as follows. Figure 7 As shown, the depth downsampling feature extraction module (DSA) is obtained by combining the improved point cloud feature extraction module with the hybrid downsampling method.
[0123] A 3D centrality loss is introduced to replace the target loss in the baseline model. Although foreground seed points will try to get as close as possible to the target center after voting, there are inaccurate foreground seed points. Their votes may be within the target box, but they are far from the center. Votes closer to the target center produce more accurate predicted target boxes. Therefore, the predicted target boxes generated by votes far from the center should be suppressed. The 3D centrality loss is used to suppress the target boxes predicted by low-quality votes. The 3D centrality is defined as:
[0124]
[0125] In the formula, 3d-centerness * For three-dimensional centrality, l * r * t * d * f * b * These represent the distances from the voting point to the left, right, top, bottom, front, and back boundaries of the true 3D target bounding box, respectively. The 3D centrality loss is a binary classification cross-entropy loss function, calculated as follows:
[0126] Lctr-cls =-Y t logY-(1-Y t log(1-Y)
[0127] In the formula, L ctr-cls For the three-dimensional centrality loss, Y t Y is the true centrality of the point cloud, which is 0 when the point cloud is not in the true target box.
[0128] To verify the effectiveness of the downsampling method proposed in this invention, this embodiment tested the target retention rate when downsampling to 256 points using different downsampling methods on the outdoor KITTI validation set, including FPS, the FT-FPS proposed in this invention, and the hybrid downsampling method MS. If the point cloud in a target box is completely unretained after downsampling, it is considered that the target has not been retained. The results are shown in Table 1. Figure 8 A comparison chart of the effects of various downsampling methods is provided.
[0129] Table 1
[0130]
[0131] Through Table 1 and Figure 8 The comparison results show that the hybrid downsampling method proposed in this invention can better retain the target and obtain sufficient target points compared with the FPS method used in the baseline model.
[0132] This embodiment also uses numerical metrics combined with the visual effects of the images. The mean average precision (mAP) is used to compare the performance of 3D object detection, thus determining the model's object detection capability. To verify the effectiveness of the proposed method, ablation experiments were designed for several aspects, comparing the baseline model MSVNet_0, the "MSVNet_0 + hybrid downsampling" ablation model MSVNet_1, the "MSVNet_0 + hybrid downsampling + non-residual DSA" ablation model MSVNet_2, the "MSVNet_0 + hybrid downsampling + DSA" ablation model MSVNet_3, and the 3D point cloud object detection model MSVNet based on foreground information downsampling, in outdoor KITTI and indoor SUN tests. The detection results on the RGB-D validation set were used. During training, the dataset was repeated twice in each round, and each model was trained for 80 rounds. The experimental environment for each model was the same, and the average of 5 experiments for each model was taken as the result. The final experimental results are shown in Tables 2 and 3. Table 2 includes the AP and total mAP for cars, pedestrians and cyclists in the dataset under different difficulty levels. When calculating mAP, the IoU threshold for cars was 0.7, and the IoU threshold for the other two classes was 0.5. Table 3 includes the AP and total mAP for 10 classes of objects in the dataset. When calculating mAP, the IoU threshold for each class was 0.25.
[0133] Table 2
[0134]
[0135] Table 3
[0136]
[0137]
[0138] The comparison results in Tables 2 and 3 show that the baseline model, affected by downsampling, performs poorly in outdoor datasets with a small proportion of target points. Introducing a hybrid downsampling method significantly improves the detection performance for small targets. Further improvements to the point cloud feature extraction module by introducing residual connections mitigate the vanishing gradient phenomenon and enhance feature extraction capabilities. The introduction of 3D centrality loss further improves the model's detection accuracy. Figure 9 It can be seen that both models converge within 80 rounds, and the 3D point cloud target detection model based on foreground information downsampling has a lower final convergence loss value than the baseline model.
[0139] S3. Introduce a point-level interpolation pooling feature fusion module based on attention mechanism into the 3D point cloud target detection model based on foreground information downsampling, and construct a 3D point cloud target detection model based on foreground information downsampling point-level feature fusion to improve the model's ability to recognize objects with similar point cloud shapes and further improve the missed detection situation.
[0140] The structure of the 3D point cloud target detection model based on foreground information downsampling and point-level feature fusion is as follows: Figure 10 As shown, the specific method for constructing a 3D point cloud target detection model based on foreground information downsampling and point-level feature fusion is as follows:
[0141] Construct an attention-based point-level interpolation pooling feature fusion module, the structure of which is as follows: Figure 11 As shown, this invention establishes a correspondence between each point and image features to achieve point-by-point feature fusion, integrating image information without altering the point cloud feature structure. The correspondence between the point cloud and image features can be obtained through perspective projection. For a point P(x,y,z), the coordinates of P on the original pixel plane can be obtained. I (u,v), assuming the original image size is W×H and the feature map size is W'×H', the coordinates P in the feature map plane can be obtained by the following formula. F (x',y'):
[0142]
[0143] After obtaining the coordinates of the point cloud on the feature map, it is considered that there is not a good one-to-one correspondence between continuous point cloud coordinates and discrete feature map pixel blocks, especially for fine texture details. This invention uses bilinear interpolation to obtain the corresponding image features at any continuous coordinate position, achieving a more refined feature correspondence between the point cloud and the image. This method uses the feature values of the four nearest pixels adjacent to the interpolation point to calculate the feature value of that point.
[0144] Figure 12 A schematic diagram of bilinear interpolation is given, consisting of cubic interpolation. Assume the point to be interpolated on the feature map plane is P(x,y), and the interpolation is performed by two points Q below P. 11 (x1,y1),Q 21 The characteristic f(Q) of (x2,y1) 11 f(Q) 12 The feature f(x,y1) of R1(x,y1) is obtained by linear interpolation, and is generated from two points Q above P. 12 (x1,y2),Q 22 The characteristic f(Q) of (x2,y2) 12 f(Q) 22The feature f(x,y1) of R2(x,y2) is obtained through linear interpolation. Finally, the feature f(x,y1) of P(x,y) is obtained through linear interpolation of the features f(x,y1) of R1(x,y1) and R2(x,y2). The specific operation can be expressed as follows:
[0145]
[0146]
[0147]
[0148] Interpolation only yields the fine features of the feature map coordinates corresponding to a point, without effectively utilizing the image features of the corresponding neighborhood. Assuming that the pixel block corresponding to point P(x,y,z) in the feature map is p(i,j), the interpolation features f of P can be obtained from equation (4-2). B After (P), a 3×3 max pooling operation is used to aggregate the neighborhood features centered at p(i,j). The formula for this pooling operation is as follows:
[0149] f N (P) = Pool max ({f(a,b)|i-1≤a≤i+1,j-1≤b≤j+1})
[0150] The neighborhood features and interpolation features are added together by channel to obtain the image features f corresponding to P. I (P), ultimately f I The calculation method for (P) can be expressed as:
[0151] f I (P)=f N (P)+f B (P)
[0152] Image features may contain erroneous semantic and textural information due to factors such as occlusion or lighting variations. Receiving this information can mislead or interfere with 3D point cloud object detection. Therefore, an attention mechanism is introduced to allow each point to autonomously select the level of importance given to corresponding image features to alleviate this problem. This ensures that features containing meaningful semantic information receive high attention while suppressing features containing interfering information. Thus, a single point f can be... F (P) characteristics f P (P) The query in the attention mechanism corresponds to the image feature f I (P) serves as the key and value in the attention mechanism, and the attention weights are learned autonomously using a neural network.
[0153] like Figure 11 As shown, the corresponding image features f for point P are first obtained through interpolation and pooling operations. I(P), then the point-level image features f I (P) Features and the feature f of this point P (P) is input into the weight learning module to learn the corresponding weights w, and w is compared with f I (P) Multiplication yields the image features f after attention is applied. AI (P), ultimately f AI (P) and f P (P) Single-point characteristics of spliced ground to merged ground f F (P), then the feature fusion module enables each point in the point cloud to adaptively acquire corresponding image features, and the single-point fusion feature f F The formula for calculating (P) is:
[0154] f F (P)=f P (P)|wf I (P)
[0155] Figure 13 To understand the specific details of the single-point weight learning module, we first consider the input feature f. I (P) and f P (P) is applied to an MLP layer for further encoding, and then the two are added together by channel. The Tanh activation function, commonly used in binary classification models, is then used to compress this feature value to the interval [-1, 1]. The Tanh activation function is the hyperbolic tangent function, and its mathematical expression is:
[0156]
[0157] Finally, an MLP layer is used to reduce the feature channels to 1 to obtain the initial weights. The Sigmoid activation function is then used to compress the weights to the [0,1] interval to obtain the final single-point weight w. The Sigmoid activation function is commonly used in the output layer of binary classification models, and its mathematical expression is:
[0158]
[0159] The final weight calculation formula is as follows:
[0160] w = sigmoid(w1tanh(w2f)) P (P)+w3f I (P)))
[0161] In the formula, w1, w2 and w3 are the weights learned by each MLP layer, and each point shares the weights of each MLP layer when calculating the corresponding weights.
[0162] Constructing a multi-scale feature fusion network, the simplified structure of the image feature extraction network in the 3D point cloud target detection model based on foreground information downsampling is as follows: Figure 14As shown, this structure references the PANet (Path-Aggregation Network) model to form an image feature pyramid (PAFPN). In the entire image feature extraction network, the multi-scale feature maps output by PAFPN simultaneously possess semantic and textural detail information; therefore, this multi-scale feature map is chosen for fusion with point cloud features.
[0163] The point cloud feature extraction network has three DSA modules that sample the point cloud to 4096, 1024, and 256 points respectively. To better integrate image information, feature maps of different resolutions are correlated with the three downsampling stages, and feature fusion is performed using three feature fusion modules. Small-scale texture and semantic information are incorporated in the early stage of point cloud feature extraction, and large-scale texture and semantic information are incorporated in the later stage. This helps the point cloud feature extraction network to better learn the point cloud structure and semantic features at different scales. Furthermore, the early integration of image features can further widen the feature distance between point clouds, enabling FT-FPS to better preserve target points during sampling. The multi-scale feature dimension output by PAFPN is 256, while the feature dimensions output by the three point cloud downsampling feature extraction modules DSA1, DSA2, and DSA3 are 64, 128, and 256 respectively. To maintain channel balance during feature fusion, 1×1 convolutions are used before feature fusion to reduce the dimensions of N3 and N2 to 64 and 128 respectively. After the above operations, a multi-scale feature fusion network is constructed.
[0164] S4. A 3D point cloud target detection model based on foreground information downsampling and point-level feature fusion generates 3D target detection results.
[0165] The multimodal data consisting of the sampled point cloud and the corresponding image is input into the point-level feature fusion 3D point cloud target detection model based on foreground information downsampling, and the 3D target detection result is output.
[0166] To verify the effectiveness of the proposed method, this embodiment designed ablation experiments for several aspects, comparing the detection results of the 3D point cloud target detection model MSVNet based on foreground information downsampling, the ablation model PF-MSVNet_1 (MSVNet + direct feature fusion), the ablation model PF-MSVNet_2 (MSVNet + point-level feature fusion + feature interpolation pooling), the ablation model PF-MSVNet_3 (MSVNet + point-level feature fusion + feature interpolation pooling + weight learning module), and the 3D point cloud target detection model PF-MSVNet based on foreground information downsampling point-level feature fusion on the KITTI and SUN RGB-D validation sets. During training, the dataset was repeated twice in each round, and each model was trained for 80 rounds. The experimental environment for each model was the same, and the average of 5 experiments was taken as the result. The final experimental results are shown in Tables 4 and 5. Figure 15 The loss curves for the two models and the baseline model are given.
[0167] Table 4
[0168]
[0169] Table 5
[0170]
[0171] The comparison results in Tables 4 and 5 show that while direct fusion of image features can improve detection accuracy, the effect is limited. However, by using interpolation pooling, introducing an attention mechanism, and constructing a multi-scale feature fusion network, the detection accuracy can be further improved. Ultimately, the detection performance of the 3D point cloud target detection model based on foreground information downsampling and point-level feature fusion is significantly better than that of the 3D point cloud target detection model based on foreground information downsampling. Figure 15 It can be seen that all three models converge within 80 rounds, and the final convergence loss value of the 3D point cloud target detection model based on foreground information downsampling point-level feature fusion is lower.
[0172] Tables 6 and 7 present the comparison results of the point-level feature fusion 3D point cloud target detection model based on foreground information downsampling with other models on the KITTI validation set and the SUN RGB-D validation set.
[0173] Table 6
[0174]
[0175] Table 7
[0176]
[0177] The comparison results in Tables 6 and 7 show that, compared to other single-modal and multi-modal comparison models, the model in this embodiment can learn point cloud and image features well and effectively combine them, achieving the highest detection accuracy in both indoor and outdoor datasets, and exhibiting the best detection performance in almost all categories. The comparative experiments conclude that the model in this embodiment can better handle indoor and outdoor 3D target detection tasks than other comparison models.
[0178] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A point-level feature fusion method for 3D target detection based on foreground information downsampling, characterized in that, Includes the following steps: A baseline 3D point cloud target detection model is constructed, and a hybrid downsampling method incorporating image foreground information is introduced into the baseline 3D point cloud target detection model to construct a 3D point cloud target detection model based on foreground information downsampling. Based on the foreground information downsampling three-dimensional point cloud target detection model, combined with the attention mechanism-based point-level interpolation pooling feature fusion module, a point-level feature fusion three-dimensional point cloud target detection model based on foreground information downsampling is constructed. The method for constructing a 3D point cloud target detection model based on foreground information downsampling and point-level feature fusion includes: In the 3D point cloud target detection model based on foreground information downsampling, a point-level interpolation pooling feature fusion module based on attention mechanism is introduced. The downsampling points of the attached features and the downsampling results detected by the 2D target detection model are respectively input into the point-level interpolation pooling feature fusion module based on attention mechanism to obtain the downsampling points of the attached fusion features. The downsampling points of the attached fusion features are used to replace the downsampling points of the attached features to construct the 3D point cloud target detection model based on foreground information downsampling point-level feature fusion. The method for constructing the attention-based point-level interpolation pooling feature fusion module includes: The downsampling points of the attached features and the single-scale image features in the downsampling result are used as inputs. The coordinates of the downsampling points of the attached features in the corresponding image pixel plane are obtained through perspective projection relationship, and then converted into the coordinates of the feature map plane composed of the single-scale image features. Interpolation pooling is performed on the feature map plane based on the coordinates to obtain the image features corresponding to the downsampled points of the attached features; the attached point cloud features of the downsampled points of the attached features and the image features corresponding to the downsampled points of the attached features are fed into a single-point weight learning module to learn weights and obtain fused features; the attached point cloud features of the downsampled points of the attached features are replaced with the fused features to obtain the downsampled points of the attached fused features as output; the fused feature calculation expression is: in, This indicates a splicing operation. This represents the image features corresponding to the downsampling points with attached features. The attached point cloud features represent the downsampled points with additional features. Indicates weight; The interpolation pooling operation includes: The corresponding interpolated features of the pixel plane coordinate points are obtained by using bilinear interpolation in the feature map plane. Max pooling operation yields the corresponding pooling feature of the pixel plane coordinate point. The interpolated feature and the pooling feature are added channel by channel to obtain the image feature corresponding to the downsampled point of the attached feature. The expression for the image feature corresponding to the downsampled point of the attached feature is: In the formula, This represents the corresponding interpolation feature of a pixel plane coordinate point. The corresponding pooling feature representing the coordinates of a pixel plane; The point cloud and its corresponding image are input into the point-level feature fusion 3D point cloud target detection model based on foreground information downsampling for detection, and a 3D target bounding box is obtained.
2. The point-level feature fusion three-dimensional target detection method based on foreground information downsampling according to claim 1, characterized in that, The method for constructing a baseline 3D point cloud target detection model includes: constructing a point cloud feature extraction network and constructing a point cloud voting detection network, and constructing a baseline 3D point cloud target detection model based on the point cloud feature extraction network and the point cloud voting detection network. The point cloud feature extraction network includes three SA modules, each of which includes a downsampling operation module and a point cloud feature extraction module. The point cloud feature extraction network obtains downsampled points of attached features through downsampling operations and point cloud feature extraction.
3. The point-level feature fusion three-dimensional target detection method based on foreground information downsampling according to claim 2, characterized in that, The point cloud voting detection network includes a voting point generation module, a clustering and grouping module, and a target bounding box generation module; The workflow of the point cloud voting detection network is as follows: the downsampled points with attached features are used as seed points, voting prediction and center offset calculation are performed on the seed points to obtain voting points, the voting points are clustered by point-by-point ball query to obtain clusters, the clusters are predicted to obtain predicted target boxes, and the predicted target boxes are subjected to non-maximum suppression to obtain the final target boxes. The overall loss function of the point cloud voting detection network is: In the formula, For the loss of votes, Classify the loss for the target foreground. Classification loss for the target category, To predict the regression loss for the size deviation between the target frame and the anchor frame, To predict the regression loss for the center of the target box, To predict the angle classification loss of the target bounding box, To predict the regression loss of the target bounding box, The regression loss is used to predict the target bounding box and the eight corner points of the true target bounding box.
4. The point-level feature fusion three-dimensional target detection method based on foreground information downsampling according to claim 1, characterized in that, Methods for constructing 3D point cloud target detection models based on foreground information downsampling include: A hybrid downsampling method incorporating foreground information is adopted to replace the farthest point sampling method in the downsampling operation module. A residual connection block is introduced into the point cloud feature extraction module to obtain a new downsampling feature extraction module. The new downsampling feature extraction module replaces the SA module. Then, a three-dimensional centrality loss is introduced into the point cloud voting detection network to obtain the three-dimensional point cloud target detection model based on foreground information downsampling.
5. The point-level feature fusion three-dimensional target detection method based on foreground information downsampling according to claim 4, characterized in that, The hybrid downsampling method that incorporates foreground information of the image includes: The corresponding image is detected by a two-dimensional target detection model to obtain a two-dimensional detection result. The two-dimensional detection result is then segmented by perspective projection relationship to obtain a foreground point set and a background point set. The foreground point set is randomly downsampled to obtain the foreground point sampling result; the background point set is sampled based on the farthest point of the composite distance to obtain the background point sampling result; the downsampling result is obtained by combining the foreground point sampling result and the background point sampling result.
6. The point-level feature fusion three-dimensional target detection method based on foreground information downsampling according to claim 4, characterized in that, The three-dimensional centrality loss is expressed as: In the formula, For the three-dimensional centrality loss, The true centrality of the voting points. The predictive centrality corresponding to the voting points; The true centrality can be expressed as: in, , , , , , These represent the distances from the voting point to the left, right, top, bottom, front, and back boundaries of the actual 3D target bounding box, respectively.
7. The point-level feature fusion three-dimensional target detection method based on foreground information downsampling according to claim 1, characterized in that, The method for constructing the single-point weight learning module includes: The attached point cloud features of the downsampled points of the attached features and the corresponding image features are further encoded with an MLP layer to obtain encoded features. The encoded features are then added by channel to obtain summed features. The Tanh activation function is used to compress the summed features to a range. The compressed features are obtained, and an MLP layer is used to reduce the channels of the compressed features to 1 to obtain the initial weights. The sigmoid activation function is then used to compress the initial weights to a range to obtain the final weights. The learning expression for these weights is as follows: in, The weights learned by the first MLP layer, The weights learned by the second MLP layer. These are the weights learned by the third MLP layer.