3D indoor object detection method based on relational features

Abstract

The present invention discloses a three-dimensional indoor object detection method based on relational features. In the feature extraction stage, the relational features between points are deeply studied, and a fused convolutional pooling module is proposed, so that the model can capture more key features of the neighborhood in the point cloud scene; in the voting stage, inspired by the attention mechanism, the point cloud is divided into foreground points and background points according to the relationship between points and objects, and a voting weight module is designed to reduce the influence of background points on the detection results; then, by learning the spatial relationship between objects, after generating object proposals, the proposals are screened according to the object relationship, and object combinations with low probability are eliminated, thereby improving the object detection accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and artificial intelligence, and relates to a three-dimensional indoor object detection method based on relationship features. Background Art

[0002] As a crucial step in 3D reconstruction of indoor scenes, object detection requires fully learning the shape characteristics of objects. Leveraging advanced algorithms and neural networks, it uses bounding box methods to detect objects in the scene, thereby locating each object and extracting its semantic information. Therefore, point cloud object detection technology holds significant research significance and broad application prospects.

[0003] The most commonly used object detection network based on point cloud data is the VoteNet framework, proposed by Charles et al. in 2019. Since its introduction, this network has achieved state-of-the-art (SOTA) results on multiple datasets and across multiple evaluation metrics, attracting widespread attention from researchers in the point cloud field.

[0004] Although the VoteNet network achieved good results at the time, it still had many shortcomings. For example, the feature extraction part used the PointNet++ network, and its feature dimensionality reduction method did not well preserve key features. Secondly, when the seed points voted to the object center, it did not distinguish between foreground and background points, ignoring the impact of background points on the results. Finally, when generating object proposals, it ignored the specific relationships between objects and the characteristics of the combination. Summary of the Invention

[0005] The purpose of the present invention is to provide a three-dimensional indoor object detection method based on relational features, which effectively improves the accuracy of point cloud object detection.

[0006] The technical solution adopted by the present invention is a three-dimensional indoor object detection method based on relationship features, which is specifically implemented according to the following steps:

[0007] Step 1: Downsample and extract features of the input point cloud through the SA layer embedded with the fused convolutional pooling layer, and then use the FP layer to upsample and optimize the point cloud, finally obtaining multiple seed points with high-dimensional features;

[0008] Step 2: Use an MLP to let the seed points vote for the possible object center to obtain the initial voting points. Then, use a weight module to optimize the features of the voting points to obtain the final voting points.

[0009] In step 3, the SA layer is used to cluster the voting points to generate preliminary object proposals. The preliminary proposals are then input into the spatial relationship module, and the spatial relationship features between the proposals are concatenated with the proposal features. The proposed features are then input into the MLP for training to obtain optimized proposal features. Finally, the 3D NMS method is used to eliminate object proposals with low confidence, thus obtaining the final object bounding box and completing point cloud object detection.

[0010] The present invention is also characterized in that:

[0011] Step 1 is as follows:

[0012] Step 1.1, point cloud downsampling: Use the farthest point sampling method to sample the input point cloud data points;

[0013] Step 1.2, point cloud grouping: M point as the center, and then use the ball query method to find points, thus forming a local area;

[0014] Step 1.3, obtain local point cloud features: use MLP to extract features and upgrade the dimensions of each local area to obtain point cloud data with local grouping features;

[0015] Step 1.4, pooling operation: input the point cloud data obtained in step 1.3 into the average pooling layer and the maximum pooling layer respectively, and obtain the average value and maximum value of each local area feature respectively;

[0016] Step 1.5, convolution operation: The two results obtained in step 1.4 are first convolved with a layer Convolution is used to extract features, and then the data batch normalization operation is performed. Then the features are optimized using the rectified linear unit function, and then another layer is passed through The convolution process finally obtains two point clouds with deeper features extracted;

[0017] Step 1.6, feature aggregation: Add the features of the two point clouds obtained in step 1.5 to complete the point cloud downsampling and feature extraction of the SA1 layer;

[0018] Step 1.7, feature extraction for multiple SA layers: Use the point cloud obtained after each SA layer as the input for the next SA layer, and repeat steps 1.1-1.6 to complete the point cloud downsampling and feature extraction of SA2, SA3, and SA4 layers in sequence;

[0019] Step 1.8, point cloud upsampling and feature optimization: The point cloud and its features obtained after SA4 processing are upsampled and optimized through two consecutive FP layers. The points in the final point cloud are the seed points with high-dimensional features.

[0020] Step 2 is as follows:

[0021] Step 2.1, predict the object center point: input all seed points into the MLP, and each seed point votes for an object center point through the MLP. The MLP output result is the coordinate offset and feature offset of each seed point from the voted center point;

[0022] Step 2.2, obtain the initial voting point: add the coordinates and features of the original seed point to the coordinate offset and feature offset output by the MLP to obtain the coordinates and features of the initial voting point;

[0023] Step 2.3, calculate the weights of voting points and seed points: V and seed points and their features S The voting points are input into the weight function for processing, and different weights are assigned to each voting point and seed point according to the probability that they belong to the foreground point. Finally, the voting point weight is obtained. V and seed point weights Sw ;

[0024] Step 2.4, optimize voting point features: use voting point weights V and seed point weights Sw Check the characteristics of the polling station V Optimize to obtain the final voting features after weight module optimization, where the optimization function is:

[0025]

[0026] The weight function used in step 2.3 is:

[0027] (1)

[0028] (2).

[0029] Step 3 is as follows:

[0030] Step 3.1: Sampling at the obtained voting points by the farthest point sampling method points;

[0031] Step 3.2, with this H The ball query method is used to find neighbors near the center point and aggregate features to obtain preliminary object proposals with high-dimensional features;

[0032] Step 3.3, object support relationship judgment: support relationship refers to the vertical proximity, based on the relative height between the two object proposals and the xoy The relationship between the intersection of the projection on the plane and the set threshold is used to judge each candidate proposal. Whether it has a supporting relationship with other proposals;

[0033] Step 3.4, object attachment relationship judgment: The attachment relationship refers to the horizontal proximity, based on the horizontal distance between the two objects and the x Axis and y The relationship between the maximum intersection-over-union ratio of the projections in the two perpendicular directions of the axis and the set threshold is used to judge each candidate proposal. Whether it has an attachment relationship with other proposals;

[0034] Step 3.5, feature concatenation: concatenate the proposed features with the spatial relationship features, and then input them into the MLP for training to obtain proposals with spatial relationship features of scene objects;

[0035] In step 3.6, the object proposals obtained in step 3.5 are sorted by confidence, and the proposals with low confidence are filtered out. The remaining object proposals are the object bounding boxes predicted by the entire network.

[0036] Step 3.5 uses the cross entropy loss function to calculate the loss of the relationship module:

[0037] (4)

[0038] in Indicates the The results of the samples The true value of the label, Indicates the The results of the samples The predicted value of the label, Indicates shared samples, Indicates that each sample has tags.

[0039] The beneficial effects of the present invention are:

[0040] The present invention proposes a three-dimensional indoor object detection method based on relational features. In the feature extraction stage, this method deeply learns the relational features between points and proposes a fused convolutional pooling module, which enables the model to capture more key neighborhood features in point cloud scenes. In the voting stage, inspired by the attention mechanism, the point cloud is divided into foreground points and background points according to the relationship between points and objects, and a voting weight module is designed to reduce the influence of background points on the detection results. Then, by learning the spatial relationship between objects, after generating object proposals, the proposals are screened according to the object relationship and object combinations with low probability are eliminated, thereby improving the object detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is the seed point obtained after the original point cloud data is processed by step 1 of the present invention;

[0042] Figure 2 is the voting point obtained after the seed point obtained in the present invention is processed in step 2;

[0043] Figure 3 is the object bounding box obtained after processing in step 3 of the present invention;

[0044] Figure 4 This is the ground truth data of the indoor objects detected in Example 4. DETAILED DESCRIPTION

[0045] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] Example 1

[0047] This embodiment provides a three-dimensional indoor object detection method based on relationship features, which is specifically implemented according to the following steps:

[0048] Step 1: Downsample and extract features from the input point cloud through the SA layer (Set Abstraction Layer) embedded with the fused convolutional pooling layer. There are multiple SA layers, including SA1, SA2, SA3, and SA4. Then, the FP layer (Feature Propagation Layer) is used to upsample and optimize the point cloud features, ultimately obtaining multiple seed points with high-dimensional features.

[0049] Step 2: Use an MLP (Multi-Layer Perceptron) to let the seed points vote towards the possible object center to obtain the initial voting points. Then, use a weight module to optimize the features of the voting points to obtain the final voting points.

[0050] In step 3, the SA layer is used to cluster the voting points to generate preliminary object proposals. The preliminary proposals are then input into the spatial relationship module, and the spatial relationship features between the proposals are concatenated with the proposal features. The proposed features are then input into the MLP for training to obtain optimized proposal features. Finally, the 3D NMS (Non-Maximum Suppress) method is used to eliminate object proposals with low confidence, thus obtaining the final object bounding box and completing point cloud object detection.

[0051] Example 2

[0052] This embodiment provides a three-dimensional indoor object detection method based on relationship features. Based on the first embodiment, step 1 is specifically as follows:

[0053] Step 1.1, point cloud downsampling:

[0054] Use the farthest point sampling method (FPS) to sample the input point cloud data points;

[0055] Step 1.2, point cloud grouping: M Points are taken as the center, and then the ball query method is used to find points, thus forming a local area;

[0056] Step 1.3, obtain local point cloud features: use MLP to extract features and upgrade the dimensions of each local area to obtain point cloud data with local grouping features;

[0057] Step 1.4, pooling operation: input the point cloud data obtained in step 1.3 into the average pooling layer and the maximum pooling layer respectively, and obtain the average value and maximum value of each local area feature respectively;

[0058] Step 1.5, convolution operation: The two results obtained in step 1.4 are first convolved with a layer Convolution is used to extract features, and then batch normalization (BN) is performed on the data. Then, the features are optimized using the rectified linear unit function (ReLU). Finally, a layer of The convolution process finally obtains two point clouds with deeper features extracted;

[0059] Step 1.6, feature aggregation: Add the features of the two point clouds obtained in step 1.5 to complete the point cloud downsampling and feature extraction of the SA1 layer;

[0060] Step 1.7, feature extraction for multiple SA layers: Use the point cloud obtained after each SA layer as the input for the next SA layer, and repeat steps 1.1-1.6 to complete the point cloud downsampling and feature extraction of SA2, SA3, and SA4 layers in sequence;

[0061] Step 1.8, point cloud upsampling and feature optimization: The point cloud and its features obtained after SA4 layer processing are subjected to point cloud upsampling and feature optimization through two consecutive FP layers. The points in the final point cloud are as follows: Figure 1 As shown, it is the seed point with high-dimensional features.

[0062] Example 3

[0063] This embodiment provides a three-dimensional indoor object detection method based on relationship features. Based on embodiment 2, step 2 is specifically as follows:

[0064] Step 2.1, predict the object center point: input all seed points into the MLP, and each seed point votes for an object center point through the MLP. The MLP output result is the coordinate offset and feature offset of each seed point from the voted center point;

[0065] Step 2.2, obtain the initial voting point: add the coordinates and features of the original seed point to the coordinate offset and feature offset output by the MLP to obtain the coordinates and features of the initial voting point;

[0066] Step 2.3, calculate the weights of voting points and seed points: V and seed points and their features S The voting points are input into the weight function for processing, and different weights are assigned to each voting point and seed point according to the probability that they belong to the foreground point. Finally, the voting point weight is obtained. V and seed point weights Sw ;

[0067] The weight function used is:

[0068] (1)

[0069] (2).

[0070] Step 2.4, optimize voting point features: use voting point weights V and seed point weights Sw Check the characteristics of the polling station V Optimize to obtain the final voting features after weight module optimization, where the optimization function is:

[0071]

[0072] The optimized voting points are as follows Figure 2 shown.

[0073] Example 4

[0074] This embodiment provides a three-dimensional indoor object detection method based on relationship features. Based on the embodiment 3, step 3 is specifically as follows:

[0075] Step 3.1: Sampling at the obtained voting points by the farthest point sampling method points;

[0076] Step 3.2, with this H The ball query method is used to find neighbors near the center point and aggregate features to obtain preliminary object proposals with high-dimensional features;

[0077] Step 3.3, object support relationship judgment: support relationship refers to the vertical proximity, based on the relative height between the two object proposals and the xoy The relationship between the intersection of the projection on the plane and the set threshold is used to judge each candidate proposal. Whether it has a supporting relationship with other proposals;

[0078] Step 3.4, object attachment relationship judgment: The attachment relationship refers to the horizontal proximity, based on the horizontal distance between the two objects and the x Axis and y The relationship between the maximum intersection-over-union ratio of the projections in the two perpendicular directions of the axis and the set threshold is used to judge each candidate proposal. Whether it has an attachment relationship with other proposals;

[0079] Step 3.5, feature concatenation: concatenate the proposed features with the spatial relationship features, and then input them into the MLP for training to obtain proposals with spatial relationship features of scene objects;

[0080] The cross entropy loss function is used to calculate the loss of the relationship module:

[0081] (4)

[0082] in Indicates the The results of the samples The true value of the label, Indicates the The results of the samples The predicted value of the label, Indicates shared samples, Indicates that each sample has tags;

[0083] Step 3.6: Sort the object proposals obtained in step 3.5 by confidence and filter out the proposals with low confidence, such as Figure 3 As shown in , the remaining object proposals are the object bounding boxes predicted by the entire network.

[0084] like Figure 4 The following is the real data of indoor objects detected in Example 4. Figure 3 and Figure 4 It can be found that the three-dimensional indoor object detection method based on relationship features of the present invention solves the problems of insufficient key feature extraction ability of the VoteNet network, interference of background points on the voting results of seed points, and failure to consider the specific relationships and combination features between objects, effectively improving the accuracy of point cloud object detection.

Claims

1. A three-dimensional indoor object detection method based on relational features, characterized in that: Please follow the steps below to implement it: Step 1: Downsample and extract features of the input point cloud through the SA layer embedded with the fused convolutional pooling layer, and then use the FP layer to upsample and optimize the point cloud, finally obtaining multiple seed points with high-dimensional features; Step 2: Use an MLP to let the seed points vote for the possible object center to obtain the initial voting points. Then, use a weight module to optimize the features of the voting points to obtain the final voting points. The step 2 is specifically as follows: Step 2.1, predict the object center point: input all seed points into the MLP, and each seed point votes for an object center point through the MLP. The MLP output result is the coordinate offset and feature offset of each seed point from the voted center point; Step 2.2, obtain the initial voting point: add the coordinates and features of the original seed point to the coordinate offset and feature offset output by the MLP to obtain the coordinates and features of the initial voting point; Step 2.3, calculate voting point and seed point weights: the voting points and their features V and seed points and their features S The voting points are input into the weight function for processing, and different weights are assigned to each voting point and seed point according to the probability that they belong to the foreground point. Finally, the voting point weight is obtained. V w and seed point weights S w ; Step 2.4, optimize voting point features: use voting point weights V w and seed point weights S w Check the characteristics of the polling station V Optimize to obtain the final voting features after weight module optimization, where the optimization function is: In step 3, the SA layer is used to cluster the voting points to generate preliminary object proposals. The preliminary proposals are then input into the spatial relationship module, where the spatial relationship features between the proposals are concatenated with the proposal features. The proposed features are then input into the MLP for training to obtain optimized proposal features. Finally, the 3D NMS method is used to remove object proposals with low confidence, thus obtaining the final object bounding box and completing point cloud object detection. The step 3 is specifically as follows: Step 3.1, sample the voting points obtained by the farthest point sampling method H points; Step 3.2, with this H The ball query method is used to find neighbors near the center point and aggregate features to obtain H preliminary object proposals with high-dimensional features; Step 3.3, object support relationship judgment: support relationship refers to the vertical proximity, based on the relative height between the two object proposals and the xoy The relationship between the intersection of the projection on the plane and the set threshold is used to judge each candidate proposal. P i Whether it has a supporting relationship with other proposals; Step 3.4, object attachment relationship judgment: The attachment relationship refers to the horizontal proximity, based on the horizontal distance between the two objects and the x Axis and y The relationship between the maximum intersection-over-union ratio of the projections in the two perpendicular directions of the axis and the set threshold is used to judge each candidate proposal. P i Whether it has an attachment relationship with other proposals; Step 3.5, feature concatenation: concatenate the proposed features with the spatial relationship features, and then input them into the MLP for training to obtain proposals with spatial relationship features of scene objects; In step 3.6, the object proposals obtained in step 3.5 are sorted by confidence, and the proposals with low confidence are filtered out. The remaining object proposals are the object bounding boxes predicted by the entire network.

2. The three-dimensional indoor object detection method based on relationship features according to claim 1, characterized in that: Step 1 is as follows: Step 1.1, point cloud downsampling: Use the farthest point sampling method to sample the input point cloud data points; Step 1.2, point cloud grouping: M point as the center, and then use the ball query method to find points, thus forming a local area; Step 1.3, obtaining local point cloud features: Using MLP to extract features and increase dimensionality of each local area, point cloud data with local grouping features is obtained; Step 1.4, pooling operation: input the point cloud data obtained in step 1.3 into the average pooling layer and the maximum pooling layer respectively, and obtain the average value and maximum value of each local area feature respectively; Step 1.5, convolution operation: The two results obtained in step 1.4 are first convolved with a layer Convolution is used to extract features, and then the data batch normalization operation is performed. Then the features are optimized using the rectified linear unit function, and finally a layer of The convolution process finally obtains two point clouds with deeper features extracted; Step 1.6, feature aggregation: Add the features of the two point clouds obtained in step 1.5 to complete the point cloud downsampling and feature extraction of the SA1 layer; Step 1.7, feature extraction for multiple SA layers: Use the point cloud obtained after each SA layer as the input for the next SA layer, and repeat steps 1.1-1.6 to complete the point cloud downsampling and feature extraction of SA2, SA3, and SA4 layers in sequence; Step 1.8, point cloud upsampling and feature optimization: The point cloud and its features obtained after SA4 processing are upsampled and optimized through two consecutive FP layers. The points in the final point cloud are the seed points with high-dimensional features.

3. The three-dimensional indoor object detection method based on relationship features according to claim 1, characterized in that: The weight function used in step 2.3 is: （1） （2）。 4. The three-dimensional indoor object detection method based on relationship features according to claim 1, characterized in that: The step 3.5 uses the cross entropy loss function to calculate the loss of the spatial relationship module: in Indicates the The results of the samples The true value of the label, Indicates the The results of the samples The predicted value of the label, Indicates shared samples, Indicates that each sample has tags.

Images