A method and system for constructing a three-dimensional object detector

Abstract

The application relates to a three-dimensional object detector construction method and system, point feature, edge feature, surface feature and line feature of collected point cloud data are extracted, and N seed points are obtained; a first layer geometry perception module is used to learn four-channel weighted graphs of four types of features and to calculate weighted features, a first layer context perception module is used to construct the relationship between the seed points; three convolution layers are used to extract three kinds of geometric primitives; a second layer context perception module is used to establish the relationship between the features of the three kinds of geometric primitives, that is, the region blocks, and to obtain weighted features by using a second layer geometry perception module to learn three-channel weighted graphs of the three kinds of geometric primitives and to calculate the weighted features; a proposal generation module is used to obtain region proposals of objects, and a three-dimensional object detector construction task is completed. The application improves the precision of three-dimensional object detection and improves the multi-scale object detection precision of an indoor scene.

Description

TECHNICAL FIELD

[0001] The application relates to a three-dimensional object detector construction method and system, and belongs to the technical field of computer application. BACKGROUND

[0002] In recent years, research results on three-dimensional object detection have been widely applied in different fields, including identification and understanding of scenes in virtual reality and augmented reality, identification of types of obstacles and accurate positions and directions of obstacles in automatic driving, and replacement of humans by intelligent robots to complete specific tasks, such as inspection robots and medical robots, which have become a hot issue in the fields of augmented reality, automatic driving and robot navigation, and have brought great convenience to people's daily life and work. At present, there are many researches on three-dimensional object detection, but in actual application, there are still problems of object occlusion, truncation and robustness of surrounding dynamic environment, so that three-dimensional object detection is of great challenge, and the research has been a basic and classic problem in the fields of virtual reality and computer vision.

[0003] Three-dimensional object detection belongs to the task of target detection, uses RGB images, RGB-D depth images and laser point clouds, and outputs object categories and length, width, height and rotation angle in three-dimensional space, which is called three-dimensional target detection. With the rapid development of deep neural networks, target detection has developed from traditional detection methods based on manual features to detection methods based on deep learning. At present, the input data of three-dimensional target detection methods are based on cameras and radars, cameras save detailed semantic information, and images obtained by using ordinary cameras are generally called RGB images, images obtained by using depth cameras are called RGB-D images, and data obtained by data acquisition of laser radars is point cloud data. Because point cloud data has the characteristics of disorder, interaction between points and transformation invariance, it is often used as the input of three-dimensional target detection. According to different processing methods of point clouds, there are the following types of methods: methods based on voxelization or grid projection: the method based on grid projection projects the point cloud data into an aerial view, while the method based on voxelization converts the input data into a three-dimensional grid voxel. Then, standard two-dimensional or three-dimensional convolution operations are used to extract point cloud feature representations in the transformed space. However, these two projection strategies may lose important spatial and structural information. Methods based on original point clouds: directly starting from the original point cloud data, effective information is obtained through local feature extraction and other means. The pioneering exploratory work of this method is PointNet and PointNet++, based on these researches, many algorithm networks have been proposed to solve the problem of three-dimensional target detection.

[0004] For example, VoteNet network, which is an end-to-end deep voting method, and the subsequent H3DNet network further improves the performance of VoteNet, which uses PointNet++ to extract four types (point, edge, face, line) of features and three geometric primitives (three-dimensional bounding box center, three-dimensional bounding box face center, three-dimensional bounding box edge center), which greatly improves the three-dimensional target detection accuracy.

[0005] Most of the subsequent algorithms for directly processing raw point clouds are based on improvements of PointNet, PointNet++ and VoteNet. SUMMARY

[0006] The technical problem to be solved by the present application is to overcome the shortcomings of the prior art, provide a three-dimensional object detector construction method, which can obtain more effective geometric primitive feature information through two layers of geometric perception modules, learn the correlation between objects through two layers of context perception modules, and realize the combination of point cloud object geometric information and context information to solve the problem that the prior art such as H3DNet network loses part of the information when extracting features, and the prediction effect of the three-dimensional object bounding box is poor (for example, H3DNet can obtain an mAP@0.25 accuracy of 67.2 on the ScanNet dataset), and the present application can obtain a high three-dimensional object detection precision (the present application can obtain an mAP@0.25 accuracy of 71.1).

[0007] Technical solution of the present application:

[0008] The three-dimensional object detector construction method of the present application is realized based on two layers of context perception and geometric perception, and includes the following steps:

[0009] Step 1, label the input point cloud data to obtain point cloud data with label information; or convert the depth data collected in the indoor scene into point cloud data based on the camera intrinsic parameter, and give the point cloud data label information to obtain point cloud data with label information; perform feature extraction on the point cloud data with label information to obtain point f p , edge f e , face f s and line f l four types of features;

[0010] Step 2, fuse the four types of features, and input the fused features to the first layer of geometric perception module Ga1 to learn, to obtain four-channel weighted graph of four types of features The four-channel weighted graph is multiplied by the four types of features to obtain weighted features f g ;

[0011] Step 3, the weighted feature f g The relationship between the N seed points is constructed in the first layer of the context perception module Ca1, and the contact feature between the three-dimensional objects is obtained.

[0012] Step 4, the contact feature obtained in step 3 is input into three convolutional layers to extract three geometric primitive features

[0013] Step 5, the three geometric primitive features obtained in step 4 are input into the second layer of the context perception module Ca2 to construct the relationship between the features, i.e., the relationship between the region blocks; at the same time, the enhanced features are input into the second layer of the geometry perception module Ga2 to learn a three-channel geometric primitive weight map The three-channel weight map is multiplied with the enhanced three-channel geometric primitive features to obtain effective feature information

[0014] Step 6, the effective feature information is subjected to region proposal generation to generate a large number of M region proposals, and the large number of M region proposals generated are refined and classified to obtain the bounding box of the three-dimensional object in the scene, and the construction of the three-dimensional object detector is completed.

[0015] Further, the step 1: the depth data collected in the indoor scene is converted into point cloud data based on the camera intrinsic parameters, and the point cloud data is given a label information, to obtain the point cloud data with label information, which is implemented as:

[0016] The number of categories of three-dimensional objects in the indoor scene is determined, and the collected point cloud data is marked with a three-dimensional bounding box. The three-dimensional bounding box is given a label information according to the category of the three-dimensional object, and the point cloud data with label information is obtained.

[0017] Further, in step 1, the point cloud data extraction adopts a point cloud neural network structure, i.e., based on the PointNet++ network, the input point cloud data is subjected to four times of PointNet++ network extraction of point, edge, face, and line features, to obtain fp, f e , f s , and f l .

[0018] Further, the first layer of the geometry perception module Ga1 and the second layer of the geometry perception module Ga2 are implemented in the same way, and the specific implementation is as follows:

[0019] The point f p , edge f e , face f s , and line f l are subjected to the following operations to obtain ​

[0020]

[0021] will Split into four features according to the first dimension Then the following operations are performed:

[0022]

[0023] where, N is the number of seeds, c is the feature channel, conv(·) is a 1x1 convolution network, and finally a four-channel weighted graph is obtained through a Softmax layer, and cat(·) represents a fusion operation, represents an element-wise multiplication operation (Hadamard product).

[0024] Further, the first context awareness module Ca1 and the second layer context awareness module Ca2 have the same structure and are both composed of three network structures gap-ffn-sigmoid, wherein the first layer gap is a global average pooling layer for processing channels, the middle is a two-layer feedforward network ffn for learning feature relationships, and the last layer is a sigmoid layer for calculating feature weights.

[0025] The three-dimensional object detector construction system of the application comprises a point cloud acquisition module, a feature extraction module, a first layer geometric perception module, a first layer geometric perception module, a second layer context awareness and a second layer geometric perception module, and a region proposal generation module.

[0026] The three-dimensional object detector construction system of the application comprises a point cloud acquisition module, a feature extraction module, a first layer geometric perception module, a first layer context awareness module, a geometric primitive feature module, a second layer context awareness and a second layer geometric perception module, and a three-dimensional object detection module.

[0027] The point cloud acquisition module: labels the input point cloud data to obtain point cloud data with label information; or converts the depth data collected in an indoor scene into point cloud data based on the camera intrinsic parameters, and gives the point cloud data label information to obtain point cloud data with label information; extracts features from the point cloud data with label information to obtain point f p , edge f e , face fs and line f l Four types of features;

[0028] The feature extraction module: fuses the four types of features, and inputs the fused features into the first layer geometric perception module Ga1

[0029] The first layer geometric perception module learns the four types of features to obtain a four-channel weighted graph of the four types of features The four-channel weighting map is multiplied with the four types of features to obtain weighted features f The four-channel weighting map is multiplied with the four types of features to obtain weighted features f g ;

[0030] The first layer of context awareness module is based on the weighted features f g The relationship between the N seed points is constructed to obtain the contact features between the three-dimensional objects.

[0031] The geometric primitive feature module inputs the obtained contact features into three convolution layers to extract three geometric primitive features

[0032] The second layer of context awareness module is based on the three geometric primitive features The relationship between the features, i.e., the region blocks, is constructed, and the enhanced features are input into the second layer of geometric awareness module to learn a three-channel geometric primitive weighting map. The three-channel weighting map is multiplied with the three enhanced geometric primitive features to obtain effective feature information

[0033] The three-dimensional object detection module: the effective feature information is subjected to region proposal generation to generate a large number of M region proposals (the specific number of region proposals is a hyperparameter, and M is set to 512 in the present application), the large number of M region proposals are refined and classified to obtain the bounding boxes of the three-dimensional objects in the scene, and the construction of the three-dimensional object detector is completed.

[0034] The advantages of the present application compared with the prior art are as follows:

[0035] ​(1) The application can be applied to an indoor automatic navigation system. Although the objects detected in the indoor scene are separated, there is a correlation between the objects. Therefore, the application proposes two layers of context perception modules to learn the relationship between the objects. The application takes H3DNet network as the basis, adds two layers of geometric perception modules and two layers of context perception modules to obtain more effective feature information, integrates the added modules and the H3DNet network basic module, and optimizes the H3DNet network in an end-to-end manner, so as to achieve better three-dimensional object detection accuracy. Since the objects in the indoor scene captured by the depth camera lose part of the information, such as edge information and front information, directly using the geometric primitive features extracted by the classic point cloud detection network (such as H3DNet network) will lose part of the information, and the prediction effect of the three-dimensional object bounding box is poor. Therefore, the application proposes two layers of geometric perception modules to obtain more effective feature information. Since the objects in the indoor scene captured by the depth camera lose part of the information, such as edge information and front information, directly using the classic point cloud detection network (such as H3DNet network) will cause information loss, and the three-dimensional object detection effect is poor. Therefore, the application proposes two layers of geometric perception modules to obtain more effective feature information. The three-dimensional object detector based on two layers of context perception and geometric perception proposed by the application has higher three-dimensional object detection accuracy compared with some classic three-dimensional object detectors (such as MLCVNet network, 3DETR network and H3DNet network). Especially for the multi-scale scene such as indoor scene, it has good application prospect.

[0036] (2) The designed framework of the application is excellent, and the designed algorithm framework can be trained in an end-to-end manner and is easy to modify. The application is an end-to-end network structure, wherein the network module for feature extraction can be common or designed later, and the two layers of geometric perception modules proposed by the application can change the output dimension according to the number of extracted features. If the network module for feature extraction is designed later, the design will not have a great impact on the several modules proposed by the application. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 A three-dimensional object detector construction method flowchart of the application;

[0038] Figure 2 A structure framework diagram of the three-dimensional object detector construction system of the application. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical scheme and advantages of the application more clear and explicit, the application is further described in detail below in combination with specific embodiments and with reference to the drawings.

[0040] The application relates to a three-dimensional object detector construction method, which comprises the following steps: collecting point cloud scene data of an indoor scene through a consumer-grade depth camera, extracting point features f p , edge features f e , face features f s , line features f l , and simultaneously obtaining N seed points; using a first layer geometric perception module Ga1 to learn four-channel weighted graphs of four types of features and to calculate weighted features f g , and using a first layer context perception module Ca1 to construct the relationship between the seed points (small scale); secondly, three convolutional layers are used to extract the features of three geometric primitives (BB center, BB face center and BB edge center) (also called region blocks) respectively denoted as to predict three geometric constraints (namely, the center of an object bounding box (BB center), the face center of the object bounding box (BB face center) and the edge center of the object bounding box (BB edge center)); then, a second layer context perception module Ca2 is used to establish the relationship between the three geometric primitives (large scale) to obtain , and a second layer geometric perception module Ga2 is used to learn three-channel weighted graphs of the three geometric primitives and to calculate weighted features Finally, a proposal generation module is used to obtain object region proposals, and the three-dimensional object detector construction task is completed. The application can be applied to indoor object detection, and the two-layer context perception and geometric perception modules designed by the application can improve the accuracy of three-dimensional object detection, preferably improve the multi-scale object detection accuracy in an indoor scene, and effectively help indoor automatic navigation and other applications through the organic combination of the above modules.

[0041] The execution environment of the application adopts a 4.0GHZ central processor and a 128G byte memory core 4 nuclear computer, and is trained on an RTX P6000 GPU using a batch size of 8 Adam optimizer through PyTorch. Meanwhile, a program of an indoor scene three-dimensional object detector construction method based on a convolutional neural network is compiled by using languages such as Python and C++, and serves point cloud input data. Under the premise of allowing computer memory and display memory, the application can also be based on other execution environments, such as a GeForce GTX 1080TI environment.

[0042] Embodiment one

[0043] As shown in Figure 1 , the three-dimensional object detector construction method provided by the application comprises the following steps:

[0044] Step 1, the depth data collected by the indoor scene is converted into point cloud data based on the camera intrinsic parameter, the number of categories of objects in the indoor scene is determined, the collected point cloud data is marked with a three-dimensional bounding box, the three-dimensional bounding box is labeled with label information according to the object category, and the point cloud data with labels is obtained.

[0045] The above specifically includes:

[0046] The indoor scene is data collected by using a consumer-level depth camera or radar device, such as Microsoft Kinect, etc., to obtain depth data, and the captured depth data is converted into point cloud data P based on the camera intrinsic parameter.

[0047] The total number of categories of objects in the current indoor scene is counted, and the number of the total categories is the number of categories of objects in the indoor scene after the counting is completed. The collected point cloud data is marked with a three-dimensional bounding box by using artificial (which can also be assisted by a tool to perform rough labeling, and then manually refined and adjusted), and the three-dimensional bounding box is labeled with label information according to the object category.

[0048] Step 2, the feature extraction module is used to extract features from the input point cloud data with labels, and four types of features, i.e., point f p , edge f e , face f s and line f l are obtained, and N seed points are obtained.

[0049] In the above step 2, the feature extraction module is used to extract features from the input point cloud data with labels, and four types of features, i.e., point f p , edge f e , face f s and line f I are obtained, and N seed points are obtained. Specifically, the above step 2 specifically includes:

[0050] The point cloud data is input into four parallel PointNet++ feature extraction modules for feature extraction, and corresponding f p , edge f e , face f s and line f l features are obtained.

[0051] In this step, a classic point cloud feature extraction neural network structure is used, based on but not limited to PointNet++, to extract features from the input point cloud data. This embodiment of the invention uses standard PointNet++ as the encoder, and PointNet++ employs a standard encoder-decoder structure. Since this invention only uses the PointNet++ network as the feature extraction module, it only uses the PointNet++ encoder, discarding the last and subsequent layers of multi-level downsampling. The feature f extracted from the input point cloud data in this step is... p 、side f e 、face f s And the line fl.

[0052] Step 3: Integrate the four types of features, namely point f p 、side f e 、face f s and line f l The fused features are input into the first-layer geometric perception Ga1 module for learning, resulting in a four-channel weighted image of the four types of features. The four-channel weighted graph Multiplying the weighted feature f with the four types of features yields the weighted feature f. g This allows for the acquisition of more effective feature information;

[0053] Specifically, it includes:

[0054] Point f p 、edge fe、face f s and line f l Features are fused at the feature channel layer. For ease of representation, we set... Where N is the number of seed points and c is the number of feature channels, the four types of features are processed by channel and then fused together by the Cat layer (i.e., the connection layer) to form a new feature with 4 × c channels. Will The input is fed into a 1×1 convolutional network, where both the input and output channels have 4×c channels. Then f... cat The input is fed into a softmax network to learn a four-channel weighted image of four types of features. The process is as follows:

[0055]

[0056] Then calculate the four-channel weighted graph. Based on the channel, it can be divided into four characteristics. Divide into four equal parts, keeping the order of division unchanged and at point f. p 、side f e 、face f s and line f lThe features are multiplied, and the specific process is as follows:

[0057]

[0058] Wherein, conv(·) is a 1x1 convolution network, cat(·) represents a fusion operation, represents an element-wise multiplication operation (Hadamard product).

[0059] Step 4, f g The features are input into the first layer of the up-down perception module Ca1 to build the relationship between the N seed points, so as to capture more contact information between objects, including:

[0060] The first layer of the up-down perception module Ca1 is a gap-ffn-sigmoid three-layer network structure, wherein the first layer gap is a global average pooling layer for processing channels, the middle is a two-layer feedforward network ffn for learning feature relationships, and the last layer is a sigmoid layer for calculating feature weights. Through the module, the extracted features can capture more contact information between objects.

[0061] The specific implementation is as follows: An input global average pooling layer gap obtains a Then the feature is input into the subsequent ffn two-layer feedforward network. The two-layer feedforward network does not change the feature dimension of the input and output. In the present application, the output dimension of the first layer is 32, and the output dimension of the second layer is 1. Finally, the obtained feature is input into the sigmoid layer for calculating the feature weight Each element of the feature weight represents the relationship between the N seed points.

[0062] The feature f g is multiplied element by element with the feature weight to obtain f g ', obtaining the weighted feature between the N seed points to obtain more effective feature information.

[0063] Step 5, input the feature information obtained in step 4 into three convolution layers to extract three geometric primitive features The module is composed of three parallel convolution networks, each of which is composed of a nonlinear activation layer ReLU, a batch normalization layer and a 1x1 convolution layer. f g ' is input into the three parallel convolution networks to extract three geometric primitive features.

[0064] Step 6, input the three geometric primitive features The input is fed into the second-layer context-aware module Ca2 to construct relationships between features (i.e., region blocks) to enhance feature representation capabilities; simultaneously, the enhanced features are fed into the second-layer geometry-aware module Ga2 to learn a three-channel geometric primitive weight graph. The three-channel weight map is multiplied with the enhanced three geometric primitive features to obtain more effective feature information.

[0065] The second-layer context-aware module Ca2 consists of a three-layer network structure: a gap2-ffn2-sigmoid2. The first layer, gap1, is a global average pooling layer that processes channels. The middle layer is a two-layer feedforward network ffn2 that learns feature relationships. The last layer is a sigmoid2 layer that calculates feature weights. This module enables the extracted features to capture more information about the relationships between objects.

[0066] The specific implementation is as follows: Each input is a global average pooling layer gap2 to obtain for The operation is exactly the same.

[0067] by To explain, The features are input into the subsequent two-layer ffn2 feedforward network. These two layers do not change the feature dimensions of the input and output. In this invention, the output dimension of the first layer is 128, and the output dimension of the second layer is 1. Finally, the obtained features are input into the sigmoid2 layer to calculate the feature weights. Each element of this feature weight represents The relationship between feature elements.

[0068] Features With feature weights Element-wise multiplication yields more efficient features. about Operation and The operation process is exactly the same.

[0069] Will Characteristics were obtained from Ca2. These features are then input into the second-layer geometry sensing module Ga2, as follows:

[0070] Will Features are fused at the feature channel layer. For ease of representation, we set... (Where N' is the number of feature points, c' is the number of feature channels), the four types of features are passed through the Cat layer, i.e. the connection layer, to fuse the four features together to form a new feature with a channel number of 3xc' Input into a 1x1 convolutional network, the input and output of the convolution have a channel number of 3xc', and then f c at Input into a softmax network to learn a three-channel weighting map of the three types of features, the specific process is as follows:

[0071]

[0072] Then the calculated three-channel weighting map is split into three equal parts according to the channel level, and the split order is kept unchanged and The features are multiplied,

[0073] Split into three types of features according to the channel Then the following calculation is performed:

[0074]

[0075] Where conv b (·) is a 1x1 convolutional network, and finally a four-channel weighting map is obtained through a Softmax layer, cat(·) represents a fusion operation, Represents an element-wise multiplication operation (Hadamard product).

[0076] Step 7, input Into the region proposal generation module to obtain a large number of M region proposals, and finally through classification and refinement to obtain the bounding box of the three-dimensional object in the scene, completing the construction of the three-dimensional object detector.

[0077] The region proposal generation module is from H3DNet, which uses the SA layer (setabstraction module) in the PointNet++ network to sample the input features using the farthest point sampling method to obtain k sampling points, and then considers all points within a sphere with a radius R centered at each sampling point as a group, and then votes on all points in the group to generate a large number of region proposals. According to the voting results, the loss is calculated, and finally three-dimensional non-maximum suppression (NonMaximum Suppression) is used to refine and classify the large number of region proposals generated to obtain the final three-dimensional object detection result, completing the construction of the three-dimensional object detector.

[0078] Embodiment two

[0079] ​​​like Figure 2 As shown, this embodiment of the invention provides a three-dimensional object detector construction system, including a point cloud acquisition module 71, a feature extraction module 72, a first-layer geometry perception module 73, a first-layer context perception module 74, a geometric primitive feature module 75, a second-layer context perception module and a second-layer geometry perception module 76, and a region proposal generation module 77.

[0080] Point cloud acquisition module 71: Labels the input point cloud data to obtain point cloud data with label information; or converts depth data acquired from an indoor scene into point cloud data based on camera intrinsic parameters, and assigns label information to the point cloud data to obtain point cloud data with label information; performs feature extraction on the point cloud data with label information to obtain point f. p 、side f e 、face f s and line f l Four types of characteristics;

[0081] Feature extraction module 72: fuses the four types of features and inputs the fused features into the first layer geometric perception module Ga1;

[0082] The first-layer context-aware module 73 provides a four-channel weighted graph of four types of features. The four-channel weighted graph Multiplying the weighted feature f with the four types of features yields the weighted feature f. g ;

[0083] The first-layer geometric perception module 74 is based on weighted features f. g Construct the relationship between N seed points to obtain the connection features between 3D objects;

[0084] The geometric primitive feature module 75 inputs the obtained connection features into three convolutional layers to extract three types of geometric primitive features.

[0085] The second-layer context awareness and the second-layer geometry awareness module 76 are based on three geometric primitive features. Construct features, i.e., the relationships between regions; simultaneously, input the enhanced features into the second-layer geometry perception module to learn a three-channel geometric primitive weight graph. The effective feature information is obtained by multiplying the three-channel weight map with the enhanced three geometric primitive features.

[0086] 3D Object Detection Module 77: For valid feature information Region proposals are generated to produce a large number of M region proposals. These M region proposals are then refined and classified to obtain the bounding boxes of 3D objects in the scene, thus completing the construction of the 3D object detector.

[0087] The application can be applied in related applications in indoor scenes, especially object detection tasks. Through effective object detection of the currently captured scene, indoor robot navigation and other related applications can be effectively assisted. Compared with some existing classic three-dimensional object detectors (such as MLCVNet network, 3DETR network, H3DNet network), the three-dimensional object detector based on two-layer context perception and geometric perception proposed in the application achieves higher three-dimensional object detection accuracy. The specific comparison is shown in the following table.

[0088] Method mAP@0.25 MLCVNet network 64.5 3DETR network 65.5 H3DNet network 67.2 The present invention 71.1

[0089] The above is only a preferred embodiment of the present application, not other forms of limitations, any skilled in the art can use the above disclosed technical content to change or modify as equivalent embodiments applied to other fields, but any simple modification, equivalent change and modification of the above embodiments without departing from the technical solution content of the present application, according to the technical essence of the present application, still belongs to the present application.

Claims

1. A method for constructing a three-dimensional object detector, characterized in that: Based on two layers of context awareness and geometry awareness, the implementation includes the following steps: Step 1: Label the input point cloud data to obtain point cloud data with label information; or convert the depth data collected from the indoor scene into point cloud data based on camera intrinsic parameters, and assign label information to the point cloud data to obtain point cloud data with label information; perform feature extraction on the point cloud data with label information to obtain point... ,side ,noodle and lines Four types of characteristics; Step 2: Fuse the four types of features and input the fused features into the first-layer geometric perception module. Learning yields four-channel weighted graphs of four types of features. The four-channel weighted graph Multiply by the four types of features to obtain the weighted features. ; Step 3: Weighted features Input first-level context-aware module China Construction The relationship between seed points is used to obtain the connection features between three-dimensional objects; Step 4: Input the connection features obtained in Step 3 into three convolutional layers to extract three types of geometric primitive features. ; Step 5: Combine the three geometric primitive features obtained in Step 4. Input to the second-layer context-aware module The system constructs relationships between features, i.e., regions; simultaneously, the enhanced features are input into the second-layer geometry perception module. Learning a three-channel geometric primitive weight graph The effective feature information is obtained by multiplying the three-channel weight map with the enhanced three geometric primitive features. ; Step 6: Analyze the effective feature information. Region proposals are generated to produce M region proposals. The M region proposals are then refined and classified to obtain the bounding boxes of 3D objects in the scene, thus completing the construction of the 3D object detector.

2. The method for constructing a three-dimensional object detector according to claim 1, characterized in that, Step 1: Converting the depth data collected from the indoor scene into point cloud data based on camera intrinsic parameters, and assigning label information to the point cloud data to obtain labeled point cloud data is implemented as follows: The number of categories of 3D objects in the indoor scene is determined, and 3D bounding boxes are used to label the collected point cloud data. The 3D bounding box labels are given label information according to the category of the 3D object, resulting in point cloud data with label information.

3. The method for constructing a three-dimensional object detector according to claim 1, characterized in that, In step 1, point cloud data extraction employs a point cloud neural network structure, specifically based on the PointNet++ network. The PointNet++ network is used to extract features from the input point cloud data four times: points, edges, surfaces, and lines. , , as well as .

4. The method for constructing a three-dimensional object detector according to claim 1, characterized in that: First layer geometric perception module Second layer geometric perception module The implementation is the same, and the specific implementation is as follows: Point ,side ,noodle and lines Perform the following operations to obtain : （1） Will Based on the first dimension, it can be divided into four features. Then perform the following operations: （2） in, , It is the number of seeds. It is a feature channel. It is The convolutional network is then passed through a softmax layer to obtain a four-channel weighted image. The fusion operation is represented. This represents the element-wise product operation (Hadamard product).

5. The method for constructing a three-dimensional object detector according to claim 1, characterized in that: First layer context-aware module Second-layer context-aware module They have the same structure, both consisting of a three-layer network. Composition, of which the first layer It consists of a global average pooling layer that processes channels, and two feedforward layers in the middle that learn the relationships between features. The last layer calculates the feature weights. layer.

6. A three-dimensional object detector construction system, characterized in that... include: Point cloud acquisition module, feature extraction module, first-layer geometry perception module, first-layer context perception module, geometric primitive feature module, second-layer context perception and second-layer geometry perception module, 3D object detection module; Point cloud acquisition module: Labels the input point cloud data to obtain point cloud data with label information; Alternatively, depth data collected from indoor scenes can be converted into point cloud data based on camera intrinsic parameters, and then labeled to obtain labeled point cloud data; feature extraction can be performed on the labeled point cloud data to obtain point... ,side ,noodle and lines Four types of characteristics; Feature extraction module: fuses the four types of features and inputs the fused features into the first-layer geometric perception module. ; The first-layer geometric perception module learns four types of features and obtains a four-channel weighted image of the four types of features. The four-channel weighted graph Multiply by the four types of features to obtain the weighted features. ; The first-layer context-aware module is based on weighted features. Build The relationship between seed points is used to obtain the connection features between three-dimensional objects; The geometric primitive feature module inputs the obtained connection features into three convolutional layers to extract three types of geometric primitive features. ; The second-layer context awareness and the second-layer geometry awareness module are based on three geometric primitive features. Constructing features, i.e., the relationships between regions; simultaneously, inputting the enhanced features into the second-layer geometry perception module to learn a three-channel geometric primitive weight graph. The effective feature information is obtained by multiplying the three-channel weight map with the enhanced three geometric primitive features. ; 3D object detection module: for valid feature information Region proposals are generated to produce M region proposals. The M region proposals are then refined and classified to obtain the bounding boxes of 3D objects in the scene, thus completing the construction of the 3D object detector.

Images