A three-dimensional object detection method based on RGBD data and a view frustum

Abstract

The application discloses a three-dimensional target detection method based on RGBD data and a view cone, and comprises the following steps: acquiring a feature image of a scene RGB image according to the scene RGB image; inputting the feature image of the scene RGB image into a region generation network to acquire a bounding box of a to-be-detected object and a category vector of the to-be-detected object; acquiring a 3D view cone point cloud according to the bounding box of the to-be-detected object and a depth image of the scene; inputting the position coordinates of points in the 3D view cone point cloud and the category vector of the to-be-detected object into a PointNet network with an attention mechanism to acquire the center point coordinates of the to-be-detected object; and further acquiring the parameters of a 3D bounding box of the to-be-detected object to detect the three-dimensional to-be-detected object. The application simultaneously considers the plane information and the depth information of a scene, improves the detection precision, and makes the detection result more accurate while ensuring the efficiency.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and specifically relates to a three-dimensional target detection method based on RGBD data and view frustum. Background Technology

[0002] The application of 3D object detection algorithms in AR requires the addition of scene recognition and understanding. In order for intelligent robots in indoor scenes to perform tasks such as object grasping, they need to know the location and category information of objects. For example, if you want the robot to take a water cup to a certain place, it must know the indoor environment. For example, the water cup is on the table, and there is a coffee table between the robot and the table. Only by understanding these things can the robot design a route to avoid the coffee table and get the water cup. Or, if you want the robot to put dirty clothes into the washing machine, the robot also needs to see where the dirty clothes are and where the washing machine is.

[0003] Existing 3D target detection technologies cannot achieve both high detection efficiency and high detection accuracy when dealing with target object detection in complex indoor scenes, resulting in poor detection performance for the target objects to be detected. Summary of the Invention

[0004] This invention provides a three-dimensional target detection method based on RGBD data and view frustum to overcome the above-mentioned technical problems.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows:

[0006] A three-dimensional target detection method based on RGBD data and view frustum includes the following steps:

[0007] S1: Obtain the RGB-D image of the scene, wherein the RGB-D image of the scene includes the RGB image of the scene and the depth image of the scene;

[0008] S2: Based on the RGB image of the scene, obtain the feature image of the RGB image of the scene;

[0009] S3: Input the feature image of the RGB image of the scene into the region generation network to obtain the bounding box of the object to be detected and the category vector of the object to be detected;

[0010] S4: Based on the bounding box of the object to be detected and the depth image of the scene, obtain a 2D candidate region, and then obtain a 3D frustum point cloud based on the 2D candidate region, and then obtain the position coordinates of the points in the 3D frustum point cloud.

[0011] S5: Input the position coordinates of the points in the 3D frustum point cloud and the category vector of the object to be detected into the PointNet network with an attention mechanism to obtain the center point coordinates of the object to be detected;

[0012] S6: Based on the center point coordinates of the object to be detected, obtain the parameters of the 3D bounding box of the object to be detected in order to detect the three-dimensional object.

[0013] The parameters of the 3D bounding box of the object to be detected include the center coordinates, length, width, height, and heading angle of the 3D bounding box of the object to be detected.

[0014] Furthermore, the PointNet network with the added attention mechanism includes: an input module, an encoding unit, a decoding unit, and an output module;

[0015] The input module is used to input the position coordinates of points in the 3D frustum point cloud;

[0016] The encoding unit includes an upsampling convolutional layer, a first SENet convolutional attention module, and a max pooling layer.

[0017] The upsampling convolutional layer is connected to the input module and is used to obtain the initial center position coordinates of the object to be detected.

[0018] The first SENet convolutional attention module is connected to the upsampled convolutional layer to obtain the weights of points in the 3D frustum point cloud based on the initial center position coordinates of the object to be detected.

[0019] The max pooling layer is connected to the decoding unit to obtain the weighted features of the points in the 3D frustum point cloud based on the weights of the points in the 3D frustum point cloud.

[0020] The decoding unit includes a downsampling convolutional layer and a second SENet convolutional attention module;

[0021] The downsampling convolutional layer is connected to the max pooling layer and is used to obtain the residual between the center point coordinates of the object to be detected and the initial center position of the object to be detected, based on the weighted features of the points in the 3D frustum point cloud and the category vector of the object to be detected.

[0022] The second SENet convolutional attention module is connected to the downsampling convolutional layer and is used to obtain the center point coordinates of the object to be detected based on the residual between the center point coordinates of the object to be detected and the initial center position of the object to be detected.

[0023] The output module is connected to the second SENet convolutional attention module and is used to output the center point coordinates of the object to be detected.

[0024] Furthermore, the method for obtaining the center point coordinates of the object to be detected is as follows:

[0025] S51: Input the position coordinates of points in the 3D frustum point cloud into the PointNet network with an attention mechanism to obtain the initial center position coordinates (x', y', z') of the object to be detected.

[0026] S52: Obtain the distance d between the position coordinates of a point in the 3D frustum point cloud and the initial center position coordinates of the object to be detected. i ; to obtain the weight of the i-th point in the 3D view frustum point cloud;

[0027] d i =p i -(x',y',z')

[0028] Where, p i Let (x', y', z') be the position coordinates of the i-th point in the 3D frustum point cloud; (x', y', z') be the initial center position coordinates of the object to be detected.

[0029] have to:

[0030] w i =Sigmoid(d i )

[0031] In the formula, w i represents the weight of the i-th point in the 3D frustum point cloud; Sigmoid(·) is a non-linear action function used to obtain the weight of the i-th point in the 3D frustum point cloud.

[0032] S53: Obtain the features of the i-th point in the 3D frustum point cloud. And perform weighted summation to obtain the weighted features of the i-th point in the 3D view frustum point cloud:

[0033]

[0034] In the formula: The weighted feature of the i-th point in the 3D view frustum point cloud; The feature of the i-th point in the 3D view frustum point cloud;

[0035] S54: Obtain the coordinates of the center point of the object to be detected;

[0036]

[0037]

[0038] In the formula: (x,y,z) are the coordinates of the center point of the object to be detected. This is the residual between the center point coordinates of the object to be detected and the initial center position of the object. k is the category vector of the object to be detected; concat(·) is a function that fuses the weighted features of the i-th point in the 3D frustum point cloud with the category vector of the object to be detected.

[0039] Furthermore, the method for obtaining the parameters of the 3D bounding box of the object to be detected is as follows:

[0040] S61: Input the center point coordinates of the object to be detected into the template supervision network T-Net to obtain the true center point coordinates of the object to be detected;

[0041] S62: Input the coordinates of the true center point of the object to be detected and fine-tuned into the 3D bounding box regression network to obtain the parameters of the 3D bounding box of the object to be detected.

[0042] Furthermore, the total loss function of the PointNet network with added attention mechanism, the template supervision network T-Net, and the 3D bounding box regression network is obtained as follows:

[0043] L multi-task =L mask +L bbox =L mask +λ(L c1-reg +L c2-reg +L h-cls 10L h-reg 10L s-cls 10L s-reg +λL corner )

[0044] In the formula: L multi-task The total loss function for the PointNet network with added attention mechanism, template supervision network, and 3D bounding box regression network; L mask The loss function for the PointNet network with an added attention mechanism; L bbox The loss functions for the template-supervised network and the 3D bounding box regression network include the center position regression loss L generated by the template-supervised network. c1-reg And 3D bounding box regression network regression loss L c2-reg L h-cls For heading angle classification loss; L h-reg L is the heading angle regression loss; λ is the hyperparameter; corner L refers to the sum of the losses of the vertices of the 3D bounding box of the object to be detected; s-cls The classification loss is the size of the 3D bounding box of the object to be detected; L s-reg The regression loss is the size of the 3D bounding box of the object to be detected.

[0045] Beneficial effects: The three-dimensional target detection method based on RGBD data and view frustum of the present invention processes the RGB-D image based on the scene, while taking into account the planar information and depth information of the scene. It uses a PointNet network with an attention mechanism to detect the center point coordinates of the object to be detected, thereby improving the detection accuracy and efficiency. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a flowchart of the three-dimensional target detection method based on RGBD data and view frustum of the present invention;

[0048] Figure 2 This is a structural diagram of the overall framework for three-dimensional target detection in an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram of the PointNet network structure with an added attention mechanism in an embodiment of the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] This embodiment provides a three-dimensional target detection method based on RGBD data and view frustum, such as Figure 1 This paper relates to the field of 3D target detection technology in complex indoor scenes, specifically a 3D target detection method based on RGB-D data. It includes the following steps:

[0052] S1: Obtain the RGB-D image of the scene, which includes the RGB image of the scene and the depth image of the scene; wherein, the RGB image of the scene and the depth image of the scene can be obtained simultaneously when the RGB-D image of the scene is obtained, which is common knowledge in the field.

[0053] S2: Use a two-dimensional object detection algorithm to extract features from the RGB image of the scene, obtain the feature image of the RGB image of the scene, and save it;

[0054] S3: Input the feature image of the RGB image of the scene into the Region Proposal Network (RPN) to obtain the bounding boxes and category vectors of the objects to be detected in the scene; specifically, the bounding boxes of the objects to be detected in the scene are 2D bounding boxes.

[0055] S4: Based on the bounding box of the object to be detected and the depth image of the scene, obtain a 2D candidate region, and then obtain a 3D frustum point cloud based on the 2D candidate region, and then obtain the position coordinates of the points in the 3D frustum point cloud.

[0056] Specifically, based on the bounding box of the object to be detected and the depth image of the corresponding scene, a 2D candidate region is obtained; the 2D candidate region is transformed into a 3D frustum region using a camera projection matrix; all points within the 3D frustum region are collected to form 3D frustum point cloud data, thus transforming the 2D candidate region into a 3D frustum point cloud; wherein, the camera projection matrix is ​​an internal parameter of the camera, which is known when the RGB-D image of the scene is acquired by the camera; the method of transforming the 2D candidate region into a 3D frustum region is existing technology in the field and will not be described in detail here.

[0057] Specifically, in the 3D frustum point cloud data generation process, the bounding boxes of the objects to be detected in the scene are mapped onto the 3D frustum point cloud using a known projection matrix, and the corresponding 3D frustum point cloud data is extracted. Each extracted frustum point cloud data has a unique orientation in the camera coordinate system. To facilitate data processing, the coordinate system of the 3D frustum point cloud data needs to be converted from the camera coordinate system to the frustum coordinate system, and the centerline of the frustum point cloud is rotated to a position orthogonal to the scene's RGB image plane to normalize the frustum.

[0058] S5: Input the position coordinates of the points in the 3D frustum point cloud and the category vector of the object to be detected into the PointNet network with an attention mechanism to segment the 3D frustum point cloud; to obtain the center point coordinates of the object to be detected.

[0059] Specifically, this involves instance segmentation of a 3D frustum point cloud to obtain the target point cloud. The purpose of 3D instance segmentation is to segment the target point cloud (the point cloud representation of the object to be detected), therefore, the performance of the segmentation network directly affects the detection result of the final regressed 3D bounding box. In this embodiment, to improve the feature representation capability of the convolutional layers in the segmentation network, a PointNet network with an added attention mechanism is used for 3D instance segmentation. This considers the correlation between the positions of each point in the 3D frustum point cloud, resulting in better fine-grained segmentation. The specific method for 3D instance segmentation is as follows: using a PointNet network with an added attention mechanism, that is, fusing the 3D instance segmentation network PointNet with the SENet attention mechanism to improve the feature representation capability of the convolutional layers and segment the target point cloud information of interest. The 3D instance segmentation network PointNet is a point cloud deep learning network used for 3D classification and segmentation.

[0060] The instance segmentation network structure diagram in this scheme is as follows: Figure 3 As shown.

[0061] The PointNet network model incorporating an attention mechanism uses a PointNet network combined with two SENet convolutional attention modules as its network structure. This scheme concatenates the output of the SENet convolutional attention module in the 3D instance segmentation network with other global features as the target through a max-pooling layer, along with other features in the channel dimension. These attention weights are multiplied by the channel weights of the original n×c feature tensor, and then the corresponding channels are multiplied by a dot product, resulting in weighting of each channel of the original feature tensor. This allows for more non-linear cross-channel interactions while reducing feature dimensionality to avoid excessive model complexity. The PointNet network with the attention mechanism includes: an input module, an encoding unit, a decoding unit, and an output module, as follows: Figure 3 As shown in the figure. n represents the number of points in the 3D frustum point cloud, and c represents the number of features for each point in the 3D frustum point cloud.

[0062] The input module is used to input the position coordinates of points in the 3D frustum point cloud (point cloud data of an n×c dimensional array);

[0063] The encoding unit includes an upsampling convolutional layer, a first SENet convolutional attention module, and a max pooling layer.

[0064] The upsampling convolutional layer is connected to the input module and is used to obtain the initial center position coordinates of the object to be detected.

[0065] The first SENet convolutional attention module is connected to the upsampled convolutional layer to obtain the weights of points in the 3D view frustum point cloud;

[0066] The max pooling layer is connected to the decoding unit to obtain the weighted features of points in the 3D frustum point cloud;

[0067] Specifically, the upsampling convolutional layer comprises several cascaded 1D convolutional layers, with the output dimension of each 1D convolutional layer being twice that of the previous one. The upsampling convolutional layer is connected to the first SENet convolutional attention module to obtain the initial center position coordinates of the object to be detected. The first SENet convolutional attention module is connected to the upsampling convolutional layer and the max pooling layer at both ends to perform weight acquisition processing on each point in the 3D frustum point cloud. The output of the max pooling layer is used as the global feature of the target and concatenated with the category vector of the object to be detected in the channel dimension to perform channel weight multiplication on the feature vector. The result is then output as the input of the decoding unit.

[0068] The category vector of the object to be detected and the two features of the max pooling layer are concatenated at the channel level and used as the input of the decoding unit.

[0069] The decoding unit includes a downsampling convolutional layer and a second SENet convolutional attention module;

[0070] The downsampling convolutional layer is connected to the max-pooling layer and receives the output of the max-pooling layer. Based on the weighted features of points in the 3D frustum point cloud and the category vector of the object to be detected, these features are used as input to the decoding unit to obtain the residual between the center point coordinates of the object to be detected and its initial center position.

[0071] The second SENet convolutional attention module is connected to the downsampling convolutional layer and is used to obtain the center point coordinates of the object to be detected; the output module is connected to the second SENet convolutional attention module and is used to output the center point coordinates of the object to be detected.

[0072] The downsampling convolutional layer consists of three cascaded 1D convolutions, with each convolution's output dimension being half that of the previous one. Incorporating an attention mechanism into the segmentation network allows for better extraction of global features, significantly improving instance segmentation performance and increasing the completeness of the segmented target.

[0073] Specifically, the downsampling convolutional layer is connected to the second SENet convolutional attention module to optimize the output of the encoding unit. The second SENet convolutional attention module is connected to the downsampling convolutional layer and then to the output module at both ends to obtain the center point position of the target object. The resulting structure is used as the input to the output module. Finally, the output module obtains the center point coordinates of the object to be detected.

[0074] Preferably, the method for obtaining the center point coordinates of the object to be detected is as follows:

[0075] S51: Input the position coordinates of the points in the 3D frustum point cloud into the PointNet network with an attention mechanism. Use the feature extraction network of the input module to perform preliminary segmentation of the 3D frustum point cloud and obtain the initial center position coordinates (x', y', z') of the object to be detected. In this embodiment, the coordinate system is the camera coordinate system and the position of the camera is the origin.

[0076] S52: Obtain the distance d between the position coordinates of a point in the 3D frustum point cloud and the initial center position coordinates of the object to be detected. i ; will d i The data is fed into an attention network to obtain the weight of the i-th point in the 3D view frustum point cloud;

[0077] d i =p i -(x',y',z')

[0078] Where, p i Let (x', y', z') be the position coordinates of the i-th point in the 3D frustum point cloud; (x', y', z') be the initial center position coordinates of the object to be detected.

[0079] have to:

[0080] w i =Sigmoid(d i )

[0081] In the formula, w i represents the weight of the i-th point in the 3D frustum point cloud; Sigmoid(·) is a non-linear action function used to obtain the weight of the i-th point in the 3D frustum point cloud.

[0082] Specifically, the distance d from the position coordinates of each point in the 3D frustum point cloud to the initial center position coordinates of the object to be detected. i d i =p i -(x',y',z'), then d i The data is fed into an attention network to obtain the weight w for each point. i p i The input point cloud is a three-dimensional array (x i ,y i ,z i ), representing the position coordinates of point i in three-dimensional coordinates.

[0083] S53: d i Feed the data into a max pooling layer and obtain the features of the i-th point in the 3D view frustum point cloud. And perform weighted summation to obtain the weighted features of the i-th point in the 3D view frustum point cloud:

[0084]

[0085] In the formula: The weighted feature of the i-th point in the 3D view frustum point cloud; The feature of the i-th point in the 3D frustum point cloud; w i Let be the weight of the i-th point in the 3D view frustum point cloud;

[0086] Specifically, weighting is performed through the decompression layer of the PointNet network with an added attention mechanism to obtain...

[0087] S54: Based on the output module of the PointNet network with added attention mechanism, the weighted features of the i-th point in the 3D frustum point cloud are... By combining the category vector k of the object to be detected and these two features at the channel level, regression is performed to obtain the residual between the center point coordinates of the object to be detected and the initial center position of the object. Finally, the coordinates of the center point of the object to be detected are obtained;

[0088]

[0089]

[0090] In the formula: (x,y,z) are the coordinates of the center point of the object to be detected. This is the residual between the center point coordinates of the object to be detected and the initial center position of the object. k is the category vector of the object to be detected; concat(·) is a function that fuses the weighted features of the i-th point in the 3D frustum point cloud with the category vector of the object to be detected.

[0091] Specifically, the input is an n×c dimensional point cloud data array, where n is the number of input points. For each point, its spatial coordinates are input, and feature extraction is performed point by point. First, a multi-layer convolutional neural network is used to extract individual features for each point, expanding the dimension to 1024. The features of each point are fed into the SENet convolutional attention module, and then a max pooling layer is used to perform max pooling on the n×1024 dimensional feature vector. The resulting 1×1024 dimensional vector is used as the global vector of the point cloud. Finally, the global vector is fed into the SEblock attention module, which combines the category of the object to be detected with downsampling convolution to obtain an n×128 dimensional vector. This vector is then fed into the SEblock attention module to obtain the decompressed coordinates of the center point of the object to be detected.

[0092] S6: Based on the center point coordinates of the object to be detected, obtain the parameters of the 3D bounding box of the object to be detected, so as to detect the three-dimensional object to be detected; the parameters of the 3D bounding box of the object to be detected include the center coordinates, length, width, height and heading angle of the 3D bounding box of the object to be detected.

[0093] Specifically, the method in this embodiment obtains a cuboid as the 3D bounding box of the object to be detected. The length, width, height, center coordinates, and heading angle of the cuboid are determined, which can uniquely determine the position of the object to be detected.

[0094] S61: Input the center point coordinates of the object to be detected into the template supervision network T-Net to obtain the true center point coordinates of the object to be detected;

[0095] S62: Input the coordinates of the true center point of the object to be detected and fine-tuned into the 3D bounding box regression network to obtain the parameters of the 3D bounding box of the object to be detected.

[0096] Specifically, the coordinates of the center point of the object to be detected are input into a template supervision network (T-Net) for a given task to obtain the true center point coordinates of the object; then, after coordinate transformation, they are input into a 3D bounding box regression network to obtain 3D bounding box parameters. Both the 3D bounding box regression network and the template supervision network (T-Net) are existing technologies and will not be described in detail here.

[0097] The decompressed center position of the target object is input into a T-Net to obtain the true center position of the object to be detected, which is then input into a 3D bounding box regression network. The center position of the target object is used as input data and processed by the lightweight T-NET network, while simultaneously incorporating a global feature vector generated from the object's category to adjust the center position of the detected object.

[0098] The input data is fed into the feature extraction layer (128, 128, 256) of the T-Net network to obtain global features. These features are then combined with the category of the object to be detected, and the true 3D coordinates of the target object's center are derived through three fully connected layers (123, 123, 256). This data is then processed by the non-modal 3D bounding box estimation network PointNet with an attention mechanism. The network structure of PointNet with the attention mechanism is similar to that of PointNet with an attention mechanism, differing only in the number of convolutions and the number of decoding units. PointNet with the attention mechanism has n upsampling convolutions (n ​​= 3 in this embodiment), n+1 sampling convolutions, and the decoding unit uses fully connected layers instead of downsampling convolutions. Finally, the parameters of the 3D bounding box of the object to be detected are output.

[0099] Specifically, the coordinates of the true center point of the object to be detected are input into the 3D bounding box regression network for processing. First, a multi-layer convolutional neural network is used to extract individual features of each point, expanding the dimension to 512. Then, the features of each point are sent to the attention module. Then, a max pooling layer is used to perform max pooling on the n×512-dimensional feature vector in n dimensions. The resulting 1×512-dimensional vector is used as the global vector of the point cloud. Then, the category of the object to be detected is combined and passed through two fully connected layers to obtain an n×256-dimensional vector. This vector is then sent to the attention module and then through a fully connected layer to obtain all the parameter information of the 3D bounding box.

[0100] The entire network structure uses multiple network modules to train on 3D frustum point cloud data, including the PointNet network with an attention mechanism, the T-Net network for center position adjustment, and a 3D bounding box regression network. The total loss function, which combines the losses generated by these network structures during training, is calculated as follows:

[0101] L multi-task =L mask +L bbox =L mask +λ(L c1-reg +L c2-reg +L h-cls 10L h-reg 10L s-cls 10L s-reg +λL corner )

[0102] In the formula: L multi-task The total loss function for the PointNet network with added attention mechanism, template supervision network, and 3D bounding box regression network; L mask The loss function for the PointNet network with an added attention mechanism; L bbox The loss functions for the template-supervised network and the 3D bounding box regression network include the center position regression loss L generated by the template-supervised network. c1-reg And 3D bounding box regression network regression loss L c2-reg L h-cls For heading angle classification loss; L h-reg L is the heading angle regression loss; λ is the hyperparameter; corner L refers to the sum of the losses of the vertices of the 3D bounding box of the object to be detected; s-cls The classification loss is the size of the bounding box; L s-reg The regression loss is for the bounding box size;

[0103] One embodiment of this invention was conducted on the Sun-RGBD dataset. Ten common indoor objects were selected for detection: bed, dining table, sofa, chair, toilet, desk, dressing table, bedside table, bookshelf, and bathtub. The categories were represented by 10-dimensional one-hot vectors. In the category label for each object, only one bit was 1, and the rest were 0, with each position representing a different object category. For example, (1, 0, 0, 0, 0, 0, 0, 0, 0, 0) represented a sofa, and (0, 1, 0, 0, 0, 0, 0, 0, 0, 0) represented a dining table. The resulting vector, combined with the category information, outputs all the parameters for evaluating the 3D bounding box of the detected object: a 7-dimensional vector (including the center coordinates, length, width, and height of the 3D bounding box, and the heading angle of the 3D bounding box). The mAP (maximum average accuracy) with a 3DIoU threshold of 0.25 was used as the evaluation metric.

[0104] In this embodiment, λ is set to 1 and 10 respectively, L corner It refers to the sum of the losses of the 8 vertices corresponding to the 3D bounding box of the object to be detected.

[0105] The experimental results are shown in Table 1. By comparing with two existing target detection methods in RGBD images: 2d-driven 3dobject detection in RGBD images [DOI:10.1109 / ICCV.2017.495] and Frustum-PointNets [DOI:10.1109 / CVPR.2018.00102], it is verified that the mAP detection accuracy of the proposed scheme in this invention is the highest.

[0106] Table 1 Comparative Experimental Results of RGB-D Image Target Detection

[0107]

[0108]

[0109] This invention first obtains the RGB image of the color image in an RGB-D image, then converts the depth image into a 3D frustum point cloud to obtain spatial structural features. Based on a PointNet network with an added attention mechanism, the final feature map is used for segmentation, regression, and other operations to complete the object detection task. Simultaneously, the detection results are compared with some other outstanding 3D object detection algorithms, as shown in Table 1. It can be seen that the detection method of this invention has certain advantages in overall detection performance and improves accuracy. In addition, comparisons are made regarding some detailed method choices, such as the impact of different model initialization methods and the role of grouping and extracting features during point cloud processing, demonstrating that each process in the entire complex network structure has a positive impact on the final detection result. Experiments show that the color image detection process affects the final detection result while also improving the model's accuracy. This patent provides a new method for object detection in indoor scenes based on RGBD images and also offers new ideas on how to fully utilize RGBD image information.

[0110] This embodiment uses a 3D target detection method based on RGBD data and view frustums, drawing inspiration from the FrustumPointNets3D target detection method. The overall network structure diagram is shown below. Figure 2 As shown, a region generation network generates 2D candidate boxes from an RGB image and combines them with a depth image to extract corresponding 3D frustum point cloud data. Then, a PointNet network with an attention mechanism and a 3D bounding box regression network are used to detect 3D objects in the frustum point cloud data. Here, n represents the frustum point cloud extracted by the frustum extraction module, m represents the target point cloud after instance segmentation, c represents the number of features for each point, and k represents the number of target categories in the point cloud. This approach can improve detection accuracy while maintaining detection speed.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A three-dimensional target detection method based on RGBD data and view frustum, characterized in that, Includes the following steps: S1: Obtain the RGB-D image of the scene, wherein the RGB-D image of the scene includes the RGB image of the scene and the depth image of the scene; S2: Based on the RGB image of the scene, obtain the feature image of the RGB image of the scene; S3: Input the feature image of the RGB image of the scene into the region generation network to obtain the bounding box of the object to be detected and the category vector of the object to be detected; S4: Based on the bounding box of the object to be detected and the depth image of the scene, obtain a 2D candidate region, and then obtain a 3D frustum point cloud based on the 2D candidate region, and then obtain the position coordinates of the points in the 3D frustum point cloud. S5: Input the position coordinates of the points in the 3D frustum point cloud and the category vector of the object to be detected into the PointNet network with an attention mechanism to obtain the center point coordinates of the object to be detected; The method for obtaining the center point coordinates of the object to be detected is as follows: S51: Input the position coordinates of points in the 3D frustum point cloud into the PointNet network with an attention mechanism to obtain the initial center position coordinates of the object to be detected. S52: Obtain the distance between the position coordinates of a point in the 3D frustum point cloud and the initial center position coordinates of the object to be detected. ; to obtain the first in the 3D view frustum point cloud i The weight of each point; in, The first in the 3D view frustum point cloud i The position coordinates of each point; The initial center position coordinates of the object to be detected; have to: In the formula, The first in the 3D view frustum point cloud i The weight of each point; For obtaining the first point cloud of a 3D view frustum i The nonlinear action function of the weights of each point; S53: Obtain the first point in the 3D view frustum point cloud i Features of each point And perform weighted sampling to obtain the first point in the 3D view frustum point cloud. i Weighted features of points: In the formula: The first in the 3D view frustum point cloud i Weighted features of points; The first in the 3D view frustum point cloud i Features of each point; S54: Obtain the coordinates of the center point of the object to be detected; ； In the formula: Here are the coordinates of the center point of the object to be detected. This is the residual between the center point coordinates of the object to be detected and the initial center position of the object. ; k This is the category vector of the object to be detected; The first in the 3D view frustum point cloud i A function that fuses the weighted features of each point with the category vector of the object to be detected; S6: Based on the center point coordinates of the object to be detected, obtain the parameters of the 3D bounding box of the object to be detected in order to detect the three-dimensional object. The parameters of the 3D bounding box of the object to be detected include the center coordinates, length, width, height, and heading angle of the 3D bounding box of the object to be detected.

2. The three-dimensional target detection method based on RGBD data and view frustum according to claim 1, characterized in that, The PointNet network with added attention mechanism includes: an input module, an encoding unit, a decoding unit, and an output module; The input module is used to input the position coordinates of points in the 3D frustum point cloud; The encoding unit includes an upsampling convolutional layer, a first SENet convolutional attention module, and a max pooling layer. The upsampling convolutional layer is connected to the input module and is used to obtain the initial center position coordinates of the object to be detected. The first SENet convolutional attention module is connected to the upsampled convolutional layer to obtain the weights of points in the 3D frustum point cloud based on the initial center position coordinates of the object to be detected. The max pooling layer is connected to the decoding unit to obtain the weighted features of the points in the 3D frustum point cloud based on the weights of the points in the 3D frustum point cloud. The decoding unit includes a downsampling convolutional layer and a second SENet convolutional attention module; The downsampling convolutional layer is connected to the max pooling layer and is used to obtain the residual between the center point coordinates of the object to be detected and the initial center position of the object to be detected, based on the weighted features of the points in the 3D frustum point cloud and the category vector of the object to be detected. ; The second SENet convolutional attention module is connected to the downsampling convolutional layer and is used to obtain the center point coordinates of the object to be detected based on the residual between the center point coordinates of the object to be detected and the initial center position of the object to be detected. The output module is connected to the second SENet convolutional attention module and is used to output the center point coordinates of the object to be detected.

3. The three-dimensional target detection method based on RGBD data and view frustum according to claim 1, characterized in that, The method for obtaining the parameters of the 3D bounding box of the object to be detected is as follows: S61: Input the center point coordinates of the object to be detected into the Template-Supervised Network (T-Net) to obtain the true center point coordinates of the object to be detected; S62: Input the true center point coordinates of the object to be detected into the 3D bounding box regression network to obtain the parameters of the 3D bounding box of the object to be detected.

4. The three-dimensional target detection method based on RGBD data and view frustum according to claim 3, characterized in that, The total loss function of the PointNet network with added attention mechanism, the template supervision network T-Net, and the 3D bounding box regression network is obtained as follows: In the formula: The total loss function for the PointNet network with an added attention mechanism, template supervision network, and 3D bounding box regression network; The loss function for the PointNet network with an added attention mechanism; The loss functions for the template-supervised network and the 3D bounding box regression network include the center position regression loss generated by the template-supervised network. Regression loss of 3D bounding box regression network ; Loss is classified by heading angle; The heading angle regression loss is λ, which is a hyperparameter. This refers to the sum of the losses of the vertices of the 3D bounding box of the object to be detected; The classification loss is the size of the 3D bounding box of the object to be detected; The regression loss is the size of the 3D bounding box of the object to be detected.

Images