A point cloud target detection method based on structure information enhancement and related equipment

Abstract

The application discloses a point cloud target detection method based on structure information enhancement and related equipment, and relates to the field of target detection. The method comprises the following steps: obtaining a training sample set and inputting the training sample set into a point cloud target detection network; extracting enhanced voxel features of point cloud data in the training sample set based on a backbone network of the point cloud target detection network; extracting initial candidate box parameters according to the enhanced voxel features; performing non-maximum suppression operation, region of interest pooling operation, point cloud completion operation, structure feature extraction operation and feature fusion operation on the initial candidate box parameters to obtain aggregated features; performing class information prediction operation and box regression operation according to the aggregated features to obtain target class score and box parameter residual, updating parameters of the point cloud target detection network, obtaining a point cloud target detection model; and detecting a to-be-detected sample set by using the point cloud target detection model to generate a point cloud target detection box and point cloud target class information, and completing point cloud target detection.

Description

TECHNICAL FIELD

[0001] The present specification relates to the field of target detection, and more particularly, the present application relates to a point cloud target detection method based on structure information enhancement and related equipment. BACKGROUND

[0002] With the rise of computer vision and artificial intelligence technology, the automatic driving technology develops rapidly, and its core task is to perceive and locate each target around in a complex environment, therefore, it is crucial for the automatic driving system to study the target detection method in a complex scene.

[0003] The traditional target detection method uses RGB image as input data, and realizes target detection by outputting the category and position information of the target. However, due to the difficulty of two-dimensional image information to meet the needs of target detection in the automatic driving scene, the point cloud target detection method based on laser radar point cloud data has obtained wide attention, which increases the estimation of target size, depth and posture and other information on the basis of two-dimensional target detection method, thereby greatly improving the target detection performance.

[0004] In the actual traffic scene, due to the imbalance of laser radar sensor and the complexity of the environment, the occluded target and the long-distance target are too sparse and incomplete in shape in the point cloud collected by the sensor, which leads to the loss of important geometric and semantic information of these targets, making it difficult to be detected, thereby affecting the overall detection performance of the automatic driving system. Therefore, how to enhance the complete structure information of these targets to improve their feature expression in complex scenes is crucial for improving the target detection ability of the automatic driving system in actual complex scenes. SUMMARY

[0005] A series of simplified concepts are introduced in the summary section, which will be further described in detail in the specific embodiment section. The summary section of the present application does not mean to try to limit the key features and necessary technical features of the claimed technical solutions, nor does it mean to try to determine the protection scope of the claimed technical solutions.

[0006] In a first aspect, the present application provides a point cloud target detection method based on structure information enhancement, the method comprises:

[0007] Obtaining a training sample set, inputting into a point cloud target detection network, and extracting enhanced voxel features of point cloud data in the training sample set based on a backbone network in the point cloud target detection network, wherein the backbone network comprises a voxelization operation, a context information enhancement encoder and an up-sampling operation;

[0008] According to the enhanced voxel features, point-wise features are obtained, and the point-wise features are used to extract initial candidate box parameters based on a region proposal network in the point cloud target detection network;

[0009] The initial candidate box parameters are used to perform a non-maximum suppression operation, a region of interest pooling operation, a point cloud completion operation, a structure feature extraction operation, and a feature fusion operation in the point cloud target detection network to obtain aggregated features;

[0010] According to the aggregated features, a class information prediction operation and a box regression operation are performed in the point cloud target detection network to obtain target class scores and box parameter residuals, and a total loss value is calculated based on the initial candidate box parameters, the target class scores, and the box parameter residuals to update parameters of the point cloud target detection network to obtain a point cloud target detection model;

[0011] The point cloud target detection model is used to detect a set of to-be-detected samples to generate point cloud target detection boxes and point cloud target class information, and point cloud target detection is completed.

[0012] In some embodiments, the enhanced voxel features of the point cloud data in the training sample set are extracted based on a backbone network in the point cloud target detection network, including:

[0013] According to the point cloud data in the training sample set, voxel data and voxel features are obtained based on the voxelization operation;

[0014] The voxel data and the voxel features are subjected to H first iteration operations by the context information enhancement encoder to obtain enhanced voxel data and attention-enhanced features, wherein the context information enhancement encoder includes H layers, H is a positive integer greater than or equal to 2, and each layer performs one of the first iteration operations;

[0015] The enhanced voxel data and the attention-enhanced features are used to perform the upsampling operation to obtain the enhanced voxel features.

[0016] In some embodiments, the first iteration operation includes a downsampling operation, an aggregation expansion operation, a local attention range enhancement weighting operation, an inflation attention range enhancement weighting operation, a first splicing operation, and a first feedforward network operation,

[0017] The voxel data and the voxel features are subjected to H first iteration operations by the context information enhancement encoder to obtain enhanced voxel data and attention-enhanced features, including:

[0018] The voxel data and the voxel features are used as first-layer input voxel data and first-layer input voxel features of the context information enhancement encoder;

[0019] performing the down-sampling operation and the aggregation expansion operation on the first-layer input voxel data and the first-layer input voxel feature of the context information enhanced encoder to obtain expanded non-empty voxels and expanded non-empty voxel features, and taking the expanded non-empty voxels as first-layer output voxel data;

[0020] performing the local attention range enhanced weighting operation and the dilated attention range enhanced weighting operation on the expanded non-empty voxels to obtain first-layer local attention weighted voxel features and first-layer dilated attention weighted voxel features;

[0021] performing the first splicing operation and the first feedforward network operation on the first-layer local attention weighted voxel features and the first-layer dilated attention weighted voxel features to obtain first-layer output voxel features;

[0022] continuing to perform the first iteration operation (H-1) times through the remaining (H-1) layers of the context information enhanced encoder on the first-layer output voxel data and the first-layer output voxel features to obtain enhanced voxel data and attention enhanced features, wherein the enhanced voxel data and the attention enhanced features are output by the last layer of the context information enhanced encoder.

[0023] In some embodiments, the initial candidate box parameters are subjected to a non-maximum suppression operation, a region of interest pooling operation, a point cloud completion operation, a structure feature extraction operation, and a feature fusion operation in the point cloud object detection network to obtain aggregated features, including:

[0024] performing the non-maximum suppression operation on the initial candidate box parameters to obtain filtered candidate boxes;

[0025] performing the region of interest pooling operation on the filtered candidate boxes to obtain region of interest features;

[0026] performing the point cloud completion operation on the filtered candidate boxes to obtain dense point clouds;

[0027] performing the structure feature extraction operation on the dense point clouds to obtain target structure features;

[0028] performing the feature fusion operation on the region of interest features and the target structure features to obtain the aggregated features.

[0029] In some embodiments, the point cloud completion operation includes a farthest point sampling operation, an extraction of center point neighborhood range feature operation, a first embedding operation, a second iteration operation, a max-pooling operation, a multi-layer perceptron, a second embedding operation, a third iteration operation, and a neural network operation,

[0030] The point cloud completion operation is performed on the candidate box to obtain a dense point cloud, including:

[0031] The farthest point sampling operation is performed on the candidate box to obtain first center point data;

[0032] The center point neighborhood range feature extraction operation is performed on the first center point data to obtain a neighborhood feature;

[0033] The first embedding operation is performed on the first center point data and the neighborhood feature to obtain a position feature sequence;

[0034] The position feature sequence and the first center point data are input into a target encoder to perform the second iteration operation D times to obtain global features, wherein the target encoder includes D layers, D is a positive integer greater than or equal to 2, each layer performs the second iteration operation once, the second iteration operation includes a first geometric attention operation, a second feedforward network operation, and a first normalization batch operation, and the first geometric attention operation includes a first semantic feature output branch and a first geometric feature output branch;

[0035] The global features are subjected to the max-pooling operation and the multi-layer perceptron to obtain second center point data and region features;

[0036] The second embedding operation is performed on the second center point data and the region features to obtain a coarse shape feature sequence;

[0037] The coarse shape feature sequence and the second center point data are input into a target decoder to perform the third iteration operation D times to obtain refinement features, wherein the target decoder includes D layers, D is a positive integer greater than or equal to 2, each layer performs the third iteration operation once, the third iteration operation includes a second geometric attention operation, a cross-attention operation, a third feedforward network operation, and a second normalization batch operation, and the second geometric attention operation is composed of a second semantic feature output branch and a second geometric feature output branch;

[0038] The neural network operation is performed on the refinement features to obtain the dense point cloud.

[0039] In some embodiments, the feature fusion operation includes a second concatenation operation, a spatial attention operation, a channel attention operation, an element-wise multiplication operation, an activation operation, and a weighted summation operation,

[0040] The feature fusion operation is performed on the region of interest feature and the target structure feature to obtain the aggregated feature, including:

[0041] performing the second splicing operation on the region-of-interest feature and the target structure feature to obtain a spliced feature;

[0042] performing the spatial attention operation on the spliced feature to obtain a spatial attention feature;

[0043] performing the channel attention operation on the spliced feature to obtain a channel attention feature;

[0044] performing the element-wise multiplication operation and the activation operation on the spatial attention feature and the channel attention feature to obtain a feature weight;

[0045] performing the weighted summation operation on the region-of-interest feature and the target structure feature by using the feature weight to obtain the aggregated feature.

[0046] In some embodiments, the class information prediction operation and the bounding box regression operation in the point cloud target detection network according to the aggregated feature are performed to obtain a target class score and a bounding box parameter residual, and a total loss value is calculated based on the initial candidate bounding box parameter, the target class score and the bounding box parameter residual to update the parameters of the point cloud target detection network to obtain a point cloud target detection model, including:

[0047] the class information prediction operation in the point cloud target detection network is performed on the aggregated feature to obtain the target class score;

[0048] the bounding box regression operation in the point cloud target detection network is performed on the aggregated feature to obtain the bounding box parameter residual;

[0049] the total loss value is calculated based on the initial candidate bounding box parameter, the target class score and the bounding box parameter residual to update the parameters of the point cloud target detection network to obtain the point cloud target detection model.

[0050] In a second aspect, the present application further provides a point cloud target detection device based on structure information enhancement, comprising:

[0051] a first extraction unit configured to obtain a training sample set and input the training sample set into a point cloud target detection network, and extract an enhanced voxel feature of point cloud data in the training sample set based on a backbone network in the point cloud target detection network, wherein the backbone network comprises a voxelization operation, a context information enhancement encoder and an up-sampling operation;

[0052] a second extraction unit configured to obtain a point-by-point feature according to the enhanced voxel feature, and extract an initial candidate bounding box parameter based on the point-by-point feature and a region candidate network in the point cloud target detection network;

[0053] The first obtaining unit is configured to obtain aggregated features by performing a non-maximum suppression operation, a region of interest pooling operation, a point cloud completion operation, a structure feature extraction operation and a feature fusion operation on the initial candidate box parameters in the point cloud target detection network.

[0054] The second obtaining unit is configured to perform a class information prediction operation and a box regression operation on the aggregated features in the point cloud target detection network to obtain target class scores and box parameter residuals, and calculate a total loss value based on the initial candidate box parameters, the target class scores and the box parameter residuals to update parameters of the point cloud target detection network to obtain a point cloud target detection model.

[0055] The generating unit is configured to perform detection on a to-be-detected sample set by using the point cloud target detection model to generate point cloud target detection boxes and point cloud target class information, and complete point cloud target detection.

[0056] In a third aspect, an electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor is configured to implement the steps of the point cloud target detection method based on structure information enhancement according to any one of the first aspect when executing the computer program stored in the memory.

[0057] In a fourth aspect, the present application further provides a computer readable storage medium having a computer program stored thereon, and the computer program is executable by a processor to implement the point cloud target detection method based on structure information enhancement according to any one of the first aspect.

[0058] In summary, the point cloud target detection method based on structure information enhancement in the embodiment of the application comprises: obtaining a training sample set and inputting into a point cloud target detection network, extracting enhanced voxel features of point cloud data in the training sample set based on a backbone network in the point cloud target detection network, wherein the backbone network comprises a voxelization operation, a context information enhancement encoder and an up-sampling operation; obtaining point-by-point features according to the enhanced voxel features, and extracting initial candidate box parameters based on a region candidate network in the point cloud target detection network; using a non-maximum suppression operation, a region of interest pooling operation, a point cloud completion operation, a structure feature extraction operation and a feature fusion operation in the point cloud target detection network to obtain aggregated features for the initial candidate box parameters; performing a class information prediction operation and a box regression operation in the point cloud target detection network according to the aggregated features to obtain target class score and box parameter residual, and calculating a total loss value based on the initial candidate box parameters, the target class score and the box parameter residual to update parameters of the point cloud target detection network to obtain a point cloud target detection model; and detecting a to-be-detected sample set by using the point cloud target detection model to generate a point cloud target detection box and point cloud target class information, and completing point cloud target detection. The application expands the voxel space by aggregating the spatial information of empty voxels and non-empty voxels, so that the network fully captures the context information in a larger receptive field to generate a high-quality initial candidate box. Meanwhile, the application proposes a novel point cloud completion operation, which is introduced into the point cloud target detection network to fully explore the global structure association of the target point cloud, so as to predict the missing part of the target and supplement the complete shape information for the incomplete target point cloud. The application also constructs a feature fusion operation based on a hybrid attention, which integrates the reconstructed shape information into the spatial features of the target, thereby enhancing the geometric feature expression ability of the target and effectively improving the target detection performance in a complex scene.

[0059] The point cloud target detection method based on structure information enhancement, other advantages, objects and features of the application will be embodied in part through the following description, and will be understood by those skilled in the art through research and practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0060] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are merely illustrative and are not considered to be limiting in any sense. Throughout the drawings, like reference numerals refer to similar parts. In the drawings:

[0061] Figure 1 A point cloud target detection method based on structure information enhancement provided by the embodiment of the application is shown in the flowchart;

[0062] Figure 2 A structural schematic diagram of a point cloud target detection network model based on structural information enhancement provided by an embodiment of the present application;

[0063] Figure 3 A comparison diagram of average target detection precision of a point cloud target detection method based on structural information enhancement provided by an embodiment of the present application and other methods under three difficulty levels in a first sample set to be detected;

[0064] Figure 4 A comparison diagram of average target detection precision of a point cloud target detection method based on structural information enhancement provided by an embodiment of the present application and other methods under three difficulty levels in a second sample set to be detected;

[0065] Figure 5 A comparison diagram of point cloud target detection boxes obtained by different methods provided by an embodiment of the present application;

[0066] Figure 6 A structural schematic diagram of a point cloud target detection device based on structural information enhancement provided by an embodiment of the present application;

[0067] Figure 7 A structural schematic diagram of a point cloud target detection electronic device based on structural information enhancement provided by an embodiment of the present application. DETAILED DESCRIPTION

[0068] The terms "first", "second", "third", "fourth" and the like in the description and claims of the present application and in the above drawings (if there are) are used to distinguish similar objects, and do not necessarily have to be used to describe a particular order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. The technical solutions in the embodiments of the present application will be described in detail below with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments.

[0069] In actual automatic driving scenarios, for occlusion and long-distance targets, the points obtained by laser radar sensor scanning are too few, and the fewer the points, the more incomplete the shape of the target, and the less the geometric feature information, thereby causing the target to be difficult to be perceived and detected, and affecting the overall detection accuracy. In order to solve this problem, the application constructs a point cloud target detection method based on structure information enhancement, which can effectively improve the target detection performance by restoring the complete shape features of the incomplete target.

[0070] Please refer to Figure 1 , which is a point cloud target detection method based on structure information enhancement provided by an embodiment of the application, and specifically can include steps S110-S150. The method adopted by the application can be implemented through a point cloud target detection network model as shown in Figure 2 . In the first stage of the network model, a voxelization operation, a context information enhancement encoder and an upsampling operation are first introduced to construct a novel backbone network for extracting enhanced voxel features of point cloud data in the input training sample set, and then initial candidate box parameters are extracted based on a region candidate network. Subsequently, in the second stage of the network, the application first performs a non-maximum suppression operation on the initial candidate box parameters to obtain a screening candidate box, and performs a region of interest pooling operation on the screening candidate box to obtain a region of interest feature. Then, the application designs a point cloud completion operation for restoring the complete shape of the target in the screening candidate box, and extracts the structure features of the target based on a structure feature extraction operation. Subsequently, the application constructs a feature fusion operation based on hybrid attention, which integrates the restored structure features of the target into the region of interest features to enhance the geometric information of the target for box refinement, thereby generating an accurate target detection box.

[0071] S110, obtain a training sample set and input it into a point cloud target detection network, extract enhanced voxel features of point cloud data in the training sample set based on a backbone network in the point cloud target detection network, wherein the backbone network includes a voxelization operation, a context information enhancement encoder and an upsampling operation;

[0072] For example, the application uses training samples in a KITTI public data set to construct a training sample set, and inputs the training sample set into a point cloud target detection network to extract enhanced voxel features of point cloud data in the training sample set through the backbone network thereof.

[0073] In one embodiment, step S110 specifically further includes:

[0074] S1101, obtain voxel data p (1) and voxel features F (1) based on the voxelization operation according to the point cloud data in the training sample set.

[0075] For example, the point cloud data with a size of N0xC0in the training sample set is input into the point cloud target detection network. Firstly, the point cloud data is voxelized into voxel data p (1) with a size of N (1) ×3, and then the average value of the point coordinates in the voxel is calculated to obtain voxel features F (1) with a size of N (1) ×C (1) , wherein N0represents the number of the original point cloud, C0represents the feature dimension of the original point cloud, N (1) represents the number of the voxel, C (1) represents the feature dimension of the voxel features, and N0, C0, N (1) , C (1) are all positive integers.

[0076] S1102, performing H times of first iteration operations on the voxel data p (1) and the voxel features F (1) by using the context information enhanced encoder to obtain enhanced voxel data p (H+1) and attention enhanced features F (H+1) , wherein the context information enhanced encoder comprises H layers, H is a positive integer greater than or equal to 2, and each layer performs one of the first iteration operations;

[0077] In an embodiment, the first iteration operation comprises a downsampling operation, an aggregation expansion operation, a local attention range enhanced weighting operation, an inflation attention range enhanced weighting operation, a first splicing operation and a first feedforward network operation, and step S1102 further comprises steps S1102-A to S1102-E:

[0078] S1102-A, taking the voxel data p (1) and the voxel features F (1) as the first layer input voxel data and the first layer input voxel features of the context information enhanced encoder;

[0079] S1102-B, performing the downsampling operation and the aggregation expansion operation on the first layer input voxel data p (1) and the first layer input voxel features F (1) to obtain expanded non-empty voxels and expanded non-empty voxel features , and taking the expanded non-empty voxels as the first layer output voxel data p (2) ;

[0080] S1102-C, performing the local attention range enhanced weighting operation and the inflation attention range enhanced weighting operation on the expanded non-empty voxels to obtain first layer local attention weighted voxel features and the first layer expanded attention weighted voxel feature

[0081] S1102-D, the first layer local attention weighted voxel feature and the first layer expanded attention weighted voxel feature the first splicing operation and the first feedforward network operation are performed to obtain the first layer output voxel feature F (2) ;

[0082] S1102-E, the first layer output voxel data p (2) and the first layer output voxel feature F (2) the first iteration operation is performed (H-1) times by the remaining (H-1) layers of the context information enhanced encoder to obtain the enhanced voxel data p (H+1) and the attention enhanced feature F (H+1) , wherein the enhanced voxel data p (H+1) and the attention enhanced feature F (H +1) is the output of the last layer of the context information enhanced encoder.

[0083] For example, the voxel data p (1) and the voxel feature F (1) are taken as the first layer input voxel data and the first layer input voxel feature of the context information enhanced encoder, and the first iteration operation is performed H times by the context information enhanced encoder to obtain the enhanced voxel data p (H+1) and the attention enhanced feature F (H+1) .

[0084] The process of performing the first iteration operation in the first layer of the context information enhanced encoder is as follows:

[0085] For the first layer input voxel data p (1) and the first layer input voxel feature F (1) , the voxel data is first down-sampled at a specific step, and then the spatial information of empty voxels and non-empty voxels is aggregated to expand the non-empty voxel space, so that the expanded non-empty voxel p (2) with a size of N (2) ×3 is obtained. Since the expanded non-empty voxel has no feature, the voxel feature corresponding to the non-empty voxel in the neighborhood range is aggregated to obtain the expanded non-empty voxel feature corresponding to the expanded non-empty voxel with a size of

[0086] The first layer query voxel is selected in the expanded non-empty voxel, and the distance is selected in the local attention range ​Different voxels within the first layer are respectively used as the first-layer key voxels and the first-layer value voxels. Then, the first-layer query voxels, the first-layer key voxels, and the first-layer value voxels are used to perform effective grouped self-attention computation in order to capture contextual relationships in a voxel space with a larger receptive field, thereby obtaining the first-layer local attention-weighted voxel features.

[0087] Specifically, in the grouped self-attention calculation, for a single attention head, the first query vector Q of the first layer is first calculated. (1) The first key vector K in the first layer (1) and the first value vector V of the first layer (1) As shown in equation (1):

[0088]

[0089] in, These represent sizes N respectively. (2) ×N (2) The linear projection matrix of the first query vector, the first key vector, and the first value vector of the first layer, and the first layer position encoding u. (1) The calculation process is shown in equation (2).

[0090]

[0091] in, These are the center point coordinates of the first-level query voxel and the first-level key voxel, respectively. Indicates the first-level position code u (1) The linear projection matrix.

[0092] Then, the dimensions of the first query vector and the first key vector in the first layer are respectively... The data is uniformly divided into M groups, where the query vector for the m-th group (m = 1, 2, ..., M) is... The size is Grouping key vectors The size is N (2) , M are all positive integers. By calculating the m-th group... and The attention features of the m-th group are obtained by multiplying the products of the products. The attention features of all groups are then summed to obtain a result of size N. (2) ×N (2) First-layer cumulative attention features As shown in equation (3):

[0093]

[0094] Where T represents the matrix transpose.

[0095] Will Attention weights are obtained through the first activation function. Then, these attention weights are applied to the first value vector V of the first layer. (1) Reweighting is performed, and a residual structure is used to obtain a value of... First-layer local attention-weighted voxel features As shown in equation (4):

[0096]

[0097] in, Q represents the first query vector at the first level. (1) The channel dimension is a positive integer, and σ represents the first activation function.

[0098] To capture contextual information within a larger receptive field, B expanded attention ranges are further defined. Within these ranges, different expanded non-empty voxels are selected to perform the aforementioned grouped self-attention operation, resulting in a size of... First-layer dilated attention-weighted voxel features Will and The voxel features from different attention ranges are concatenated through the first concatenation operation, and then further updated by the first feedforward network to obtain a size of N. (2) ×C (2) First layer output voxel feature F (2) .

[0099] The first layer output voxel data p (2) and the first layer output voxel features F (2) The remaining (H-1) layers of the context-information-enhanced encoder are then used to perform (H-1) more first-iteration operations to obtain the enhanced voxel data p of the final layer output of the context-information-enhanced encoder. (H+1) and attention enhancement feature F (H+1) .

[0100] S1103, Based on the above enhanced voxel data p (H+1) And the aforementioned attention enhancement feature F (H+1) Perform the above upsampling operation to obtain the enhanced voxel features.

[0101] For example, to recover the structural details contained in the original voxels lost in the downsampling operation of the context-information-enhanced encoder, this invention further constructs an upsampling operation consisting of H layers of sparse deconvolution to sequentially reduce the attention-enhancing features F. (H+1) The channel dimension, and gradually restore the enhanced voxel data p (H+1)spatial resolution. In each layer of the up-sampling operation, lateral connections are adopted to integrate the output features of each layer of the context information enhanced encoder into the features output by the corresponding layer of the up-sampling operation with the same scale, so as to enhance the voxel features output by the up-sampling operation, and obtain enhanced voxel features with a size of N (1) ×C2, where C2 represents the feature dimension of the enhanced voxel features and is a positive integer.

[0102] S120, obtaining point-wise features according to the enhanced voxel features, and extracting initial bounding box parameters based on the region proposal network in the point cloud target detection network;

[0103] For example, the backbone network constructed based on the voxelization operation, the context information enhanced encoder and the up-sampling operation can sufficiently learn the spatial context information of the target, so as to obtain enhanced voxel features with a size of N (1) ×C2, which is the same as the original non-empty voxel resolution. Then, the center points of each non-empty voxel are calculated, and the enhanced voxel features are regarded as the features of the center points, so as to form a new point cloud with point-wise features. The point-wise features of the new point cloud are sent into the region proposal network to segment the foreground points and predict the residual of the corresponding initial bounding box parameters, so as to generate initial bounding box parameters with a size of N2×7 based on the position information of the foreground points, where N2 represents the number of initial bounding box parameters and is a positive integer.

[0104] S130, performing the non-maximum suppression operation, the region of interest pooling operation, the point cloud completion operation, the structure feature extraction operation and the feature fusion operation on the initial bounding box parameters to obtain aggregated features;

[0105] In an embodiment, step S130 further comprises steps S1301-S1305:

[0106] S1301, performing the non-maximum suppression operation on the initial bounding box parameters to obtain filtered bounding boxes;

[0107] S1302, performing the region of interest pooling operation on the filtered bounding boxes to obtain region of interest features F roi ;

[0108] S1303, performing the point cloud completion operation on the filtered bounding boxes to obtain dense point clouds;

[0109] For example, based on the initial candidate box parameters described above, non-maximum suppression is first used to filter out redundant candidate boxes, resulting in a set of N3×7 filtered candidate boxes, where N3 represents the number of filtered candidate boxes and is a positive integer. Then, regions of interest (ROIs) are defined according to the parameters of the filtered candidate boxes, and ROI pooling is performed on the corresponding point-by-point features. By aggregating the global spatial information of the points within the boxes, a ROI feature F of size N3×C3 is obtained. roi Where C3 represents the feature dimension of the region of interest and is a positive integer. Furthermore, since the target shape in the candidate filtering box is usually incomplete, this invention designs a point cloud completion operation to reconstruct the complete shape point cloud of the target.

[0110] In one implementation, the point cloud completion operation includes a farthest point sampling operation, a center point neighborhood feature extraction operation, a first embedding operation, a second iterative operation, a max pooling operation, a multilayer perceptron, a second embedding operation, a third iterative operation, and a neural network operation.

[0111] Step S1303 specifically includes steps S1303-A to S1303-H:

[0112] S1303-A: Perform the farthest point sampling operation on the above-mentioned candidate filtering boxes to obtain the first center point data.

[0113] S1303-B, Regarding the aforementioned first center point data Perform the above operation to extract the neighborhood features of the center point to obtain the neighborhood features F. h ;

[0114] S1303-C, Regarding the aforementioned first center point data and the aforementioned neighborhood features F h Perform the first embedding operation described above to obtain the location feature sequence F. s (1) ;

[0115] S1303-D, the above positional feature sequence F s (1) And the above-mentioned first center point data The input target encoder performs the second iteration operation D times to obtain the global features F. s (D+1) The target encoder includes D layers, where D is a positive integer greater than or equal to 2. Each layer performs the second iteration operation once. The second iteration operation includes a first geometric attention operation, a second feedforward network operation, and a first normalization batching operation. The first geometric attention operation includes a first semantic feature output branch and a first geometric feature output branch.

[0116] For example, N4 points are sampled from the region of interest determined by the candidate box parameters as input points for the point cloud completion operation. In the point cloud completion operation, the farthest point sampling operation is first used to sample points of size N from the input points. c ×3 First center point data Then, a dynamic graph edge convolutional network is used to extract the neighborhood features of the center point to extract the neighborhood features F of the center point. h Then, the data from the first center point... Embedded into neighborhood features F h In the middle, we get a size of Location feature sequence F s (1) ,in, The feature dimension of the positional feature sequence is N₄,N. c , All are positive integers. The positional feature sequence F... s (1) and the first center point data The input target encoder performs D second-iteration operations to obtain global features F. s (D+1) .

[0117] The location feature sequence F s (1) The first-layer input position feature sequence of the target encoder is used to perform D second-step iterations to obtain global features F that include the known and missing structural relationships of the target. s (D+1) The target encoder includes D layers, each of which performs a second iteration operation. The second iteration operation includes a first geometric attention operation, a second feedforward network operation, and a first normalization batching operation. The first geometric attention operation consists of a first semantic feature output branch and a first geometric feature output branch.

[0118] The process of performing the second iteration operation in the first layer of the target encoder is as follows:

[0119] For the first layer input position feature sequence F of the target encoder s (1) First, the first semantic feature output branch of the first geometric attention operation is of size 1. The first layer input position feature sequence F s (1) Calculate the first-level second query vector in a single attention head First layer second key vector and the first layer second value vector As shown in equation (5):

[0120]

[0121] wherein, are linear projection matrices of the first-layer second query vector, the first-layer second key vector and the first-layer second value vector with size N c ×N c .

[0122] Subsequently, structural similarity is captured by multiplying and to obtain the first-layer attention score S (1) , as shown in equation (6):

[0123]

[0124] wherein, denotes the feature dimension of the first-layer second query vector and is a positive integer.

[0125] Then, a sparse attention mask operation is adopted on S (1) to select the first a elements of each row. Specifically, assuming that the a-th maximum value of the i-th row is o i , if the value of the j-th element of the i-th row is greater than o i , the value is retained, otherwise, the value is set to -∞, where a is a hyperparameter.

[0126] Thus, the attention scores in S (1) representing strong correlation between and are retained, and then a second activation function is adopted to obtain the attention weighting value. Then, the attention weighting value is multiplied with the first-layer second value vector to enhance the key features strongly correlated with , thereby obtaining the feature of a single attention head containing key information with size Subsequently, the features of each attention head are spliced to obtain the output of multi-head sparse attention, i.e., the center point semantic feature

[0127] Meanwhile, the first geometric feature output branch of the first geometric attention operation of the target encoder is used to process the input first center point data The first PointNet network is adopted to aggregate the structural information in the neighborhood of the center point by using the ball query strategy, thereby outputting the center point geometric feature with size The center point semantic feature and the center point geometric feature The output feature of the first geometric attention operation of the first layer of the target encoder is obtained, and the feature is further updated by using a second feedforward network operation and a first normalization batch operation, to obtain a first-layer output position feature sequence F . s (2) .

[0128] The first-layer output position feature sequence F s (2) (D-1) times of second iterative operations are continued to be performed through the remaining (D-1) layers of the target encoder, to obtain the global feature F s (D+1) .

[0129] S1303-E, the above global feature F s (D+1) The above max-pooling operation and the above multi-layer perceptron are used to obtain the second center point data and the region feature F region ;

[0130] S1303-F, the above second center point data and the above region feature F region are subjected to the above second embedding operation, to obtain a coarse shape feature sequence F c (1) ;

[0131] S1303-G, the above coarse shape feature sequence F c (1) and the above second center point data are input into the target decoder to perform D times of the above third iterative operation, to obtain the refined feature F c (D+1) , wherein the target decoder comprises D layers, D is a positive integer greater than or equal to 2, each layer performs one time of the above third iterative operation, the third iterative operation comprises a second geometric attention operation, a cross-attention operation, a third feedforward network operation and a second normalization batch operation, and the second geometric attention operation is composed of a second semantic feature output branch and a second geometric feature output branch;

[0132] S1303-H, the above refined feature F c (D+1) is subjected to the above neural network operation, to obtain the dense point cloud.

[0133] For example, the global information of the global feature output by the target encoder is extracted by using the max-pooling operation, and then the global information is converted into the second center point data c with a size of N and the region feature F with a size of Nc Region feature F of ×C4 region . Then, the second center point data is embedded into the region feature F region to form a coarse shape feature sequence F of size c (1) , and the coarse shape feature sequence F c (1) and the second center point data are fed into a target decoder for D times of third iteration operations to further refine the features, wherein denotes the feature dimension of the coarse shape feature sequence and is a positive integer.

[0134] The coarse shape feature sequence F c (1) is taken as the first-layer input shape feature of the target decoder, and D times of third iteration operations are performed by the target decoder to obtain the refined feature F c (D+1) . The target decoder includes D layers, each layer performing one time of third iteration operation, and the third iteration operation includes a second geometric attention operation, a cross-attention operation, a third feedforward network operation, and a second normalization batch operation, wherein the second geometric attention operation is composed of a second semantic feature output branch and a second geometric feature output branch.

[0135] The process of performing the third iteration operation in the first layer of the target decoder is as follows:

[0136] For the first-layer input shape feature F c (1) of the target decoder, firstly, the geometric similarity between the coarse shape feature sequences is explored through the second semantic feature output branch and the second geometric feature output branch of the second geometric attention operation; then, the output features of the second geometric attention operation and the global features output by the target encoder are taken as the inputs of the cross-attention operation respectively to explore the structural relationship between them, so as to obtain features with refined structural details, and the features are further updated through the third feedforward network operation and the second normalization batch operation to obtain the first-layer output shape feature F c (2) .

[0137] The first-layer output shape feature F c (2) continues to perform (D-1) times of third iteration operations through the remaining (D-1) layers of the target decoder to obtain the refined feature F c (D+1) output by the last layer of the target decoder.

[0138] The refined feature Fc (D+1) A neural network operation is used to recover a size N with a complete shape. cp Dense point cloud of ×3.

[0139] S1304. Perform the above structural feature extraction operation on the above dense point cloud to obtain the target structural feature F. cp ;

[0140] For example, for the recovered dense point cloud, the PointNet++ network is used to extract target structural features from the recovered complete shape point cloud. First, random sampling is performed from N cp N3 points are sampled, and n neighborhood points are searched within a specified radius for each sampled point as the center point. Then, the features within the neighborhood of the center point are aggregated using the second PointNet network to obtain a target structure feature F of size N3×C3 containing complete shape information. cp .

[0141] S1305, Apply the above-mentioned region of interest features F roi and the above target structural features F cp Perform the above feature fusion operation to obtain the above aggregated features.

[0142] In one implementation, the feature fusion operation includes a second splicing operation, a spatial attention operation, a channel attention operation, an element-wise multiplication operation, an activation operation, and a weighted summation operation. Step S1305 specifically includes S1305-A to S1305-E:

[0143] S1305-A, The above-mentioned region of interest features F roi and the above target structural features F cp Perform the second splicing operation described above to obtain the splicing feature F. cat ;

[0144] S1305-B, the above splicing feature F cat Perform the above spatial attention operations to obtain spatial attention features F. spatial ;

[0145] S1305-C, the above splicing feature F cat Perform the above channel attention operation to obtain the channel attention feature F. channel ;

[0146] S1305-D, the above spatial attention features F spatial And the aforementioned channel attention feature F channel Perform the element-wise multiplication and activation operations described above to obtain the feature weights W. e ;

[0147] S1305-E, utilizing the above feature weight W e to the above region of interest feature F roi and the above target structure feature F cp performs the above weighted summation operation to obtain the above aggregated feature.

[0148] To effectively utilize the complete structure information contained in the above target structure feature F cp , the present application designs a feature fusion operation based on mixed attention, which integrates the target structure feature F cp into the region of interest feature F roi to enhance the geometric features of the target for box refinement, thereby generating accurate target detection boxes.

[0149] Specifically, first, the region of interest feature F roi of size N3xC3 is spliced with the target structure feature F cp of size N3xC3 to obtain a spliced feature F cat of size N3xC6, and F cat is sent into the spatial attention branch and the channel attention branch respectively to generate the corresponding spatial attention feature F spatial and the channel attention feature F channel , wherein C6 represents the feature dimension of the spliced feature and is a positive integer.

[0150] In the spatial attention branch, a point-by-point pooling operation is used to compress the channel dimension, and a linear layer is used to obtain the spatial attention feature F spatial of size N3x1. In the channel attention branch, a global average pooling is used to compress the spatial dimension, and a first convolutional layer with a channel restoration ratio of r is used to compress and restore the channel dimension to generate a global context feature of size 1xC6; at the same time, a second convolutional layer is used to further learn the local relationship between channels to obtain a local context feature of size N3xC6. The global context feature and the local context feature are added element by element to obtain the channel attention feature F channel of size N3xC6.

[0151] Subsequently, the spatial attention feature F spatial is multiplied element by element with the channel attention feature F channel , and then an activation operation is performed to obtain the feature weight W e . The region of interest feature F e and the target structure feature F roi are multiplied by the feature weight W cp respectively.The weighted features are then element-wise summed to obtain a size of N3xC6 aggregated feature.

[0152] S140, performing a class information prediction operation and a bounding box regression operation in the point cloud object detection network according to the aggregated feature to obtain an object class score and a bounding box parameter residual, and calculating a total loss value based on the initial candidate bounding box parameter, the object class score and the bounding box parameter residual to update the parameters of the point cloud object detection network to obtain a point cloud object detection model;

[0153] In some embodiments, step S140 further comprises steps S1401-S1403:

[0154] S1401, using the class information prediction operation in the point cloud object detection network to obtain the object class score from the aggregated feature;

[0155] S1402, using the bounding box regression operation in the point cloud object detection network to obtain the bounding box parameter residual from the aggregated feature;

[0156] S1403, calculating the total loss value based on the initial candidate bounding box parameter, the object class score and the bounding box parameter residual to update the parameters of the point cloud object detection network to obtain the point cloud object detection model.

[0157] For example, the total loss function L of the point cloud object detection network total including a candidate bounding box generation loss L RPN and a candidate bounding box refinement loss L refine The candidate bounding box generation loss L RPN is composed of a segmentation loss L seg , a foreground point regression loss L part and a candidate bounding box regression loss L box , as shown in equation (7):

[0158]

[0159] where λ represents the loss weight, N f represents the number of foreground points, N pos represents the number of foreground bounding boxes, L seg represents the segmentation loss realized by the focal loss function, L part represents the foreground point regression loss obtained by calculating the positional deviation value between the foreground points inside the initial candidate bounding box and the foreground points inside the ground truth bounding box through the binary cross-entropy loss function, L box represents the candidate bounding box regression loss obtained by calculating the deviation between the initial candidate bounding box parameter residual and the ground truth bounding box parameter residual through the smooth L1 loss function.

[0160] candidate box refinement loss L refine class information prediction loss L cls and refinement box regression loss L reg consisting of formula (8):

[0161]

[0162] wherein M box represents the number of foreground refinement boxes, L cls represents the class information prediction loss calculated by the binary cross-entropy loss function to calculate the deviation between the target class score and the class label of the ground truth box; L reg represents the refinement box regression loss calculated by the smooth L1 loss function to calculate the deviation between the foreground refinement box parameter residual and the ground truth box parameter residual.

[0163] candidate box generation loss L RPN and candidate box refinement loss L refine to a given number of iterations, and finally generate a point cloud target detection model with optimal parameters of the point cloud target detection network.

[0164] S150, using the above point cloud target detection model to detect the sample set to be detected, generating point cloud target detection boxes and point cloud target class information, completing point cloud target detection.

[0165] For example, a sample set to be detected is obtained, and the above point cloud target detection model is used to detect point cloud data with a size of N0xC0 in the sample set to be detected. First, step S110 is performed to extract enhanced voxel features from the point cloud data using the backbone network; then step S120 is performed to obtain point-by-point features from the enhanced voxel features, and the initial candidate box parameters are extracted from the point-by-point features using the region candidate network; based on step S130, the non-maximum suppression operation, the region of interest pooling operation, the point cloud completion operation, the structure feature extraction operation and the feature fusion operation in the point cloud target detection network are used to obtain the aggregated features based on the initial candidate box parameters; finally, step S140 is performed to obtain the target class score and the box parameter residual by using the class information prediction operation and the box regression operation on the aggregated features, and the total loss value is calculated based on the initial candidate box parameters, the target class score and the box parameter residual to update the parameters of the point cloud target detection network, to obtain the point cloud target detection model; the point cloud target detection model is used to detect the sample set to be detected, to generate point cloud target detection boxes and point cloud target class information, and to complete point cloud target detection.

[0166] In some examples, the voxel-based detection network Part-A 2The receptive field of the one-stage backbone network is limited to the size of the three-dimensional sparse convolution kernel, which is difficult to effectively capture context information, resulting in the problem of being unable to generate accurate initial candidate boxes, the present application first introduces voxelization operation, context information enhancement encoder and up-sampling operation in the one-stage of the network, and constructs a novel backbone network for accurately extracting initial candidate box parameters. Then, in the two-stage of the network, the present application designs a point cloud completion operation for restoring the complete shape of the target in the candidate box and extracting the corresponding target structure features, and then constructs a feature fusion operation based on hybrid attention, which integrates the restored target structure features into the region of interest features to enhance the geometric information of the target for box refinement. Benefiting from the design of the above backbone network and point cloud completion operation, the present application can effectively reconstruct the complete shape of the target in the complex point cloud scene to enhance the geometric feature expression of the point cloud target, thereby improving the overall detection performance of the point cloud target detection network.

[0167] To verify the effectiveness of the point cloud target detection method based on structure information enhancement proposed by the present application, the point cloud target detection network constructed by the method of the present application is trained using the KITTI dataset to obtain a point cloud target detection model, and the point cloud target detection model is evaluated and tested using the KITTI dataset. The KITTI dataset provides 7481 training samples and 7518 test samples. In the implementation process of the point cloud target detection method based on structure information enhancement of the present application, 3712 samples are divided from the 7481 training samples to form a training sample set, and 3769 samples are divided to form a first to-be-detected sample set. The 7518 test samples are used as a second to-be-detected sample set.

[0168] Figure 3 And Figure 4 The comparison results of the average detection accuracy of the targets in the first to-be-detected sample set and the second to-be-detected sample set under three difficulty levels are given respectively. Figure 3 And Figure 4 Consistently show that compared with other methods, the present application obtains the highest average detection accuracy under three difficulty levels, which shows that the present application can effectively improve the overall detection performance of the network, thereby improving the target detection accuracy in complex scenes, proving the effectiveness and practicality of the present application.

[0169] To visually display the detection effect of the method of the present application, the Part-A 2 , IASSD and the visualization results of the method of the present application on the first to-be-detected sample set are given in Figure 5 , wherein the ellipses indicate undetected targets, and the arrows indicate detected targets. From Figure 5It can be seen that the method can accurately detect the long-distance and occluded targets indicated by the ellipse and the arrow, while other detection methods will ignore or incorrectly detect these targets, which fully proves that the application has good detection performance in a complex three-dimensional point cloud scene containing a large number of long-distance and occluded targets.

[0170] In summary, the point cloud target detection method based on structure information enhancement proposed in the application expands the voxel space by aggregating the spatial information of empty voxels and non-empty voxels, so that the network fully captures the context information in a larger receptive field to generate high-quality initial candidate boxes. At the same time, the application proposes a novel point cloud completion operation, which is introduced into the point cloud target detection network to fully explore the global structural association of the target point cloud, thereby predicting the missing part of the target and supplementing the complete shape information for the incomplete target point cloud. The application also constructs a feature fusion operation based on hybrid attention, which integrates the reconstructed target shape information into the spatial features of the target, thereby enhancing the geometric feature expression ability of the target and effectively improving the target detection performance in complex scenes.

[0171] Please refer to Figure 6 An embodiment of the point cloud target detection device based on structure information enhancement in the application embodiment can include:

[0172] The first extraction unit 21 is configured to obtain a training sample set and input the training sample set into a point cloud target detection network, and extract enhanced voxel features of point cloud data in the training sample set based on a backbone network in the point cloud target detection network, wherein the backbone network includes a voxelization operation, a context information enhancement encoder and an up-sampling operation.

[0173] The second extraction unit 22 is configured to obtain point-by-point features based on the enhanced voxel features, and extract initial candidate box parameters based on the point-by-point features and a region candidate network in the point cloud target detection network.

[0174] The first acquisition unit 23 is configured to obtain aggregated features by using a non-maximum suppression operation, a region of interest pooling operation, a point cloud completion operation, a structure feature extraction operation and a feature fusion operation in the point cloud target detection network on the initial candidate box parameters.

[0175] The second acquisition unit 24 is configured to perform a class information prediction operation and a box regression operation in the point cloud target detection network based on the aggregated features to obtain target class scores and box parameter residuals, and calculate a total loss value based on the initial candidate box parameters, the target class scores and the box parameter residuals to update parameters of the point cloud target detection network to obtain a point cloud target detection model.

[0176] The generating unit 25 is configured to detect the sample set to be detected by using the point cloud target detection model, to generate a point cloud target detection frame and point cloud target category information, and to complete point cloud target detection.

[0177] As shown in Figure 7 The electronic device 300 according to the embodiment of the present application includes a memory 310, a processor 320, and a computer program 311 stored in the memory 310 and executable on the processor. When the processor 320 executes the computer program 311, any step of the point cloud target detection method based on structure information enhancement described above is implemented.

[0178] Since the electronic device according to the embodiment of the present application is a device used to implement the point cloud target detection device based on structure information enhancement according to the embodiment of the present application, the specific implementation of the electronic device according to the embodiment of the present application and various changes thereof can be understood by those skilled in the art based on the method described in the embodiment of the present application. Therefore, how the electronic device implements the method according to the embodiment of the present application will not be described in detail, and any device used to implement the method according to the embodiment of the present application belongs to the scope of protection of the present application.

[0179] In the specific implementation process, the computer program 311 can implement Figure 1 any embodiment of the corresponding embodiment.

[0180] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and the parts not described in detail in an embodiment can be referred to the related description of other embodiments.

[0181] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0182] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart Figure 1 one or more flows and / or blocks. Figure 1 one or more flows and / or blocks.

[0183] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart Figure 1 one or more flows and / or blocks. Figure 1 one or more flows and / or blocks.

[0184] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart Figure 1 one or more flows and / or blocks. ​ one or more flows and / or blocks.

[0185] The embodiments of the present application also provide a computer program product, which comprises computer software instructions, when the computer software instructions are run on a processing device, causing the processing device to execute the flow of the point cloud target detection method based on structure information enhancement in the corresponding embodiments.

[0186] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the processes or functions according to the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that the computer can store or be integrated into a data storage device such as a server, data center, etc. containing one or more available media. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

[0187] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0188] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0189] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0190] In addition, each of the functional units in the various embodiments of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0191] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in part, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the various embodiments of the method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and various other media that can store program codes.

[0192] The above, the above embodiments are only to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present application.

Claims

1. A point cloud object detection method based on structure information enhancement, characterized in that, The method comprises: obtaining a training sample set, inputting into a point cloud target detection network, extracting enhanced voxel features of point cloud data in the training sample set based on a backbone network in the point cloud target detection network, wherein the backbone network comprises a voxelization operation, a context information enhancement encoder and an up-sampling operation; obtaining point-by-point features according to the enhanced voxel features, and extracting initial candidate box parameters based on a region candidate network in the point cloud target detection network; performing a non-maximum suppression operation, a region of interest pooling operation, a point cloud completion operation, a structure feature extraction operation and a feature fusion operation on the initial candidate box parameters in the point cloud target detection network to obtain aggregated features; performing a non-maximum suppression operation, a region of interest pooling operation, a point cloud completion operation, a structure feature extraction operation and a feature fusion operation on the initial candidate box parameters in the point cloud target detection network to obtain aggregated features, comprising: performing the non-maximum suppression operation on the initial candidate box parameters to obtain screened candidate boxes; performing the region of interest pooling operation on the screened candidate boxes to obtain region of interest features; performing the point cloud completion operation on the screened candidate boxes to obtain dense point clouds; performing the structure feature extraction operation on the dense point clouds to obtain target structure features; performing the feature fusion operation on the region of interest features and the target structure features to obtain the aggregated features; performing a class information prediction operation and a box regression operation in the point cloud target detection network according to the aggregated features to obtain target class scores and box parameter residuals, and calculating a total loss value based on the initial candidate box parameters, the target class scores and the box parameter residuals to update parameters of the point cloud target detection network to obtain a point cloud target detection model; detecting a to-be-detected sample set by using the point cloud target detection model to generate point cloud target detection boxes and point cloud target class information, and completing point cloud target detection.

2. The point cloud target detection method based on structure information enhancement according to claim 1, wherein extracting enhanced voxel features of point cloud data in the training sample set based on a backbone network in the point cloud target detection network comprises: obtaining voxel data and voxel features based on the voxelization operation according to the point cloud data in the training sample set; enhancing the voxel data and the voxel features by the context information enhancing encoder H a first iterative operation to obtain enhanced voxel data and attention enhanced features, wherein the context information enhancing encoder comprises H a layer, H wherein the first iterative operation is performed once per layer for a positive integer greater than or equal to 2. performing the up-sampling operation according to the enhanced voxel data and the attention enhancement features to obtain the enhanced voxel features. 3.The method of claim 2, wherein, The first iterative operation comprises a down-sampling operation, an aggregation expansion operation, a local attention range enhancement weighting operation, an inflation attention range enhancement weighting operation, a first splicing operation and a first feedforward network operation, The encoding the voxel data and the voxel features by the context information enhancement encoder H The first iteration operation to obtain the enhanced voxel data and the attention enhanced features comprises: the voxel data and the voxel features are taken as first-layer input voxel data and first-layer input voxel features of the context information enhancement encoder; performing the down-sampling operation and the aggregation expansion operation on the first-layer input voxel data and the first-layer input voxel feature of the context information enhanced encoder to obtain expanded non-empty voxels and expanded non-empty voxel features, and taking the expanded non-empty voxels as first-layer output voxel data; performing the local attention range enhanced weighting operation and the inflation attention range enhanced weighting operation on the expanded non-empty voxels to obtain first-layer local attention weighted voxel features and first-layer inflation attention weighted voxel features; performing the first splicing operation and the first feed-forward network operation on the first-layer local attention weighted voxel features and the first-layer inflation attention weighted voxel features to obtain first-layer output voxel features; output voxel data of the first layer and the output voxel features of the first layer through the remaining layers of the context information enhancement encoder H -1 layer continues to execute H -1 iteration operation to obtain enhanced voxel data and attention enhanced features, wherein the enhanced voxel data and the attention enhanced features are output by the last layer of the context information enhancement encoder. 4.The method of claim 1, wherein, the point cloud completion operation comprises a farthest point sampling operation, an extraction of a center point neighborhood range feature operation, a first embedding operation, a second iterative operation, a max-pooling operation, a multi-layer perceptron, a second embedding operation, a third iterative operation and a neural network operation, performing the point cloud completion operation on the screening candidate box to obtain a dense point cloud, comprising: performing the farthest point sampling operation on the screening candidate box to obtain first center point data; performing the extraction of a center point neighborhood range feature operation on the first center point data to obtain neighborhood features; performing the first embedding operation on the first center point data and the neighborhood features to obtain a position feature sequence; inputting the position feature sequence and the first center point data into a target encoder to perform D a second iterative operation to obtain global features, wherein the target encoder comprises D a layer, D wherein k is a positive integer greater than or equal to 2, the second iterative operation is performed once for each layer, and the second iterative operation comprises a first geometric attention operation, a second feedforward network operation, and a first normalization batch operation, the first geometric attention operation comprises a first semantic feature output branch and a first geometric feature output branch; performing the max-pooling operation and the multi-layer perceptron on the global feature to obtain second center point data and region features; performing the second embedding operation on the second center point data and the region features to obtain a coarse shape feature sequence; inputting the coarse shape feature sequence and the second center point data into a target decoder to perform D a third iterative operation to obtain refined features, wherein the target decoder comprises D a layer, D wherein k is a positive integer greater than or equal to 2, the third iterative operation is performed once for each layer, and the third iterative operation comprises a second geometric attention operation, a cross-attention operation, a third feedforward network operation, and a second normalization batch operation, and the second geometric attention operation is composed of a second semantic feature output branch and a second geometric feature output branch. performing the neural network operation on the refined feature to obtain the dense point cloud.

5. The method of claim 1, wherein, the feature fusion operation comprises a second splicing operation, a spatial attention operation, a channel attention operation, an element-wise multiplication operation, an activation operation and a weighted summation operation, performing the feature fusion operation on the region of interest feature and the target structure feature to obtain the aggregated feature, comprising: performing the second splicing operation on the region of interest feature and the target structure feature to obtain a spliced feature; performing the spatial attention operation on the spliced feature to obtain a spatial attention feature; performing the channel attention operation on the spliced feature to obtain a channel attention feature; performing the element-wise multiplication operation and the activation operation on the spatial attention feature and the channel attention feature to obtain a feature weight; performing the weighted summation operation on the region of interest feature and the target structure feature by using the feature weight to obtain the aggregated feature.

6. The structure information enhanced point cloud target detection method according to claim 1, wherein, The class information prediction operation and the bounding box regression operation in the point cloud target detection network are performed on the aggregated features to obtain a target class score and a bounding box parameter residual, and a total loss value is calculated based on the initial candidate bounding box parameters, the target class score, and the bounding box parameter residual to update the parameters of the point cloud target detection network to obtain a point cloud target detection model, including: The class information prediction operation in the point cloud target detection network is performed on the aggregated features to obtain the target class score; The bounding box regression operation in the point cloud target detection network is performed on the aggregated features to obtain the bounding box parameter residual; The total loss value is calculated based on the initial candidate bounding box parameters, the target class score, and the bounding box parameter residual to update the parameters of the point cloud target detection network to obtain the point cloud target detection model.

7. A point cloud target detection device based on structural information enhancement, characterized in that, including: A first extraction unit is configured to obtain a training sample set, input the training sample set into a point cloud target detection network, and extract enhanced voxel features of point cloud data in the training sample set based on a backbone network in the point cloud target detection network, wherein the backbone network includes a voxelization operation, a context information enhancement encoder, and an up-sampling operation; A second extraction unit is configured to obtain point-by-point features based on the enhanced voxel features, and extract initial candidate bounding box parameters based on the point-by-point features and a region candidate network in the point cloud target detection network; A first obtaining unit is configured to obtain aggregated features by using a non-maximum suppression operation, a region of interest pooling operation, a point cloud completion operation, a structure feature extraction operation, and a feature fusion operation in the point cloud target detection network on the initial candidate bounding box parameters; The non-maximum suppression operation is performed on the initial candidate bounding box parameters to obtain screened candidate bounding boxes; The region of interest pooling operation is performed on the screened candidate bounding boxes to obtain region of interest features; The point cloud completion operation is performed on the screened candidate bounding boxes to obtain dense point clouds; The structure feature extraction operation is performed on the dense point clouds to obtain target structure features; The feature fusion operation is performed on the region of interest features and the target structure features to obtain the aggregated features; A second obtaining unit is configured to perform a class information prediction operation and a bounding box regression operation in the point cloud target detection network on the aggregated features to obtain a target class score and a bounding box parameter residual, and calculate a total loss value based on the initial candidate bounding box parameters, the target class score, and the bounding box parameter residual to update the parameters of the point cloud target detection network to obtain a point cloud target detection model; A generating unit is configured to detect a to-be-detected sample set by using the point cloud target detection model to generate a point cloud target detection bounding box and point cloud target class information, and complete point cloud target detection. ​ 8. An electronic device comprising: Memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is configured to implement the steps of the method for point cloud object detection based on structure information enhancement according to any one of claims 1-6 when executing the computer program stored in the memory.

9. A computer readable storage medium having stored thereon a computer program, characterized in that: The computer program is configured to implement the method for point cloud object detection based on structure information enhancement according to any one of claims 1-6 when executed by the processor.

Images