Three-dimensional target detection algorithm based on pyramid component feature fusion

Abstract

The invention discloses a pyramid component feature fusion-based three-dimensional target detection algorithm and device. The method comprises the steps of obtaining an initial target bounding box; processing each feature image and each initial target bounding box by using a preset pyramid part construction algorithm to obtain a plurality of pyramid features corresponding to each initial target bounding box; classifying the initial target bounding box based on a pre-acquired target confidence score, a distance between a target and a sensor and a point cloud density by using a preset pyramid component feature enhancement algorithm to obtain a plurality of high-confidence target boxes and a plurality of low-confidence target boxes, and obtaining a plurality of high-confidence target boxes and a plurality of low-confidence target boxes based on cross attention between pyramid components; the pyramid features in the high-confidence target frame are utilized to strengthen the pyramid features in the low-confidence target frame, the strengthened features of the three-dimensional target to be detected are obtained, the pyramid component features of the sparse target are strengthened through the algorithm, and it is ensured that the sparse target can be recognized more accurately in the detection process through special design and optimization.

Description

Technical Field

[0001] This application relates to the field of 3D object detection technology, and particularly to a 3D object detection algorithm and device based on pyramid component feature fusion. Background Art

[0002] With the rapid development of computer vision technology, 3D object detection technology has shown great application potential in fields such as autonomous driving, robot navigation, and augmented reality. Currently, 3D object detection technology mainly relies on deep learning algorithms to extract 3D information in the scene through neural networks, and then realizes the localization and classification of objects. However, despite many breakthroughs in 3D object detection technology, there are still many challenges in practical applications. Among them, one of the most prominent problems is the poor detection effect for sparse objects in the scene. Sparse objects, such as distant vehicles, pedestrians, etc., due to their relatively scattered appearance and unclear features in the point cloud data, often lack sufficient information support during the feature extraction process, resulting in a significant drop in detection accuracy. The main reason is that existing 3D object detectors do not specifically optimize for sparse objects during the feature extraction stage. These detectors usually adopt general feature learning strategies and lack an explicit reinforcement mechanism for sparse object features. Therefore, at the feature representation level, sparse objects are often submerged in a large amount of background noise and are difficult to be effectively distinguished and recognized. Summary of the Invention

[0003] The main purpose of this application is to provide a 3D object detection algorithm and device based on pyramid component feature fusion, aiming to solve the technical problems of the 3D object detection algorithm based on pyramid component feature fusion.

[0004] To achieve the above object, the present application provides a three-dimensional object detection algorithm based on pyramid component feature fusion, including: processing multi-view images of an object to be detected through an image feature extraction network to obtain respective feature images; processing point cloud data of the object to be detected through a point cloud feature extraction network to obtain a plurality of initial object bounding boxes; using a preset pyramid component construction algorithm to process each feature image and each initial object bounding box to obtain a plurality of pyramid features corresponding to each initial object bounding box; using a preset pyramid component feature enhancement algorithm to classify the initial object bounding boxes based on a pre-obtained object confidence score, the distance between the object and the sensor, and the point cloud density to obtain a plurality of high-confidence object bounding boxes and a plurality of low-confidence object bounding boxes, and based on the cross-attention between pyramid components, using the pyramid features within the high-confidence object bounding boxes to enhance the pyramid features within the low-confidence object bounding boxes, and, using the pyramid features within the pooled low-confidence object bounding boxes to obtain low-confidence object features, and according to the pyramid features within the pooled and self-attention processed high-confidence object bounding boxes, obtaining high-confidence object features, and based on the low-confidence object features and the high-confidence object features, obtaining enhanced features of the three-dimensional object to be detected.

[0005] Optionally, the step of using a preset pyramid component construction algorithm to process each feature image and each initial object bounding box to obtain a plurality of pyramid features corresponding to each initial object bounding box includes: using a preset pyramid component construction algorithm to process each feature image and each initial object bounding box to obtain six pyramid components corresponding to each initial object bounding box; extracting multi-modal features of the point cloud included in each pyramid component, where the multi-modal features include local geometric features, image projection features, and neighborhood aggregation geometric features; performing feature concatenation on the local geometric features, image projection features, and neighborhood aggregation geometric features and then performing linear layer processing to obtain the final point cloud features of each pyramid component; querying each final point cloud feature through a preset pyramid feature query algorithm in combination with a cross-attention mechanism to obtain pyramid features.

[0006] Optionally, the object bounding box is a cuboid structure, and the step of using a preset pyramid component construction algorithm to process each feature image and each initial object bounding box to obtain six pyramid components corresponding to each initial object bounding box includes: connecting the center of each cuboid structure to the eight vertices of the cuboid structure to obtain six pyramid components.

[0007] Optionally, the process of extracting the local geometric features in the multi-modal features of the point cloud included in each pyramid component includes: transforming the six vertices of each pyramid component and the point cloud set data inside from the current coordinate system to the regular coordinate system of the target to which the six vertices and the point cloud set data inside belong; calculating the relative offsets between each point in the point cloud set data inside each pyramid component and each vertex of the pyramid based on a preset offset calculation formula to obtain six offset values for each point inside each pyramid component; cascading the six offset values of each point inside each pyramid component to obtain the cascaded features of each point; and processing the cascaded features of each point using a multi-layer perception structure to obtain the local features in the multi-modal features of the point cloud inside each pyramid component.

[0008] Optionally, the process of extracting the image projection features in the multi-modal features of the point cloud included in each pyramid component includes: projecting the point cloud set data inside each pyramid component onto each feature image to obtain point cloud projection data; and then extracting the point cloud projection data through bilinear sampling to obtain the image projection features of the point cloud inside each pyramid component.

[0009] Optionally, the process of extracting the neighborhood aggregation geometric features in the multi-modal features of the point cloud included in each pyramid component includes: using the PointNet structure to aggregate the point cloud data of each point in a spherical neighborhood with a preset radius near each point inside each pyramid component to generate the neighborhood aggregation geometric features of the point cloud inside each pyramid component.

[0010] Optionally, querying each final point cloud feature through a preset pyramid feature query algorithm combined with a cross-attention mechanism to obtain pyramid features includes: processing the final point cloud features inside each pyramid component using a preset pyramid feature query formula to obtain pyramid features, where the pyramid feature query formula is obtained based on the sum value of the parameters to be learned and the multi-head attention features, and the multi-head attention features are obtained by processing the product of the parameters to be learned and the final point cloud features using the multi-head attention algorithm.

[0011] Optionally, using the pyramid features in the high-confidence target boxes to strengthen the pyramid features in the low-confidence target boxes, and using the pyramid features in the low-confidence target boxes after pooling to obtain low-confidence target features includes: generating a first one-hot encoding based on multiple high-confidence target boxes; generating a second one-hot encoding based on multiple low-confidence target boxes, and obtaining a mask between pyramid components based on the outer product of the first one-hot encoding and the second one-hot encoding; obtaining a first feature based on the mask between pyramid components and the cross-attention between the high-confidence target boxes and the low-confidence target boxes; pooling the pyramid features in the low-confidence target boxes to obtain a second feature; pooling the pyramid features in the high-confidence target boxes to obtain a third feature; and adding the first feature, the second feature, and the third feature to obtain the low-confidence target features.

[0012] Optionally, obtaining the high-confidence target features according to the pyramid features within the high-confidence target boxes after pooling and self-attention processing includes: performing pooling processing on the pyramid features within each high-confidence target box to obtain a fourth feature; and using a self-attention algorithm to process the fourth feature to obtain the high-confidence target features.

[0013] To this end, on the basis of the above embodiments, the present application further proposes a three-dimensional object detection device based on pyramid component feature fusion, including: an image feature acquisition module for processing multi-view images of a target to be detected based on an image feature extraction network to obtain respective feature images; a bounding box acquisition module for processing point cloud data of the target to be detected based on a point cloud feature extraction network to obtain a plurality of initial target bounding boxes; a pyramid component construction module for using a preset pyramid component construction algorithm to process each feature image and each initial target bounding box to obtain a plurality of pyramid features corresponding to each initial target bounding box; a pyramid component feature enhancement module for using a preset pyramid component feature enhancement algorithm to classify the initial target bounding boxes based on a pre-acquired target confidence score, the distance between the target and the sensor, and the point cloud density to obtain a plurality of high-confidence target boxes and a plurality of low-confidence target boxes, strengthening the pyramid features within the low-confidence target boxes by using the pyramid features within the high-confidence target boxes based on the cross-attention between the pyramid components, and obtaining low-confidence target features by using the pyramid features within the pooled low-confidence target boxes, obtaining high-confidence target features according to the pyramid features within the high-confidence target boxes after pooling and self-attention processing, and obtaining enhanced features of the three-dimensional target to be detected based on the low-confidence target features and the high-confidence target features.

[0014] A 3D object detection algorithm and device based on pyramid component feature fusion proposed in an embodiment of the present application processes multi-view images of an object to be detected through an image feature extraction network to obtain respective feature images; processes the point cloud data of the object to be detected through a point cloud feature extraction network to obtain a plurality of initial object bounding boxes; uses a preset pyramid component construction algorithm to process each feature image and each initial object bounding box to obtain a plurality of pyramid features corresponding to each initial object bounding box; uses a preset pyramid component feature enhancement algorithm to classify the initial object bounding boxes based on pre-obtained object confidence scores, the distance between the object and the sensor, and the point cloud density to obtain a plurality of high-confidence object boxes and a plurality of low-confidence object boxes, and based on the cross-attention between pyramid components, uses the pyramid features within the high-confidence object boxes to enhance the pyramid features within the low-confidence object boxes, and, uses the pyramid features within the pooled low-confidence object boxes to obtain low-confidence object features, and based on the pyramid features within the pooled and self-attention processed high-confidence object boxes, obtains high-confidence object features, and based on the low-confidence object features and the high-confidence object features, obtains enhanced features of the 3D object to be detected. This algorithm effectively fuses component features of different modalities by constructing pyramid components, thereby enhancing the feature representation of sparse objects. At the same time, the present invention focuses on enhancing the pyramid component features of sparse objects, and through special design and optimization, ensures that sparse objects can be more accurately recognized during the detection process. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic flowchart provided for an embodiment of the 3D object detection algorithm based on pyramid component feature fusion of the present application;

[0016] Figure 2 It is a flowchart of the pyramid component construction algorithm provided for an embodiment of the 3D object detection algorithm based on pyramid component feature fusion of the present application;

[0017] Figure 3 It is a schematic flowchart of the pyramid component feature enhancement algorithm provided for an embodiment of the 3D object detection algorithm based on pyramid component feature fusion of the present application;

[0018] Figure 4 It is a functional module structure block diagram provided for an embodiment of the 3D object detection device based on pyramid component feature fusion of the present application.

[0019] The realization, functional characteristics, and advantages of the purpose of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0020] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0021] To overcome the deficiencies of the prior art, the present invention provides a three-dimensional object detection algorithm based on pyramid component feature fusion. It belongs to a three-dimensional object refinement technology, can be combined with various region proposal networks, and evenly divides each object bounding box generated by the region proposal network into six pyramid components. Then, it aggregates the nearby point cloud features through the point cloud inside the pyramid and fuses the image projection features near their image projection positions, and then queries the corresponding pyramid features from these point cloud features. Finally, the detection algorithm classifies the objects in the scene into high-confidence objects and low-confidence objects according to the number of point clouds contained in the object, the distance of the object from the sensor, and the confidence score of the object. Then, it uses the pyramid component features of the high-confidence objects to strengthen the pyramid component features of the low-confidence objects. After the pyramid component features of the low-confidence objects are strengthened and fused with the corresponding object features, they are sent to a target parameter regression head, so as to achieve the purpose of refining the object bounding box.

[0022] Referring to Figure 1 , the three-dimensional object detection algorithm based on pyramid component feature fusion provided by the first embodiment of the present application may include:

[0023] S10. Process the multi-view images of the object to be detected based on an image feature extraction network to obtain respective feature images;

[0024] Process the multi-view images of the object to be detected based on an image feature extraction network to obtain respective feature images. Specifically, the image feature extraction network can adopt architectures such as convolutional neural networks, and perform feature extraction operations on the multi-view images of the object to be detected taken from different angles. For example, key features in the images are gradually extracted through components such as convolutional layers and pooling layers, so as to obtain image features that can represent the multi-view images.

[0025] S20. Process the point cloud data of the object to be detected based on a point cloud feature extraction network to obtain a plurality of initial object bounding boxes;

[0026] It should be noted that the point cloud feature extraction network can be a voxel-based network structure, etc. By processing the point cloud data of the object to be detected obtained by devices such as lidar, the spatial distribution and other characteristics of the point cloud data are analyzed, and then the initial object bounding boxes corresponding to the point cloud data are determined. This bounding box preliminarily defines the position and approximate range of the object to be detected in space.

[0027] S30. Use a preset pyramid component construction algorithm to process each feature image and each initial object bounding box to obtain a plurality of pyramid features corresponding to each initial object bounding box;

[0028] Exemplarily, the algorithm can combine information such as texture and shape in the image features, as well as information such as the position and size of the initial target bounding box, to construct pyramid features according to a certain hierarchical structure. For example, features are gradually abstracted and integrated from the bottom layer to the top layer, so that the pyramid features can reflect the characteristics of the target to be detected at different scales.

[0029] S40. Use the preset pyramid component feature enhancement algorithm to classify the initial target bounding box based on the pre-acquired target confidence score, the distance between the target and the sensor, and the point cloud density to obtain multiple high-confidence target boxes and multiple low-confidence target boxes. Based on the cross-attention between pyramid components, use the pyramid features in the high-confidence target box to enhance the pyramid features in the low-confidence target box, and use the pyramid features in the pooled low-confidence target box to obtain low-confidence target features. According to the pyramid features in the pooled and self-attention processed high-confidence target box, obtain high-confidence target features. Based on the low-confidence target features and high-confidence target features, obtain the enhanced features of the three-dimensional target to be detected.

[0030] It can be understood that the pre-acquired target confidence score can be obtained through previous training and detection evaluation. The distance between the target and the sensor can reflect the distance information of the target, and the point cloud density reflects the density of the point cloud data. Based on this information, the algorithm can judge the reliability of the target in the initial target bounding box and classify it into high-confidence target boxes and low-confidence target boxes. Finally, the target bounding box and the category to which the target belongs are used as the final three-dimensional detection result.

[0031] Among them, the cross-attention mechanism between pyramid components enables the pyramid features in the high-confidence target box to interact with the pyramid features in the low-confidence target box. Through this interaction, more reliable features in the high-confidence target box can be transmitted to the low-confidence target box, and at the same time, a pooling operation is performed on the pyramid features in the low-confidence target box to further integrate the features, and finally low-confidence target features are obtained.

[0032] Specifically, perform a pooling operation on the pyramid features in the high-confidence target box to compress the feature dimension and retain key information, and then use the self-attention mechanism to perform weight distribution on the features in their own dimensions to highlight important features, so as to obtain more representative high-confidence target features.

[0033] It should be noted that the above-obtained low-confidence target features and high-confidence target features are fused and processed to combine the advantages of both. For example, combine the detailed information in the low-confidence target features and the reliable features in the high-confidence target features, so as to obtain enhanced features that can more accurately describe the three-dimensional target to be detected.

[0034] In an embodiment of the present application, step S20 may include the following execution process:

[0035] S201. Process each feature image and each initial target bounding box using a preset pyramid component construction algorithm to obtain six pyramid components corresponding to each initial target bounding box;

[0036] It should be noted that this step can obtain six pyramid components corresponding to each initial target bounding box. This process is carried out in the target pyramid component construction module, and the input image features and initial target bounding boxes are processed through a specific algorithm to generate six pyramid components.

[0037] S202. Extract the multi-modal features of the point cloud included in each pyramid component, where the multi-modal features include local geometric features, image projection features, and neighborhood aggregation geometric features.

[0038] In the pyramid component feature enhancement algorithm, for the point cloud included in each pyramid component, its rich multi-modal features are extracted to further enhance the feature expression of the target.

[0039] S203. After cascading the local geometric features, image projection features, and neighborhood aggregation geometric features and then passing through a linear layer process, obtain the final point cloud features of each pyramid component.

[0040] It can be understood that through feature cascading, various feature information can be fused, and after passing through a linear layer process, these features can be presented in a more effective form as the final point cloud features of each pyramid component. This step is also an important part of the pyramid component feature enhancement algorithm.

[0041] S204. Query each final point cloud feature through a preset pyramid feature query algorithm combined with a cross-attention mechanism to obtain pyramid features. Exemplarily, use the preset pyramid query features to interact with the final point cloud features, and with the help of the cross-attention mechanism, accurately obtain the pyramid features of the target to be detected, and complete the construction of the target features in the entire three-dimensional target detection system based on pyramid component feature fusion.

[0042] In an embodiment of the present application, the target bounding box is a cuboid structure. In an embodiment of the present application, step S201 may include the following execution process:

[0043] Process each feature image and each initial target bounding box using a preset pyramid component construction algorithm to obtain six pyramid components corresponding to each initial target bounding box.

[0044] Among them, the target pyramid component construction module is used to divide the initial target bounding boxes generated by the region proposal network into several pyramid target components according to certain division rules. At this time, the point cloud data located inside the target bounding boxes is divided into each pyramid component according to its position.

[0045] Exemplarily, as Figure 2 shown, this step is used to help the model construct pyramid components and jointly learn pyramid component features from point clouds and image projection features at the same time. The target pyramid component construction module takes the image projection features and point cloud bird's-eye view features respectively generated from the image projection feature extraction network and the point cloud feature extraction network as inputs, uses the region proposal network in VoxelNet to generate G target bounding boxes, and then connects the centers of the G target bounding boxes with the eight vertices corresponding to the target bounding boxes, so as to divide each target bounding box into six pyramid components While dividing the pyramid components, the points inside the target are naturally assigned to each pyramid component.

[0046] Next, the target pyramid component construction module respectively obtains the position relationship between the point cloud inside the pyramid and the pyramid vertices, and uses the obtained point cloud features and image projection features to help the point cloud inside the pyramid learn multi-modal features. Then, the target pyramid component construction module uses cross-attention to learn pyramid component features from these enhanced point clouds inside the pyramid by using pyramid query features.

[0047] In an embodiment of the present application, step S202 may include the following execution process:

[0048] S2021. Convert the six vertices of each pyramid component and the point cloud set data inside to the regular coordinate system of the target to which the six vertices and the point cloud set data inside belong from the current coordinate system.

[0049] S2022. Based on the preset offset calculation formula, calculate the relative offsets of each point in the point cloud set data inside each pyramid component to each vertex of the pyramid, and obtain six offset amounts of each point inside each pyramid component.

[0050] S2023. Concatenate the six offset amounts of each point inside each pyramid component to obtain the concatenated features of each point.

[0051] S2024. Use a multi-layer perception structure to process the concatenated features of each point to obtain the local features in the multi-modal features of the point cloud inside each pyramid component.

[0052] The features of the pyramid point cloud are composed of three parts, including the local features of the point cloud, the image projection features of the point cloud, and the geometric features of the point cloud. Assume the i-th pyramid Pyramid iThe six vertices of The set of point clouds contained in this pyramid is denoted as Here, K represents the number of point clouds contained in the current pyramid. To learn the local features of the point cloud, the target pyramid component construction module transforms the six vertices of each pyramid component and the internal point cloud into the regular coordinate system to which the data of the set of six vertices and the internal point cloud belong, that is, with the target center as the coordinate origin, and then rotates the target and the internal points simultaneously to the direction with an angle of 0, that is, making the target direction coincide with the X-axis. Finally, the coordinates of the internal point cloud are calculated in this coordinate system, and then the relative offset between the internal point cloud and the pyramid vertices is calculated according to the preset offset calculation formula (1):

[0053]

[0054] Here, j represents the index of the pyramid vertex, and its value range is an integer from 1 to 5. R represents the rotation matrix from the current coordinate system to the regular coordinate system. l, w, and h respectively represent the length, width, and height of the bounding box of the target to which this pyramid component belongs. Then, the target pyramid component construction module cascades the six relative offsets of each point to obtain a cascaded feature, and the cascaded feature is fed into a multi-layer perception structure to generate the local feature of the point cloud.

[0055] In an embodiment of the present application, in an embodiment of the present application, step S202 may further include the following execution process:

[0056] S2025. Project the set of point cloud data inside each pyramid component onto each feature image to obtain the image projection number

[0057] S2026. Extract the point cloud projection data through bilinear sampling to obtain the image projection features of the point cloud inside each pyramid component.

[0058] Among them, the operation of bilinear sampling in the present invention can smoothly obtain features from the projection data, reduce the influence of data noise, and further improve the quality of the point cloud image projection features. It can be understood that this sampling method makes the feature extraction process more stable, has good adaptability to point cloud data of different scales and distributions, and effectively enhances the generality and robustness of the method of the present invention.

[0059] The beneficial effect of this step is that through this projection method, the image information and the point cloud data features can be effectively combined, providing a richer data basis for subsequent feature extraction. Specifically, in the actual application scenario, this projection operation helps to integrate the data from different perspectives, thereby improving the model's understanding ability of complex scenes.

[0060] In an embodiment of the present application, step S202 may further include the following execution process:

[0061] Using the PointNet structure, near each point inside each pyramid component, aggregate the point cloud data in a spherical neighborhood with a preset radius to generate the neighborhood aggregation geometric features of the point cloud inside each pyramid component.

[0062] Among them, the present invention can effectively improve the detection accuracy of the model for sparse targets by constructing a target pyramid and using the pyramid component features of dense targets to strengthen the component features of sparse targets.

[0063] Exemplarily, through a PointNet structure, aggregate the point cloud features located therein in a spherical neighborhood with a certain radius near each point inside the pyramid component, thereby generating the neighborhood aggregation geometric features of the point cloud inside each pyramid component.

[0064] After obtaining these three types of point cloud features, perform feature concatenation on these point cloud features, and then send the concatenated features into a linear layer to generate the final point cloud features. Here, C1 is the feature channel of the point cloud features.

[0065] In an embodiment of the present application, in an embodiment of the present application, step S204 may further include the following execution process:

[0066] S2041. Process the final point cloud features inside each pyramid component using a preset pyramid feature query formula to obtain pyramid features, where the pyramid feature query formula is obtained based on the sum value of the parameter to be learned and the multi-head attention feature, and the multi-head attention feature is obtained by processing the product of the parameter to be learned and the final point cloud features using the multi-head attention algorithm.

[0067] Among them, the present invention divides the target bounding box into pyramid components, and there are final point cloud features inside these pyramid components. The pyramid features are obtained by processing them using a preset pyramid feature query formula. In the process of the present invention using pyramid components to fuse point cloud and image information to strengthen sparse target detection, the construction of the pyramid feature query formula is involved, which is based on the sum value of the parameter to be learned and the multi-head attention feature, and the multi-head attention feature is also related to the parameter to be learned and the final point cloud features.

[0068] Reference Figure 3 , exemplarily, after obtaining the final point cloud features of the pyramid components, the pyramid component construction algorithm defines a set of parameters to be learned. And use this parameter to generate pyramid query features. Then use these pyramid features to query the useful information contained in the point cloud inside the pyramid components to obtain pyramid features. The pyramid feature query process is shown in formula (2):

[0069]

[0070] Here, represents the feature of the i-th pyramid, C2 represents the number of feature channels of the pyramid feature, and MHAttn represents multi-head attention.

[0071] In an embodiment of the present application, step S40 may include the following execution process:

[0072] S401. Generate a first one-hot encoding based on multiple high-confidence object bounding boxes.

[0073] S402. Generate a first one-hot encoding based on multiple high-confidence object bounding boxes.

[0074] S403. Generate a second one-hot encoding based on multiple low-confidence object bounding boxes, and obtain an inter-pyramid component mask based on the outer product of the first one-hot encoding and the second one-hot encoding.

[0075] S404. Obtain a first feature based on the inter-pyramid component mask and the cross-attention between high-confidence object bounding boxes and low-confidence object bounding boxes.

[0076] S405. Pool the pyramid features within the low-confidence object bounding boxes to obtain a second feature.

[0077] S406. Pool the pyramid features within the high-confidence object bounding boxes to obtain a third feature.

[0078] S407. Sum up the first feature, the second feature, and the third feature to obtain a low-confidence object feature.

[0079] S408. Pool the pyramid features within each high-confidence object bounding box to obtain a fourth feature.

[0080] S409. Process the fourth feature using a self-attention algorithm to obtain a high-confidence object feature.

[0081] Specifically, in combination with Figure 3 , the pyramid component feature enhancement algorithm is used to perform feature interaction between objects on the pyramid components, thereby enhancing the object features of sparse objects and realizing the refinement of the sparse object bounding boxes. The pyramid component features divide the initial object bounding boxes into M high-confidence object bounding boxes and N low-confidence object bounding boxes by comprehensively measuring the object confidence scores generated by the region proposal network, the distance between the center of the object bounding box and the sensor, and the number of point clouds contained in the object bounding box. The sum of M and N is equal to G. Here, the low-confidence object bounding boxes usually refer to those sparse objects that are far from the sensor or blocked by other objects. The main purpose of the pyramid component feature enhancement algorithm is to enhance the features of the objects in the low-confidence object bounding boxes.

[0082] Suppose the pyramid component features of M high-confidence target bounding boxes are represented as The pyramid component features of N low-confidence target bounding boxes are represented as The pyramid component feature enhancement algorithm generates a one-hot encoding that can indicate the target category for each of them according to the target category to which the pyramid components belong. The generated set of one-hot encodings is represented as Here, n_cls is the number of targets to be predicted. Then, the pyramid component feature enhancement algorithm performs an outer product of the second one-hot encoding of the pyramid components of these N low-confidence targets with the first one-hot encoding of the pyramid components of the other M high-confidence targets, and thus generates a mask between pyramid components. This mask is represented as The target pyramid component feature enhancement module then uses this component mask and cross-attention to achieve feature interaction between pyramid components. The expression for enhancing the pyramid features in the low-confidence target bounding box using the pyramid features in the high-confidence target bounding box is shown in Equation (3):

[0083]

[0084] Here, the meanings of the parameters are: represents the pyramid feature of the nth low-confidence target, represents the pyramid feature of the mth low-confidence target, represents the relationship mask between the nth pyramid component and the mth pyramid component. n takes integer values from 1 to N, represents the pyramid component feature of the low-confidence target enhanced by the high-confidence target component feature. exp represents the exponential function. For low-confidence targets, the target pyramid component feature enhancement module performs pooling on the pyramid component features before and after enhancement within the target respectively, and then concatenates the two pooled target features to obtain the target feature of the low-confidence target. For high-confidence targets, the pyramid component features within the target are directly pooled to generate the target feature of the high-confidence target. Then, the target pyramid component feature enhancement module performs feature interaction between high-confidence targets with the help of self-attention to obtain the enhanced target feature of the high-confidence target. Finally, the target features of all targets are fed into a target parameter regression head to refine the initial target bounding box.

[0085] It should be noted that in this application, first, based on the pre-obtained target confidence scores (such as the confidence scores of each target box obtained through a pre-trained model), the actual distance between the target and the sensor, and the point cloud density and other information, the initial target bounding boxes are classified to distinguish multiple high-confidence target boxes and multiple low-confidence target boxes. Then, using the cross-attention mechanism between the pyramid components, the more reliable pyramid features within the high-confidence target boxes are transmitted to the low-confidence target boxes. At the same time, after pooling the pyramid features within the low-confidence target boxes and then fusing them with the high-confidence target features, the pyramid features within the low-confidence target boxes are strengthened to obtain low-confidence target features. For the high-confidence target boxes, after pooling their pyramid features, they are further processed by self-attention to explore the internal relationships of the pyramid features within the high-confidence target boxes, obtaining high-confidence target features. Finally, the low-confidence target features and the high-confidence target features are combined to obtain the enhanced features of the three-dimensional target to be detected, making the final features more conducive to accurately detecting the three-dimensional target. It can be understood that this method of fusing features based on target boxes with different confidences can make full use of the information of various target boxes and improve the accuracy of three-dimensional target detection.

[0086] Reference Figure 4 , based on the above embodiments, this application also provides a three-dimensional target detection device based on pyramid component feature fusion. The three-dimensional target detection device 100 includes an image feature acquisition module 101, a bounding box acquisition module 102, a pyramid component construction module 103, and a pyramid component feature enhancement module 104. Among them, the image feature acquisition module 101 is used to process the multi-view images of the target to be detected based on the image feature extraction network to obtain each feature image. The bounding box acquisition module 102 is used to process the point cloud data of the target to be detected based on the point cloud feature extraction network to obtain a plurality of initial target bounding boxes. The pyramid component construction module 103 is used to process each feature image and each initial target bounding box using a preset pyramid component construction algorithm to obtain a plurality of pyramid features corresponding to each initial target bounding box. The pyramid component feature enhancement module 104 is used to classify the initial target bounding boxes into a plurality of high-confidence target boxes and a plurality of low-confidence target boxes based on a preset pyramid component feature enhancement algorithm according to the pre-obtained target confidence scores, the distance between the target and the sensor, and the point cloud density. Based on the cross-attention between the pyramid components, use the pyramid features within the high-confidence target boxes to strengthen the pyramid features within the low-confidence target boxes, and use the pooled pyramid features within the low-confidence target boxes to obtain low-confidence target features. According to the pooled and self-attention processed pyramid features within the high-confidence target boxes, obtain high-confidence target features, and based on the low-confidence target features and the high-confidence target features, obtain the enhanced features of the three-dimensional target to be detected.

[0087] Specifically, the technical solution adopted by the present invention to solve its technical problems is as follows:

[0088] Construct a three-dimensional object detection system based on the feature fusion of pyramid components, including an object pyramid component construction module and an object pyramid component feature enhancement module;

[0089] The object pyramid component construction module is used to divide the initial object bounding boxes generated by the region proposal network into several pyramid object components according to certain division rules. At this time, the point cloud data located inside the object bounding box is divided into each pyramid component according to its position. Then, the object pyramid component construction module helps the point cloud inside the pyramid learn multimodal features from the positional relationship between the point cloud inside the pyramid and the pyramid vertex, the obtained point cloud features, and the image projection features. Next, the object pyramid component construction module uses cross-attention to learn the pyramid component features from these enhanced point clouds inside the pyramid with the pyramid query features;

[0090] After the object pyramid component feature enhancement module helps the object construct pyramid components and obtain the corresponding pyramid component features, it divides the objects in the scene into two categories: high-confidence objects and low-confidence objects. For low-confidence objects, the object pyramid component feature enhancement module uses cross-attention to enhance the pyramid component features of low-confidence objects with the pyramid component features of high-confidence objects. The enhanced pyramid component features of low-confidence objects are then fused with the pooled object features, so as to achieve the purpose of enhancing sparse object features. For high-confidence objects, the object pyramid component feature enhancement module first pools the pyramid component features inside the object to obtain the object features of high-confidence objects, and then uses self-attention to realize the feature interaction between high-confidence objects, and finally realizes the enhancement of high-confidence object features.

[0091] The above are only the preferred embodiments of this application, and do not limit the patent scope of this application. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of this application, or directly or indirectly applied in other related technical fields, shall be included in the patent protection scope of this application by the same token.

Claims

1. A 3D object detection algorithm based on pyramid component feature fusion, characterized in that Including: Processing multi-view images of a target to be detected based on an image feature extraction network to obtain respective feature images; Processing point cloud data of a target to be detected based on a point cloud feature extraction network to obtain a plurality of initial target bounding boxes; Processing each feature image and each initial target bounding box by using a preset pyramid component construction algorithm to obtain a plurality of pyramid features corresponding to each initial target bounding box; Using a preset pyramid component feature enhancement algorithm to classify the initial target bounding boxes based on a pre-obtained target confidence score, the distance between the target and the sensor, and the point cloud density, to obtain a plurality of high-confidence target boxes and a plurality of low-confidence target boxes; based on the cross-attention between pyramid components, using the pyramid features within the high-confidence target boxes to enhance the pyramid features within the low-confidence target boxes, and, using the pyramid features within the pooled low-confidence target boxes, to obtain low-confidence target features; according to the pyramid features within the pooled and self-attention processed high-confidence target boxes, to obtain high-confidence target features, using a target parameter regression head to process the low-confidence target features and the high-confidence target features, to obtain a plurality of refined initial target bounding boxes, and detecting a three-dimensional target based on the plurality of refined initial target bounding boxes to obtain the detected three-dimensional target.

2. The 3D object detection algorithm based on pyramid component feature fusion according to claim 1, characterized in that, The process of processing each feature image and each initial target bounding box by using a preset pyramid component construction algorithm to obtain a plurality of pyramid features corresponding to each initial target bounding box includes: Processing each feature image and each initial target bounding box by using a preset pyramid component construction algorithm to obtain six pyramid components corresponding to each initial target bounding box; Extracting multi-modal features of the point cloud included in each pyramid component, where the multi-modal features include local geometric features, image projection features, and neighborhood aggregation geometric features; Performing feature concatenation on the local geometric features, the image projection features, and the neighborhood aggregation geometric features and then performing linear layer processing to obtain the final point cloud features of each pyramid component; Querying each final point cloud feature through a preset pyramid feature query algorithm in combination with a cross-attention mechanism to obtain pyramid features.

3. The 3D object detection algorithm based on pyramid component feature fusion according to claim 2, characterized in that, The target bounding box is a cuboid structure. The process of processing each feature image and each initial target bounding box by using a preset pyramid component construction algorithm to obtain six pyramid components corresponding to each initial target bounding box includes: Connecting the center of each cuboid structure to the eight vertices of the cuboid structure to obtain six pyramid components.

4. The 3D object detection algorithm based on pyramid component feature fusion according to claim 2, characterized in that, The process of extracting the local geometric features in the multi-modal features of the point cloud included in each pyramid component includes: Converting the six vertices of each pyramid component and the point cloud set data inside from the current coordinate system to the regular coordinate system of the target to which the six vertices and the point cloud set data inside belong; Based on a preset offset calculation formula, calculating the relative offsets between each point in the point cloud set data inside each pyramid component and the vertices of the pyramid to obtain six offset amounts for each point inside each pyramid component; Concatenating the six offset amounts of each point inside each pyramid component to obtain the concatenated feature of each point. The cascade features of each point are processed using a multi-layer perceptron structure to obtain the local features in the multi-modal features of the point cloud inside each pyramid component.

5. The 3D object detection algorithm based on pyramid component feature fusion according to claim 2, characterized in that, The process of extracting the image projection features in the multi-modal features of the point cloud included in each pyramid component includes: Projecting the point cloud set data inside each pyramid component onto each feature image to obtain point cloud projection data; Extracting the point cloud projection data through bilinear sampling to obtain the image projection features of the point cloud inside each pyramid component.

6. The three-dimensional object detection algorithm based on pyramid component feature fusion according to claim 3, characterized in that, The process of extracting the neighborhood aggregation geometric features in the multi-modal features of the point cloud included in each pyramid component includes: Using the PointNet structure to aggregate each point cloud data with a spherical neighborhood of a preset radius near each point inside each pyramid component to generate the neighborhood aggregation geometric features of the point cloud inside each pyramid component.

7. The three-dimensional object detection algorithm based on pyramid component feature fusion according to claim 3, characterized in that, Querying each final point cloud feature through a preset pyramid feature query algorithm combined with a cross-attention mechanism to obtain pyramid features, including: Processing the final point cloud features inside each pyramid component using a preset pyramid feature query formula to obtain pyramid features, where the pyramid feature query formula is obtained based on the sum value of the parameters to be learned and the multi-head attention features, and the multi-head attention features are obtained by processing the product of the parameters to be learned and the final point cloud features through the multi-head attention algorithm.

8. The three-dimensional object detection algorithm based on pyramid component feature fusion according to claim 1, characterized in that Strengthening the pyramid features inside the low-confidence target box using the pyramid features inside the high-confidence target box, and obtaining the low-confidence target features using the pooled pyramid features inside the low-confidence target box, including: Generating a first one-hot encoding based on multiple high-confidence target boxes; Generating a second one-hot encoding based on multiple low-confidence target boxes, and obtaining an inter-pyramid component mask based on the outer product of the first one-hot encoding and the second one-hot encoding; Obtaining a first feature based on the inter-pyramid component mask and the cross-attention between the high-confidence target box and the low-confidence target box; Pooling the pyramid features inside the low-confidence target box to obtain a second feature; Pooling the pyramid features inside the high-confidence target box to obtain a third feature; Adding and processing the first feature, the second feature, and the third feature to obtain the low-confidence target features.

9. The 3D object detection algorithm based on pyramid component feature fusion according to claim 1, characterized in that, Obtaining the high-confidence target features according to the pooled and self-attention processed pyramid features inside the high-confidence target box, including: Pooling the pyramid features inside each high-confidence target box to obtain a fourth feature; Processing the fourth feature using the self-attention algorithm to obtain the high-confidence target features.

10. A three-dimensional object detection device based on pyramid component feature fusion, characterized in that Including: An image feature acquisition module for processing the multi-view images of the target to be detected based on an image feature extraction network to obtain each feature image; A bounding box acquisition module for processing the point cloud data of the target to be detected based on a point cloud feature extraction network to obtain multiple initial target bounding boxes; A pyramid component construction module for processing each feature image and each initial target bounding box using a preset pyramid component construction algorithm to obtain multiple pyramid features corresponding to each initial target bounding box. Pyramid component feature enhancement module, which is used to classify the initial target bounding box based on the pre-acquired target confidence score, the distance between the target and the sensor, and the point cloud density by using a preset pyramid component feature enhancement algorithm, so as to obtain multiple high-confidence target boxes and multiple low-confidence target boxes; based on the cross-attention between pyramid components, use the pyramid features in the high-confidence target box to enhance the pyramid features in the low-confidence target box, and use the pyramid features in the pooled low-confidence target box to obtain low-confidence target features; according to the pyramid features in the pooled and self-attention processed high-confidence target box, obtain high-confidence target features, use the target parameter regression head to process the low-confidence target features and high-confidence target features, obtain multiple refined initial target bounding boxes, and detect three-dimensional targets based on the multiple refined initial target bounding boxes to obtain the detected three-dimensional targets.

Images