A target detection method, device and terminal equipment
By combining spherical masks and global feature extraction layers, the key point sampling and encoding strategies of 3D object detection methods are improved, enhancing feature extraction and learning capabilities, and improving the accuracy and effectiveness of object detection.
Patent Information
- Application Number
- CN202310498828.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-05
- Publication Date
- 2026-02-10
- Estimated Expiration
- 2043-05-05
AI Technical Summary
Existing 3D target detection methods are insufficient in feature extraction and feature learning capabilities, failing to fully utilize the basic information of LiDAR point clouds, resulting in poor detection performance.
By acquiring the spherical mask and mask features of point cloud data, a farthest point sampling strategy based on the spherical mask is adopted for key point sampling. The mask features of the key point cloud are enhanced by the global feature extraction layer of the spherical neighborhood. In combination with a 3D region proposal network and non-maximum suppression, feature extraction and filtering are performed.
It improves feature extraction and feature learning capabilities, enhancing the accuracy and effectiveness of target detection, especially excelling in the detection of small targets such as pedestrians.
Smart Images
Figure CN116740697B_ABST
Abstract
Description
[0001] Technology Neighborhood
[0002] This invention relates to the field of image target detection technology, and more particularly to a target detection method, apparatus, and terminal device. Background Technology
[0003] Object detection, as one of the most fundamental and challenging problems in computer vision, has received extensive research and exploration from all sectors of society in recent years. As an important task in computer vision, object detection typically aims to provide the specific location of a certain type of visual object (such as a human, animal, or car) in a digital image. Taking autonomous driving technology as an example, when applied in autonomous driving technology, object detection can identify the type of static and dynamic objects, acquire their location information, and thus identify pedestrians and obstacles, helping autonomous vehicles to make effective predictions and plans.
[0004] Most 3D object detection methods use point clouds from LiDAR as initial data, which provides more accurate distance information and stronger anti-interference capabilities compared to cameras, and is less affected by external weather changes and lighting conditions. However, commonly used 3D object detection methods first use farthest point sampling to filter keypoints, and then encode the sampled keypoints to find all regions of interest in the image, determining the target, its category, and location. In the keypoint filtering step, farthest point sampling relies solely on relative distance, meaning the keypoint sampling strategy is simplistic, resulting in poor feature extraction capabilities. In the keypoint encoding step, only the relative distance between the keypoint and the sampled points is encoded, causing the encoding result to learn only local features and lack global features.
[0005] Therefore, there is an urgent need for a new 3D object detection method to improve the key point sampling strategy, make full use of the basic information of point clouds, and enhance feature extraction and feature learning capabilities. Summary of the Invention
[0006] The main objective of this invention is to propose a target detection method, apparatus, and terminal device to solve the problem that existing 3D target detection technologies do not fully utilize the basic information from point clouds obtained from lidar, and their feature extraction and feature learning capabilities cannot achieve good target detection results.
[0007] To achieve the above objectives, a first aspect of the present invention provides a target detection method, comprising:
[0008] Obtain point cloud data of the image to be detected;
[0009] Calculate the spherical mask of each original point cloud in the point cloud data, and the mask features of its spherical neighborhood;
[0010] Key point sampling is performed on the point cloud data by sampling the farthest point based on a spherical mask to obtain a sampling point set based on the point cloud data. The sampling point set includes K key point clouds, the number of which is less than the number of the original point cloud. The mask features of the K key point clouds are enhanced by a global feature extraction layer based on a spherical neighborhood. K is a positive integer.
[0011] Decoding is performed based on the coordinate information of K key point clouds and the enhanced mask features;
[0012] The decoding results are subjected to predictive processing to obtain preliminary 3D suggestions;
[0013] The primary 3D proposal is improved by a 3D region proposal network and nonmaximum suppression, resulting in a primary region of interest pool.
[0014] The primary region of interest pool is filtered according to the intersection-union ratio threshold, and features are extracted from the filtered primary region of interest pool through a corner feature extraction layer and a grid pooling layer. The region of interest pool with extracted features is used as the target detection result of the image to be detected.
[0015] Optionally, the spherical mask of each original point cloud in the point cloud data, and the mask features of its spherical neighborhood, are calculated, including:
[0016] For any original point cloud, select all point clouds within the radius of a spherical neighborhood as the spherical mask of this original point cloud; calculate the maximum offset of all point clouds within the radius of the spherical neighborhood in the vertical direction to obtain the vertical features in the vertical direction, and the vertical features are the mask features of the spherical neighborhood to which this original point cloud belongs.
[0017] Optionally, for the original point cloud with a vertical feature of 0, its spherical mask is set to 0.
[0018] Optionally, before performing keypoint sampling on the point cloud data using farthest-point sampling based on a spherical mask, the following steps are included:
[0019] Set the spherical query radius;
[0020] Based on the spherical query radius, a spherical mask is used for filtering, filtering out the original point clouds in the point cloud data whose spherical mask is 0.
[0021] Optionally, keypoint sampling is performed on the point cloud data by sampling the farthest point based on a spherical mask, including:
[0022] Take any original point cloud in the point cloud data as the starting point;
[0023] Obtain the spherical mask of the starting point;
[0024] When the spherical mask of the starting point is 1, this original point cloud is added to the sampling point set;
[0025] Calculate the distances from the remaining original point cloud to the starting point, and construct an N-dimensional distance array;
[0026] Multiply the N-dimensional distance array by the spherical mask of the starting point to obtain the mask distance array;
[0027] If the amount of data in the sampling point set is less than N, then the point with the largest value is selected from the mask distance array as the key point cloud;
[0028] The key point cloud is added to the sampling point set, and the point with the largest value is removed from the mask distance array;
[0029] Calculate the distance from the remaining original point cloud to the key point cloud. If the distance from the remaining original point cloud to the key point cloud is less than any element of the mask distance array, then take another original point cloud as the starting point, update the mask distance matrix, and return the result based on the updated mask distance matrix. If the data volume of the sampling point set is less than N, then select the point with the largest value from the mask distance array as the key point cloud.
[0030] Optionally, the mask features of the K key point clouds are enhanced by a global feature extraction layer based on a spherical neighborhood, including:
[0031] For the k-th key point cloud, the X-axis position information is extracted by a global feature extraction layer based on a spherical neighborhood; k is a positive integer less than or equal to K.
[0032] Calculate the ratio of the vertical feature to the X-axis position information, and use it as the first global feature;
[0033] The local distance offset between the sampling point and the key point is obtained by the original feature extraction layer as the second global feature, and the LiDAR feature is used as the third global feature.
[0034] The enhanced mask features include the first global feature, the second global feature, and the third global feature.
[0035] Optionally, the prediction process includes classification score prediction and 3D bounding box prediction.
[0036] A second aspect of the present invention provides a target detection device, comprising:
[0037] The point cloud data acquisition module is used to acquire point cloud data of the image to be detected;
[0038] The spherical mask calculation module is used to calculate the spherical mask of each original point cloud of the point cloud data, as well as the mask features of its spherical neighborhood.
[0039] The key point sampling and feature enhancement module is used to sample key points in the point cloud data by sampling the farthest point based on a spherical mask to obtain a set of sampling points based on the point cloud data. The set of sampling points includes K key point clouds, the number of which is less than the number of the original point cloud. The mask features of the K key point clouds are enhanced by a global feature extraction layer based on a spherical neighborhood. K is a positive integer.
[0040] The decoding module is used to decode based on the coordinate information of K key point clouds and the enhanced mask features;
[0041] The prediction module is used to perform predictive processing on the decoding results to obtain preliminary 3D suggestions;
[0042] The Region of Interest (ROI) output module is used to refine the primary 3D proposal through a 3D region proposal network and nonmaximum suppression to output a primary ROI pool.
[0043] The target detection result output module is used to filter the primary region of interest pool according to the intersection-union ratio threshold, and extract features from the filtered primary region of interest pool through a corner feature extraction layer and a grid pooling layer, and use the region of interest pool after feature extraction as the target detection result of the image to be detected.
[0044] A third aspect of the present invention provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method provided in the first aspect above.
[0045] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method provided in the first aspect above.
[0046] This invention proposes a target detection method that calculates the spherical mask and mask features of each original point cloud in point cloud data. A novel keypoint sampling strategy, based on sampling the farthest point of the spherical mask, selects representative original point clouds as keypoint clouds. Simultaneously, a global feature extraction layer based on the spherical neighborhood is used for encoding. During encoding, global features are used to enhance the mask features of the keypoint clouds, enabling them to learn both local and global features. Therefore, the target detection method proposed in this invention improves the feature-enhanced 3D target detection network in original radar point clouds by improving the keypoint sampling strategy to enhance feature extraction capabilities, improving the encoding method to enhance feature learning capabilities, and optimizing target detection performance. Furthermore, a corner feature extraction layer and a grid pooling layer are used to extract features from the filtered primary region of interest pool, further improving the learning ability for high-dimensional point-level semantic features. The target detection method proposed in this invention has strong feature extraction capabilities for detecting targets such as pedestrians, learning point features within their spherical neighborhood, extracting high-level point semantic features of regions of interest, and improving the detection accuracy of small targets. Attached Figure Description
[0047] Figure 1 This is a schematic diagram illustrating the implementation process of the target detection method provided in an embodiment of the present invention;
[0048] Figure 2 A comparison of vertical features of a spherical neighborhood at the same distance provided in an embodiment of the present invention;
[0049] Figure 3 A comparison of vertical features of a spherical neighborhood at different distances provided in this embodiment of the invention;
[0050] Figure 4 This is a schematic diagram of the composition of the target detection device provided in an embodiment of the present invention.
[0051] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0052] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
[0053] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0054] In this document, suffixes such as "module," "part," or "unit" used to denote elements are used only for the purpose of illustrative purposes and do not have any specific meaning in themselves. Therefore, "module" and "part" can be used interchangeably.
[0055] like Figure 1 As shown, an embodiment of the present invention provides a target detection method, including:
[0056] S101. Obtain the point cloud data of the image to be detected.
[0057] In step S101 above, the image to be detected is an image obtained by LiDAR scanning, which is converted into a set of vectors in a three-dimensional coordinate system, i.e., point cloud data.
[0058] S102. Calculate the spherical mask of each original point cloud of the point cloud data, and the mask features of its spherical neighborhood.
[0059] In step S102 above, the spherical mask and mask features of the original point cloud are used to represent the surrounding neighborhood environment information.
[0060] It should be noted that the spherical mask is a mask based on the original point cloud, and the mask features are the attributes of the mask. Taking any original point cloud as an example, this embodiment of the invention selects all point clouds within the radius of the spherical neighborhood as the spherical mask of this original point cloud; calculates the maximum offset in the vertical direction of all point clouds within the radius of the spherical neighborhood to obtain the vertical features in the vertical direction, and the vertical features are the mask features of the spherical neighborhood to which this original point cloud belongs.
[0061] In this embodiment of the invention, the mask features obtained in step S102 are applied to the original network, thereby improving the original network. The original network is a 3D target detection network with enhanced features from the original radar point cloud.
[0062] S103. The point cloud data is sampled for key points by sampling the farthest point based on a spherical mask to obtain a set of sampled points based on the point cloud data. The set of sampled points includes K key point clouds, and the number of key point clouds is less than the number of the original point cloud. The mask features of the K key point clouds are enhanced by a global feature extraction layer based on a spherical neighborhood. K is a positive integer.
[0063] In step S103 above, based on the farthest point sampling of the spherical mask, the mask features are combined with the relative distance used by the original network to make up for the singleness of the sampling strategy of screening key points by the farthest point sampling; based on the global feature extraction layer of the spherical neighborhood, the mask features are combined with the local features used by the original network to make up for the lack of global features in the step of encoding key points, thereby improving the feature extraction ability, i.e. feature learning ability, of the original network.
[0064] Among them, farthest-point sampling based on spherical masks utilizes spherical masks and mask features for keypoint sampling. In practical applications, the spherical mask of the original point cloud represents a spherical neighborhood with a flexible receptive field. If keypoint sampling is performed using the original network, i.e., farthest-point sampling, and only the relative distance offset of local sampling points is used as a high-dimensional feature, the keypoint cloud in the sampling point set will lack certain global characteristics, affecting the point cloud distribution within the spherical neighborhood of the keypoint cloud.
[0065] like Figure 2 As shown, the black dashed line represents the vertical field of view of the lidar, and the beam returns when it encounters a target. Both square and circular dots represent the original point cloud, and these represent the reflection points of the beam when it encounters an obstacle. The square dots are the key points sampled from the farthest point, and the circular dots are the original point cloud. dz0 represents the maximum vertical feature distance of the sampled points within the spherical neighborhood of key point P0, and dz1 is similar. When two key points obtained from sampling from the farthest point are at the same global distance x0 and maintain the same radius R for the same object, the more laser beams swept by key point P1 within the entire spherical neighborhood, the more reflection points there are, and the denser the point cloud. Based on step S103 above, in this embodiment of the invention, for a spherical neighborhood with radius R, the maximum vertical offset of the sampled points within it is a direct expression of the laser beams swept vertically within that spherical neighborhood. Therefore, the maximum vertical offset of the sampled points within the spherical neighborhood is called the vertical feature F of the spherical neighborhood. lv .
[0066] Assuming that spherical local feature extraction is performed on the key point cloud P0 with radius R = 1, the set of sampling points in the spherical neighborhood can be obtained, representing the spherical mask of the key point cloud P0, as shown in the following formula (1.1):
[0067] P g ={p1,p2,...,pi},d io ≤T(1.1)
[0068] Where d io This represents the distance from key point P0 to neighboring point p. i The distance.
[0069] Sample nsample points from this sampling point set to obtain the index idx of these sampling points and their global point coordinates group_xyz in the real environment, as shown in Formula (1.2):
[0070] group_xyz={(x1,y1,z1),...,(x i ,y i ,z i ),...,(x ns ,y ns ,z ns )}, i∈idx (1.2)
[0071] Based on these global point coordinates, the maximum offset group_dz in the vertical direction Z of these global point coordinates can be obtained, which is used to indicate the vertical feature F of the laser beam swept within the spherical neighborhood. lv As shown in formula (1.3):
[0072] F lv :group_dz=group_xyz zmax -group_xyz zmin (1.3)
[0073] The vertical feature F in the vertical direction lv It can serve as an expression of the vertical characteristics of LiDAR within a spherical neighborhood and as a supplement to the local distance offset within the spherical neighborhood. Therefore, this embodiment of the invention combines vertical features with the relative distance used by the original network, fully considering the neighborhood features of key points, reducing the influence of the original network's key point sampling relying solely on distance for selection, and compensating for the singularity of the sampling strategy of selecting key points by sampling from the farthest point. This results in better selection of key points and improves the point-level feature extraction capability of the original network.
[0074] In one embodiment, for an original point cloud with a vertical feature of 0, its spherical mask is set to 0.
[0075] Based on this, after step S102 and before step S103, the following is also included:
[0076] Set the spherical query radius;
[0077] Based on the spherical query radius, a spherical mask is used for filtering, filtering out the original point clouds in the point cloud data whose spherical mask is 0.
[0078] Through the above steps, before keypoint sampling, this embodiment of the invention deletes the original point cloud data with a spherical mask of 0. The purpose is to eliminate "empty" spherical neighborhood points within the spherical query radius, meaning that these points exist only within the R-radius region of the original point cloud. Because "empty" spherical neighborhood points contribute little to the local feature learning of the spherical neighborhood features during keypoint sampling, they are detrimental to foreground feature learning and cannot be added to the sampling point set in this embodiment of the invention.
[0079] The spherical query radius differs from the spherical neighborhood radius R in the global feature extraction layer based on spherical neighborhoods; it is a parameter used for spherical mask filtering. A smaller spherical query radius indicates a smaller local neighborhood of the sphere, resulting in a greater number of "empty" neighborhood points after mask filtering. However, an excessively small R... mask This may have a negative impact on large targets and obstacles. Therefore, when setting the spherical query radius, it is necessary to fully consider the influencing factors such as the size and volume of vehicles, pedestrians, and cyclists, so as to effectively filter out background points while retaining some local sparse foreground features.
[0080] In this embodiment of the invention, a spherical mask is used to indicate the vertical offset features of the laser within a spherical neighborhood. Based on this, spherical mask filtering is performed according to the spherical query radius. Specifically, this is implemented by filtering "empty" spherical neighborhood points among the key points using the spherical mask group_dz method based on the vertical features. Therefore, for a spherical query radius of R... mask The query result, i.e., the spherical mask group_dz, can be divided into two cases, as shown in the following formula (1.4):
[0081]
[0082] The above formula shows that for a point where the vertical feature group_dz is 0, the spherical mask group_mask is set to 0, indicating that there are no other reflection points in the neighborhood of the sphere, and the sphere is an "empty" neighborhood point.
[0083] This embodiment of the invention also provides detailed implementation steps for step S103 above, which involves sampling key points in the point cloud data by sampling the farthest point based on a spherical mask, including:
[0084] S1031. Take any original point cloud in the point cloud data as the starting point;
[0085] S1032. Obtain the spherical mask of the starting point;
[0086] S1033. When the spherical mask of the starting point is 1, add this original point cloud to the sampling point set;
[0087] S1034. Calculate the distance from the remaining original point cloud to the starting point and form an N-dimensional distance array;
[0088] S1035. Multiply the N-dimensional distance array with the spherical mask of the starting point to obtain the mask distance array;
[0089] S1036. If the amount of data in the sampling point set is less than N, then select the point with the largest value from the mask distance array as the key point cloud.
[0090] S1037. Add the key point cloud to the sampling point set, and delete the point with the largest value from the mask distance array;
[0091] S1038. Calculate the distance from the remaining original point cloud to the key point cloud. If the distance from the remaining original point cloud to the key point cloud is less than any element of the mask distance array, then take another original point cloud as the starting point, update the mask distance matrix, and return the result based on the updated mask distance matrix. If the data volume of the sampling point set is less than N, then select the point with the largest value from the mask distance array as the key point cloud.
[0092] In the above steps, when the number of sampling points is less than N, it means that a suitable number of points have been selected. At this point, a key point cloud can be output from the mask distance array based on this original point cloud and added to the sampling point set. The sampling point set is obtained when all the original point clouds of the point cloud data have been traversed.
[0093] In step S103 above, the global feature extraction layer based on the spherical neighborhood uses the local features of the original network to enhance the mask features of the K key point clouds. In practical applications, after sampling key points to obtain a set of key point clouds, the feature extraction layer of the original network extracts the point-level features of the key point clouds. However, since the feature extraction layer of the original network only calculates the relative distance from the sampling points to the key points, this embodiment of the invention introduces mask features to supplement the global features, and further supplements them with global position features.
[0094] like Figure 3As shown, two spherical neighborhoods of the same radius are placed at different distances, x1 and x0. The dashed line represents the vertical field of view of the lidar. The square points are the key points sampled by the FPS, and the circular points are the original point clouds. The key point clouds P0 and P1 are kept at the same vertical distance Z. dz0 represents the maximum vertical feature distance of the sampled points in the spherical neighborhood when the key point cloud P0 is at a distance of x0. Similarly, dz1 represents the maximum vertical feature distance of the sampled points in the spherical neighborhood when the key point cloud P1 is at a distance of x1. For the same target object, in the case of spherical neighborhoods of the same radius, the farther the distance, the fewer points in the spherical neighborhood, that is, the fewer reflection points swept by the vertical laser beam, and the lower the obtained vertical feature FPS. lver The smaller the value, the better. Therefore, for a spherical neighborhood P0 based on key point cloud P0 and a spherical neighborhood P1 based on key point cloud P1, the vertical offset distance dz1 is greater than dz0, which is affected by the global coordinate distance. Based on this, embodiments of the present invention can supplement global features by introducing mask features.
[0095] This embodiment of the invention also provides detailed implementation steps for step S103 above, which enhances the mask features of K key point clouds through a global feature extraction layer based on a spherical neighborhood, including:
[0096] For the k-th key point cloud, the X-axis position information is extracted by a global feature extraction layer based on a spherical neighborhood; k is a positive integer less than or equal to K.
[0097] Calculate the ratio of the vertical feature to the X-axis position information, and use it as the first global feature;
[0098] The local distance offset between the sampling point and the key point is obtained by the original feature extraction layer as the second global feature, and the LiDAR feature is used as the third global feature.
[0099] The enhanced mask features include the first global feature, the second global feature, and the third global feature.
[0100] The above steps actually incorporate the X-axis position information of key points as global features into feature learning through a global feature extraction layer based on spherical neighborhood.
[0101] The embodiments of the present invention also describe its calculation steps. Specifically, after obtaining the local vertical features of the spherical neighborhood... Then, find the vertical features. The ratio of the distance to the global location x is used to calculate the global features within a local spherical region. The following formula (1.5):
[0102]
[0103] This feature will be used as the local distance offset F between the sampling points and key points within the sphere. sa This supplements and allows for better learning of local features of key points, while also incorporating local vertical features of the spherical neighborhood. and global location features Name this feature F sl This refers to global spherical neighborhood features. Furthermore, similar to the original feature extraction layer, after obtaining the sampling point index idx within the neighborhood of the keypoint, the corresponding LiDAR feature F is obtained. fea Since reflectivity serves as a further supplement, the enhanced mask features output by the global feature extraction layer based on the spherical neighborhood ultimately consist of three parts:
[0104]
[0105] S104. Decode based on the coordinate information of the K key point clouds and the enhanced mask features.
[0106] S105. Perform prediction processing on the decoding results to obtain preliminary three-dimensional suggestions.
[0107] In step S105 above, the prediction process includes classification score prediction and 3D bounding box prediction.
[0108] S106. The primary three-dimensional proposal is improved by a 3D region proposal network and nonmaximum suppression, and the primary region of interest pool is output.
[0109] In steps S105 and S106 above, the primary 3D proposal is the first-stage 3D thinning box output by the embodiment of the present invention. The 3D region proposal network is used to provide classification scores, and non-maximum suppression is used to evaluate the degree of overlap between the predicted box and the ground truth box, thereby filtering out a certain number of regions of interest as input to the second-stage 3D thinning box. The output is the primary region of interest pool, which is also the second-stage 3D thinning box.
[0110] S107. The primary region of interest pool is filtered according to the intersection-union ratio threshold, and features are extracted from the filtered primary region of interest pool through the corner feature extraction layer and the grid pooling layer. The region of interest pool after feature extraction is used as the target detection result of the image to be detected.
[0111] In step S107 above, the degree of overlap between the predicted bounding box and the ground truth bounding box is first measured based on the intersection-union ratio (IU) threshold, thereby filtering the primary region of interest (ROI) pool and retaining high-quality positive samples. Then, the filtered primary ROI pool is passed through a corner feature extraction layer based on viewpoint attention to obtain the local corner features F. attcorner And obtain pooling layer features through grid pooling layers. The local corner features and pooling features are then added together to obtain the final region of interest (ROI) features, i.e., the ROI pool after feature extraction. The ROI pool F after feature extraction. roi This can be expressed as formula (3.7):
[0112]
[0113] This invention also describes the feature extraction process of the corner feature extraction layer and the grid pooling layer. In the corner feature extraction layer, the eight corner points of the region represented by the filtered primary region of interest pool are used as keypoint centers. A spherical neighborhood query is used to query neighborhood sampling points within a multi-scale radius R. The local features of these neighborhood sampling points are then used to extract high-dimensional features through MLP (Multi-layer perceptron neural networks). These high-dimensional features are the octagonal points P. corner Local query features F corner Map them to the view dimension F respectively. sper and channel dimension F cha We obtain the weights Atten for these two dimensions. Simultaneously, we extract the features F of the local corner points. corner Multiply the value by the attention weight Atten and add it to the original corner feature to obtain the attention-weighted local corner feature F. attcorner In the grid pooling layer, PV_RCNN divides the region of interest (RoI) into 216 grid points, and uses these 216 grid points as keypoints to query N within the spherical neighborhood R. sample The original point cloud is analyzed, and high-dimensional local features are extracted using MLP. PV-RCNN further refines the RoI region by converting grid points into keypoints, which is beneficial for RoI feature learning. Inspired by the pooling layer of PV-RCNN, eight grid points within the RoI region are obtained, dividing the RoI region into eight local grids. Simultaneously, using a spherical query sampling method, sampling points within a radius R centered on these eight grid points are queried to obtain their local grid features. These features also exhibit certain viewpoint characteristics, but the local features of these eight grid points are more focused on the interior of the RoI region. In practical applications, the corner feature extraction layer can focus on features outside the RoI. To a certain extent, it can compensate for the lack of RoI boundary information predicted in the previous stage. The grid pooling layer combines the corner features of the vertices with the local grid features to extract the RoI aggregated features.
[0114] Based on steps S101 to S107 above, this embodiment of the invention proposes a new keypoint sampling strategy, namely, sampling the farthest point based on a spherical mask to select representative keypoint clouds; and a new feature extraction layer, namely a global feature extraction layer, which fully considers the vertical and global characteristics within the spherical neighborhood of the keypoint cloud to supplement the contextual representation of the keypoint cloud; it also improves the feature-enhanced 3D target detection network in the original radar point cloud, enhancing feature extraction and feature learning capabilities to achieve better target detection results. Furthermore, it further improves the learning ability for high-dimensional point-level semantic features by extracting features from the filtered primary region of interest pool through a corner feature extraction layer and a grid pooling layer.
[0115] like Figure 4 As shown, this embodiment of the invention also provides a target detection device 40, comprising:
[0116] Point cloud data acquisition module 41 is used to acquire point cloud data of the image to be detected;
[0117] The spherical mask calculation module 42 is used to calculate the spherical mask of each original point cloud of the point cloud data, as well as the mask features of its spherical neighborhood.
[0118] The key point sampling and feature enhancement module 43 is used to sample key points in the point cloud data by sampling the farthest point based on a spherical mask to obtain a set of sampling points based on the point cloud data. The set of sampling points includes K key point clouds, the number of which is less than the number of the original point cloud. The mask features of the K key point clouds are enhanced by a global feature extraction layer based on a spherical neighborhood. K is a positive integer.
[0119] Decoding module 44 is used to decode based on the coordinate information of K key point clouds and the enhanced mask features;
[0120] Prediction module 45 is used to perform prediction processing on the decoding results to obtain preliminary three-dimensional suggestions;
[0121] Region of Interest (ROI) output module 46 is used to refine the primary three-dimensional proposal through a 3D region proposal network and non-maximum suppression to output a primary ROI pool.
[0122] The target detection result output module 47 is used to filter the primary region of interest pool according to the intersection-union ratio threshold, and to extract features from the filtered primary region of interest pool through the corner feature extraction layer and the grid pooling layer, and to use the region of interest pool after feature extraction as the target detection result of the image to be detected.
[0123] This invention also provides a terminal device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the various steps of the target detection method described in the above embodiments.
[0124] This invention also provides a storage medium, which is a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the various steps of the target detection method as described in the above embodiments.
[0125] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the foregoing embodiments have described the present invention in detail, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.
Claims
1. A target detection method, characterized in that, include: Obtain point cloud data of the image to be detected; Calculate the spherical mask of each original point cloud in the point cloud data, and the mask features of its spherical neighborhood; Key point sampling is performed on the point cloud data by sampling the farthest point based on a spherical mask to obtain a sampling point set based on the point cloud data. The sampling point set includes K key point clouds, the number of which is less than the number of the original point cloud. The mask features of the K key point clouds are enhanced by a global feature extraction layer based on a spherical neighborhood. K is a positive integer. Decoding is performed based on the coordinate information of K key point clouds and the enhanced mask features; The decoding results are subjected to predictive processing to obtain preliminary 3D suggestions; The primary 3D proposal is improved by a 3D region proposal network and nonmaximum suppression, resulting in a primary region of interest pool. The primary region of interest pool is filtered according to the intersection-union ratio threshold, and features are extracted from the filtered primary region of interest pool through a corner feature extraction layer and a grid pooling layer. The region of interest pool with extracted features is used as the target detection result of the image to be detected.
2. The target detection method as described in claim 1, characterized in that, Calculate the spherical mask of each original point cloud in the point cloud data, and the mask features of its spherical neighborhood, including: For any original point cloud, select all point clouds within the radius of a spherical neighborhood as the spherical mask of this original point cloud; calculate the maximum offset of all point clouds within the radius of the spherical neighborhood in the vertical direction to obtain the vertical features in the vertical direction, and the vertical features are the mask features of the spherical neighborhood to which this original point cloud belongs.
3. The target detection method as described in claim 2, characterized in that, Also includes: For the original point cloud with a vertical feature of 0, set its spherical mask to 0.
4. The target detection method as described in claim 3, characterized in that, Before performing keypoint sampling on the point cloud data using farthest-point sampling based on a spherical mask, the following steps are included: Set the spherical query radius; Based on the spherical query radius, a spherical mask is used for filtering, filtering out the original point clouds in the point cloud data whose spherical mask is 0.
5. The target detection method as described in claim 4, characterized in that, Key point sampling is performed on the point cloud data using farthest point sampling based on a spherical mask, including: Take any original point cloud in the point cloud data as the starting point; Obtain the spherical mask of the starting point; When the spherical mask of the starting point is 1, this original point cloud is added to the sampling point set; Calculate the distances from the remaining original point cloud to the starting point, and construct an N-dimensional distance array; Multiply the N-dimensional distance array by the spherical mask of the starting point to obtain the mask distance array; If the amount of data in the sampling point set is less than N, then the point with the largest value is selected from the mask distance array as the key point cloud; The key point cloud is added to the sampling point set, and the point with the largest value is removed from the mask distance array; Calculate the distance from the remaining original point cloud to the key point cloud. If the distance from the remaining original point cloud to the key point cloud is less than any element of the mask distance array, then take another original point cloud as the starting point, update the mask distance matrix, and return the result based on the updated mask distance matrix. If the data volume of the sampling point set is less than N, then select the point with the largest value from the mask distance array as the key point cloud.
6. The target detection method as described in claim 2, characterized in that, Enhance the mask features of K key point clouds through a global feature extraction layer based on spherical neighborhoods, including: For the k-th key point cloud, the X-axis position information is extracted by a global feature extraction layer based on a spherical neighborhood; k is a positive integer less than or equal to K. Calculate the ratio of the vertical feature to the X-axis position information, and use it as the first global feature; The local distance offset between the sampling point and the key point is obtained by the original feature extraction layer as the second global feature, and the LiDAR feature is used as the third global feature. The enhanced mask features include the first global feature, the second global feature, and the third global feature.
7. The target detection method as described in claim 1, characterized in that, The prediction process includes classification score prediction and 3D bounding box prediction.
8. A target detection device, characterized in that, include: The point cloud data acquisition module is used to acquire point cloud data of the image to be detected; The spherical mask calculation module is used to calculate the spherical mask of each original point cloud of the point cloud data, as well as the mask features of its spherical neighborhood. The key point sampling and feature enhancement module is used to sample key points in the point cloud data by sampling the farthest point based on a spherical mask to obtain a set of sampling points based on the point cloud data. The set of sampling points includes K key point clouds, the number of which is less than the number of the original point cloud. The mask features of the K key point clouds are enhanced by a global feature extraction layer based on a spherical neighborhood. K is a positive integer. The decoding module is used to decode based on the coordinate information of K key point clouds and the enhanced mask features; The prediction module is used to perform predictive processing on the decoding results to obtain preliminary 3D suggestions; The Region of Interest (ROI) output module is used to refine the primary 3D proposal through a 3D region proposal network and nonmaximum suppression to output a primary ROI pool. The target detection result output module is used to filter the primary region of interest pool according to the intersection-union ratio threshold, and extract features from the filtered primary region of interest pool through a corner feature extraction layer and a grid pooling layer, and use the region of interest pool after feature extraction as the target detection result of the image to be detected.
9. A terminal device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the various steps of the target detection method as described in any one of claims 1 to 7.
10. A storage medium, said storage medium being a computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements each step of the target detection method as described in any one of claims 1 to 7.