Target detection method and device, electronic equipment and machine readable storage medium

Abstract

The application provides a target detection method and device, electronic equipment and machine readable storage medium. The method comprises: voxelizing input point cloud data, and performing preliminary voxel feature extraction to obtain initial voxel features of the input point cloud data; according to the initial voxel features, classifying non-empty voxels corresponding to the input point cloud data into foreground and background, and determining the offset of each foreground voxel relative to a target center point; according to the offset of each foreground voxel relative to the target center point, clustering the foreground voxels to obtain a clustered target cluster; for any target cluster, using a self-attention mechanism to extract features of each foreground voxel in the target cluster to obtain target level features of each foreground voxel in the target cluster; and performing target detection according to the target level features of each foreground voxel to generate a target detection frame. The method can improve detection performance.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a target detection method, apparatus, electronic device, and machine-readable storage medium. Background Technology

[0002] Due to the sparsity and unstructured nature of point cloud data, Transformer (a network architecture) shows greater potential in point cloud data. Compared with traditional point cloud feature extraction methods based on sparse convolution, Transformer-based point cloud feature extraction methods can capture contextual information under a larger receptive field.

[0003] Currently, point cloud feature extraction methods based on Transformer all rely on the window mechanism, which divides all voxels into windows and inputs them into a self-attention mechanism for feature extraction.

[0004] However, the window division method in the above scheme relies too much on manual settings, and the window size cannot be adaptively adjusted. Maintaining a consistent window size for targets of different scales will affect the target detection performance. Summary of the Invention

[0005] In view of this, this application provides a target detection method, apparatus, electronic device, and machine-readable storage medium to optimize target detection performance.

[0006] Specifically, this application is implemented through the following technical solution:

[0007] According to a first aspect of the embodiments of this application, a target detection method is provided, comprising:

[0008] The input point cloud data is voxelized, and the voxelized features are initially extracted to obtain the initial voxel features of the input point cloud data.

[0009] Based on the initial voxel features, the non-empty voxels corresponding to the input point cloud data are classified as foreground and background, and the offset of each foreground voxel relative to its target center point is determined.

[0010] Based on the offset of each foreground voxel relative to its target center point, the foreground voxels are clustered to obtain the clustered target clusters;

[0011] For any target cluster, the self-attention mechanism is used to extract features from each foreground voxel in the target cluster to obtain the target-level features of each foreground voxel in the target cluster.

[0012] Based on the target-level features of each foreground voxel, target detection is performed, and target detection boxes are generated.

[0013] According to a second aspect of the embodiments of this application, a target detection apparatus is provided, comprising:

[0014] The initial feature extraction unit is used to voxelize the input point cloud data and perform preliminary voxel feature extraction to obtain the initial voxel features of the input point cloud data.

[0015] The target-level feature extraction unit is used to classify the non-empty voxels corresponding to the input point cloud data into foreground and background based on the initial voxel features, and determine the offset of each foreground voxel relative to its target center point; based on the offset of each foreground voxel relative to its target center point, the foreground voxels are clustered to obtain the clustered target clusters; for any target cluster, the self-attention mechanism is used to extract features from each foreground voxel in the target cluster to obtain the target-level features of each foreground voxel in the target cluster.

[0016] The target detection unit is used to perform target detection based on the target-level features of each foreground voxel and generate a target detection box.

[0017] According to a third aspect of the embodiments of this application, an electronic device is provided, including a processor and a memory, the memory storing machine-executable instructions executable by the processor, the processor being configured to execute the machine-executable instructions to implement the method provided in the first aspect.

[0018] According to a fourth aspect of the embodiments of this application, a machine-readable storage medium is provided, wherein machine-executable instructions are stored therein, and when the machine-executable instructions are executed by a processor, the method provided in the first aspect is implemented.

[0019] The technical solution provided in this application can bring at least the following beneficial effects:

[0020] By voxelizing the input point cloud data and performing preliminary voxel feature extraction, initial voxel features of the input point cloud data are obtained. Based on the obtained voxel features, the non-empty voxels corresponding to the input point cloud data are classified as foreground and background, and the offset of each foreground voxel relative to its target center point is determined. Based on the offset of each foreground voxel relative to its target center point, the foreground voxels are clustered to obtain clustered target clusters. Then, for any target cluster, a self-attention mechanism is used to extract features from each foreground voxel within the target cluster to obtain target-level features of each foreground voxel within the target cluster. Based on the target-level features of each foreground voxel, target detection is performed to generate target detection boxes. The feature extraction based on cluster division can adaptively adjust the range of each voxel participating in attention calculation, and each cluster can mainly consist of voxels of the same target, thereby enabling the model to focus on the extraction of target-level features and improve detection performance. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating an exemplary embodiment of the target detection method of this application;

[0022] Figure 2 This is an overall block diagram illustrating an attention-based point cloud feature extraction and object detection scheme in an exemplary embodiment of this application;

[0023] Figure 3 This is a schematic diagram of the structure of a target detection device shown in an exemplary embodiment of this application;

[0024] Figure 4 This is a schematic diagram of the structure of another target detection device shown in an exemplary embodiment of this application;

[0025] Figure 5 This is a schematic diagram of the structure of another target detection device shown in an exemplary embodiment of this application;

[0026] Figure 6 This is a schematic diagram of the hardware structure of an electronic device illustrated in an exemplary embodiment of this application. Detailed Implementation

[0027] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0028] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0029] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, and to make the above-mentioned objectives, features and advantages of the embodiments of this application more apparent and understandable, the technical solutions in the embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0030] It should be noted that the sequence number of each step in the embodiments of this application does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0031] Please see Figure 1 This is a flowchart illustrating a target detection method provided in an embodiment of this application, as shown below. Figure 1 As shown, the target detection method may include the following steps:

[0032] Step S100: Voxelize the input point cloud data and extract voxelized features to obtain the initial voxel features of the input point cloud data.

[0033] For example, the input point cloud data may include radar point cloud data, which may include, but is not limited to, point cloud data acquired by radars (such as lidar) deployed in fields such as autonomous driving or intelligent transportation.

[0034] For example, in autonomous driving scenarios, the input point cloud data can be the point cloud data obtained by the vehicle-mounted LiDAR.

[0035] Among them, voxels are a regularized representation of point clouds.

[0036] For example, feature extraction can be performed within each voxel using PointNet (a point cloud neural network) to obtain voxel features of the input point cloud data (which can be called initial voxel features).

[0037] Step S110: Based on the initial voxel features, classify the non-empty voxels corresponding to the input point cloud data into foreground and background, and determine the offset of each foreground voxel relative to its target center point.

[0038] For example, the offset of a foreground voxel relative to its target center point (i.e., the center point of the target to which the foreground voxel belongs) can be predicted using a pre-trained neural network model.

[0039] Step S120: Based on the offset of each foreground voxel relative to its target center point, cluster the foreground voxels to obtain the clustered target cluster.

[0040] In this embodiment of the application, after determining the offset of each foreground voxel relative to its target center point, the foreground voxels can be clustered according to the offset of each foreground voxel relative to its target center point to obtain the clustered target cluster, and foreground voxels that may belong to the same target can be clustered into one target cluster.

[0041] Step S130: For any target cluster, use the self-attention mechanism to extract features from each foreground voxel in the target cluster to obtain the target-level features of each foreground voxel in the target cluster.

[0042] In this embodiment of the application, considering that the voxel features extracted by traditional point cloud neural networks have limited receptive fields and lack target-level information, which affects the expressive ability of the features and is not conducive to target detection in point clouds, the self-attention mechanism in Transformer can be introduced to obtain the upper and lower information of a larger receptive field to improve the accuracy of point cloud target detection.

[0043] Accordingly, for any target cluster, the self-attention mechanism can be used to extract features from each foreground voxel within the target cluster, thereby obtaining the voxel features of each foreground voxel within the target cluster (which can be called target-level voxel features).

[0044] Step S140: Based on the target-level features of each foreground voxel, perform target detection and generate target detection boxes.

[0045] In this embodiment, target detection can be performed based on the target-level features of each foreground voxel to generate a target detection box.

[0046] For example, the target detection box can be represented by location information, length, width and height, orientation information and semantic category.

[0047] For example, the positional information of the target detection box can be characterized by the offset between the center point of the target detection box and the center coordinates of the corresponding target cluster.

[0048] For example, for any target cluster, the center coordinates of the target cluster can be the average of the coordinates of each moved foreground voxel in the target cluster.

[0049] It can be seen that, in Figure 1 In the illustrated method, the input point cloud data is voxelized, and initial voxel features are extracted to obtain initial voxel features. Based on the obtained voxel features, the non-empty voxels corresponding to the input point cloud data are classified as foreground and background, and the offset of each foreground voxel relative to its target center point is determined. Based on the offset of each foreground voxel relative to its target center point, the foreground voxels are clustered to obtain clustered target clusters. Then, for any target cluster, the self-attention mechanism is used to extract features from each foreground voxel within the target cluster to obtain target-level features of each foreground voxel within the target cluster. Based on the target-level features of each foreground voxel, target detection is performed to generate target detection boxes. The cluster-based feature extraction can adaptively adjust the range of each voxel participating in attention calculation, and each cluster can mainly consist of voxels of the same target, thereby enabling the model to focus on the extraction of target-level features and improve detection performance.

[0050] In some embodiments, the clustering of foreground voxels based on the offset of each foreground voxel relative to its target center point to obtain a clustered target cluster may include:

[0051] For any foreground voxel, based on the offset of the foreground voxel relative to the target center point, the foreground voxel is moved to a position closer to its target center point;

[0052] Based on the distance between the moved foreground voxels, add two moved foreground voxels whose distance is less than a preset distance threshold to the same connected component;

[0053] Foreground voxels belonging to the same connected domain are grouped into the same target cluster.

[0054] For example, given that the offset of each foreground voxel relative to its target center point is determined, for any given foreground voxel, the foreground voxel can be moved to a position closer to its target center point based on the offset of the foreground voxel relative to its target center point.

[0055] For example, the offset can be a three-dimensional vector, and moving a foreground voxel closer to its target center point based on the offset can be achieved by adding the offset to the three-dimensional coordinates of the foreground voxel.

[0056] For example, assuming the foreground voxel before the move is represented by v′, and the offset of the foreground voxel relative to its target center is Δ, then the foreground voxel after the move can be represented as:

[0057] v′ Δ =Δ+v′

[0058] For example, a distance matrix can be formed by calculating the pairwise distances between all moved foreground voxels. If the distance between two moved foreground voxels is less than a preset distance threshold, the two foreground voxels are determined to be connected. Then, all connected components in the distance matrix are searched, and foreground voxels belonging to the same connected component are classified into the same target cluster.

[0059] It should be noted that, in the embodiments of this application, the clustering of foreground voxels is not limited to the above-described method. For example, the moved foreground voxels can also be converted to a BEV map (coordinate projection), and the target center point can be determined based on the number of foreground voxels in each grid (pixel, or grid) in the BEV map. Once the target center point is determined, for any moved foreground voxel, the moved foreground voxel and the nearest target center point are assigned to the same target cluster based on the distance between the moved foreground voxel and each target center point.

[0060] For example, the grid with the most foreground voxels in a local area can be determined as the target center point based on the number of foreground voxels in each grid.

[0061] For example, we can traverse each grid. For any grid (which can be called the target grid), we determine the number of foreground voxels in each grid within a 3*3 region centered on the target grid. If the number of foreground voxels in the target grid is the largest, we determine the target grid as the target center point; otherwise, we continue to traverse the grid.

[0062] In some embodiments, the above-described feature extraction of each foreground voxel within the target cluster using a self-attention mechanism may include:

[0063] Using the voxel features of each foreground voxel within the target cluster as Query, Key, and Value features, a multi-head self-attention mechanism is used to extract features from each foreground voxel within the target cluster.

[0064] For example, multi-head attention mechanisms can be used to extract target-level features within target clusters.

[0065] For any target cluster, the voxel features of each foreground voxel within the target cluster can be used as Query (query vector), Key (key) features, and Value (value) features. Using a multi-head self-attention mechanism, features can be extracted from each foreground voxel within the target cluster.

[0066] For example, the voxel features of each foreground voxel in the target cluster can be used as Query, Key and Value features, and a two-layer multi-head self-attention can be used to extract features from each foreground voxel in the target cluster.

[0067] Since the voxels within each cluster typically originate from the same target, the clusters can be adaptively adjusted according to the size of the target, and the computational range of attention can also be adaptively adjusted. Therefore, this cluster-based self-attention mechanism can focus on feature extraction at the target level, thereby improving target detection performance.

[0068] In some embodiments, the above-mentioned classification of foreground and background of non-empty voxels corresponding to the input point cloud data based on initial voxel features, and determination of the offset of each foreground voxel relative to its target center point, includes:

[0069] The non-empty voxels corresponding to the input point cloud data are divided into multiple different groups according to their spatial location;

[0070] For any group, based on the initial voxel features of each non-empty voxel in the group, the self-attention mechanism is used to extract features of each non-empty voxel in the group, and the local level features of each foreground voxel in the group are obtained.

[0071] Based on the local-level features of the non-empty voxels corresponding to the input point cloud data, the non-empty voxels corresponding to the input point cloud data are classified into foreground and background, and the offset of each foreground voxel relative to its target center point is determined.

[0072] The above-mentioned use of self-attention mechanism to extract features from each foreground voxel within the target cluster, obtaining target-level features of each foreground voxel within the target cluster, may include:

[0073] Based on the local level features of each foreground voxel within the target cluster, a self-attention mechanism is used to extract features from each foreground voxel within the target cluster, thereby obtaining the target level features of each foreground voxel within the target cluster.

[0074] For example, in order to enrich the extracted voxel feature information and further improve the target detection performance, cluster-based feature extraction can be combined with window-based feature extraction.

[0075] Accordingly, once the initial voxel features are obtained, the non-empty voxels corresponding to the input point cloud data can be divided into multiple different groups according to their spatial location.

[0076] For example, the voxels corresponding to the input point cloud data can be projected onto the BEV (Bird Eye View) perspective to obtain a BEV map, and the BEV map can be divided into multiple non-overlapping windows, with non-empty voxels within the same window considered as the same group.

[0077] For example, for any group, based on the initial voxel features of each non-empty voxel within the group, a self-attention mechanism can be used to extract features from each non-empty voxel in the group, resulting in voxel features (which can be called local-level features) for each non-empty voxel. Having obtained the local-level features of the non-empty voxels corresponding to the input point cloud data in the above manner, the non-empty voxels corresponding to the input point cloud data can be classified as foreground and background based on these local-level features, and the offset of each foreground voxel relative to its target center point can be determined.

[0078] Furthermore, for any target cluster, based on the local level features of each foreground voxel within the target cluster, a self-attention mechanism can be used to further extract features from each foreground voxel within the target cluster, thereby obtaining the target level features of each foreground voxel within the target cluster.

[0079] In one example, to enable the model to better learn local-level and target-level feature information, based on the initial voxel features, multiple local-level feature extractions (i.e., window-based feature extractions) and target-level feature extractions (i.e., cluster-based feature extractions) are performed at intervals on the foreground voxels corresponding to the input point cloud data.

[0080] For example, taking two local-level feature extractions and target-level feature extractions as an example, for the initial voxel features, one local-level feature extraction can be performed first, and then a target-level feature extraction can be performed based on the extracted local-level features. Then, another local-level feature extraction can be performed based on the extracted target-level features, and then a target-level feature extraction can be performed based on the extracted local-level features.

[0081] It should be noted that, in the embodiments of this application, during the multiple local-level feature extraction and target-level feature extraction processes performed at the above intervals, the input feature for the first local-level feature extraction is the initial voxel feature, and the subsequent feature extraction is the voxel feature output by the previous feature extraction (local-level feature extraction or target-level feature extraction).

[0082] In cases where multiple target-level feature extractions are required (i.e., cluster-based feature extraction), for each cluster-based feature extraction, the foreground voxels can be re-clustered based on the current voxel features to obtain an updated target cluster, and feature extraction can be performed based on the updated target cluster.

[0083] The process of re-clustering foreground voxels may include re-predicting the center point offset of each foreground voxel based on the current voxel features, and then re-clustering the foreground voxels based on the updated center point offset.

[0084] For example, assuming that local-level feature extraction (i.e., feature extraction based on window partitioning) and cluster-based feature extraction are performed alternately, for a cluster-based feature extraction, the foreground voxels can be re-clustered based on the local-level features obtained from the previous window-based feature extraction to obtain the updated target cluster, and feature extraction can be performed based on the updated target cluster.

[0085] In some embodiments, the above-mentioned target detection based on the target-level features of each foreground voxel to generate a target detection box may include:

[0086] Based on the target level features of each foreground voxel, the foreground voxels corresponding to the input point cloud data are re-clustered to obtain the updated target clusters;

[0087] For any updated target cluster, the representative features of the updated target cluster are determined based on the target-level features of each foreground voxel within the updated target cluster.

[0088] Based on the representative features of each updated target cluster, a self-attention mechanism is used to enhance the representative features of each updated target cluster, resulting in enhanced features for each updated target cluster.

[0089] Based on the enhanced features of each updated target cluster, target detection is performed, and target detection boxes are generated.

[0090] For example, in order to enhance the global information in the extracted voxel features and further improve the target detection performance, the target-level features of each foreground voxel within each target cluster can be aggregated, and the aggregated features can be enhanced through the interaction between target clusters.

[0091] For example, if the target-level features of each foreground voxel are obtained in accordance with the manner described in the above embodiments, the foreground voxels corresponding to the input point cloud data can be re-clustered based on the target-level features of each foreground voxel to obtain an updated target cluster.

[0092] It should be noted that, in the embodiments of this application, for the scenario combining cluster-based feature extraction and window-based feature extraction, the target-level features of each foreground voxel mentioned above refer to the voxel features obtained after performing local-level feature extraction and target-level feature extraction at one or more intervals.

[0093] For example, the specific implementation of re-clustering the foreground voxels corresponding to the input point cloud data based on the target level features of each foreground voxel can be found in the relevant description of foreground voxel clustering in the above embodiments, and will not be repeated here in the embodiments of this application.

[0094] For any updated target cluster, the representative features (i.e., aggregate features) of the updated target cluster can be determined based on the target-level features of each foreground voxel within the updated target cluster.

[0095] In one example, for any updated target cluster, the target-level features of each foreground voxel within the updated target cluster can be max-pooled to obtain the representative features of the updated target cluster.

[0096] Based on the representative features of each updated target cluster, a self-attention mechanism can be used to enhance the representative features of each updated target cluster, resulting in enhanced features for each updated target cluster. Then, based on the enhanced features of each updated target cluster, target detection can be performed to generate target detection boxes.

[0097] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, the technical solutions provided in the embodiments of this application are described below with reference to specific examples.

[0098] This embodiment provides an attention-based point cloud feature extraction and target detection scheme. In addition to window partitioning, the scheme also designs a cluster-based voxel partitioning method. By predicting the center point offset of each foreground voxel, it divides them into different clusters, ensuring that voxels within the same cluster primarily belong to the same target. Based on this partitioning method, the scope of each voxel participating in attention calculation can be adaptively adjusted, enhancing the target-level information of voxel features and improving detection performance.

[0099] The overall block diagram of the attention-based point cloud feature extraction and object detection scheme provided in this embodiment can be seen as follows: Figure 2As shown, it mainly includes: an input point cloud data preprocessing module, an attention-based feature extraction module (hereinafter referred to as the feature extraction module), a feature aggregation module, and a point cloud target detection head. The attention-based feature extraction module can include: a window-based feature extraction module and a cluster-based feature extraction module.

[0100] The following is a detailed introduction to each module.

[0101] 1. Input point cloud data preprocessing module

[0102] For the input point cloud signal p∈R t×3 Where t is the number of point clouds, and 3 represents the x, y, and z coordinates of the point clouds (3D coordinates).

[0103] Voxelization of the point cloud yields voxel signals v∈R m×3 , where m is the number of non-empty voxels used, and 3 represents the coordinate position information of the voxels.

[0104] Within each voxel, PointNet is used to extract features, resulting in initial voxel features v. f ∈R m×c Where c is the number of feature channels.

[0105] Optionally, c can be set to 128.

[0106] Traditional point cloud feature extraction networks have limited receptive fields for voxel features and lack target-level information, which affects the expressive power of features and is not conducive to target detection in point clouds. Therefore, the self-attention mechanism in Transformer can be introduced to obtain vertical information of a larger receptive field to improve the accuracy of point cloud target detection.

[0107] 2. Feature Extraction Module

[0108] For example, the feature extraction module may include multiple window-based feature extraction modules and multiple cluster-based feature extraction modules cascaded at intervals.

[0109] 2.1 Feature Extraction Module Based on Window Partitioning

[0110] Based on the acquired initial voxel features v f ∈R m×c Non-empty voxels are divided into different groups according to their spatial location.

[0111] One way to implement group partitioning is as follows: Voxels v∈R m×3 When projected onto the BEV viewpoint, the length and width of the BEV image are H and W, respectively.

[0112] Divide the BEV graph into multiple non-overlapping windows W = {w1, w2, ... w} i ..w n}, where n is the number of partitioned windows.

[0113] Non-empty voxels within the same window are considered as the same group, i.e. Where, n i This represents the number of voxels within the i-th window.

[0114] A two-layer multi-head self-attention mechanism can be used to extract voxel features from each window (or group).

[0115] For example, the expression for a multi-head self-attention mechanism within a certain window is as follows:

[0116]

[0117] Where Q (Query), K (Key feature), and V (Value feature) are the three input feature vectors of the multi-head self-attention mechanism. In this embodiment, the obtained initial voxel feature v f ∈R m×c Let Q, K, and V be the values ​​respectively; D is the normalization parameter, which in this embodiment can be set to the number of channels of the voxel feature, such as 128; i represents the i-th window, O i This represents the voxel feature output by the i-th window after passing through the self-attention mechanism.

[0118] Through the above processing, voxel features can acquire contextual information within a window, thereby improving the expressive power of voxel features.

[0119] However, considering that object detection tasks require not only long-range (within-window) contextual information but also target-level contextual information to achieve better detection performance; furthermore, the size of these windows cannot be adaptively adjusted for targets of different scales. Therefore, the scheme provided in this application can further incorporate a cluster-based feature extraction module.

[0120] 2.2 Feature Extraction Module Based on Cluster Partitioning

[0121] The non-empty voxel features obtained from the above operations are input into a multilayer perceptron, for example, a multilayer perceptron consisting of two fully connected layers, each with 512 channels, to classify foreground and background voxels, and to predict the center point offset Δ∈R of the foreground voxels. (m×3) .

[0122] Here, foreground voxels represent voxels within the target detection box, and center point offset represents the offset of the foreground voxels relative to the center point of the detection box.

[0123] For all voxels classified as foreground (i.e., foreground voxels) v′, the positions of these voxels are added to the predicted center point offset Δ, so that each moved foreground voxel is closer to its respective target center, which can be expressed as:

[0124] v′ Δ =Δ+v′

[0125] Based on the distance, these moved foreground voxels are divided into different target clusters.

[0126] An alternative implementation method: a method based on connected components in three-dimensional space.

[0127] The specific implementation method is as follows: calculate the pairwise distances between all moved foreground voxels to form a distance matrix M∈R. m′×m′ (m′ represents the number of foreground voxels). If the distance between two moved foreground voxels is less than a preset distance threshold, the two foreground voxels are determined to be connected. Then, all connected components in the distance matrix are searched. Foreground voxels belonging to the same connected component are classified into the same target cluster. In this way, multiple target clusters C = {c1, c2, ..., c} can be obtained. j ...c k}, where k represents the number of target clusters, Let n represent the j-th target cluster. j Let be the number of voxels within the j-th target cluster. After obtaining the different voxels based on cluster division, two layers of multi-head self-attention can be used to further extract features within each cluster.

[0128]

[0129] In this context, all foreground voxel features that are clustered are set to Q, K, V; D is the normalization parameter, and O is the normalization parameter. j This represents the voxel feature output by the j-th target cluster after passing through the self-attention mechanism.

[0130] Since the voxels within each cluster typically originate from the same target, the clusters are adaptively adjusted according to the size of the target, and the computational scope of attention is also adaptively adjusted. Therefore, this cluster-based self-attention mechanism can focus on feature extraction at the target level, improving the accuracy of target detection.

[0131] 3. Feature aggregation module

[0132] Based on the voxel features v′ processed by the feature extraction module f Then, perform another clustering operation, which is the same as the clustering operation described above, with each cluster responsible for detecting one target.

[0133] Based on the voxel features of these clusters, a feature aggregation module is designed to aggregate and enhance the features of each cluster.

[0134] An alternative implementation: Based on the clustering results described above, perform max-pooling on the voxel features within the same cluster to obtain... And As representative features of the j-th cluster, subsequent detection results are also obtained based on the representative features of these clusters.

[0135] Since the attention operations in the feature extraction module are all within a window or a cluster, they lack global contextual information.

[0136] Therefore, after obtaining the representative features of these clusters, a multi-head self-attention mechanism is used again to allow the representative features of these clusters to interact, so as to obtain global contextual information to improve the localization accuracy of the target.

[0137] For example, the representative features of each cluster can be used as Q, K, and V, and a multi-head self-attention mechanism can be used to enhance the representative features of each cluster to obtain the enhanced features of each cluster (i.e., enhanced features).

[0138] 4. Point cloud target detection head

[0139] For example, based on the obtained cluster features (enhanced features), two fully connected layers can be used to generate the location information (Δx, Δy, Δz), length, width and height (log(l), log(w), log(h)), orientation information (sin(α), cos(α)), and semantic category of the detection box.

[0140] Where (Δx, Δy, Δz) are the offset values ​​of the detection box center point relative to the cluster center point. α represents the orientation angle information. The coordinate values ​​of the cluster center are obtained from the average position of the foreground voxel after movement within each cluster.

[0141] As can be seen, the cluster-based feature extraction module can adaptively adjust the scope of each voxel's participation in attention calculation, and each cluster can mainly consist of voxels of the same target, thereby enabling the model to focus on the extraction of target-level features and improve target detection performance.

[0142] In addition, the feature aggregation module can aggregate representative features of clusters and enhance the global information in the cluster representative features, which can further improve the target detection performance.

[0143] The method provided in this application has been described above. The apparatus provided in this application is described below:

[0144] Please see Figure 3This is a schematic diagram of the structure of a target detection device provided in an embodiment of this application, as shown below. Figure 3 As shown, the target detection device may include:

[0145] The initial feature extraction unit 310 is used to voxelize the input point cloud data and perform preliminary voxel feature extraction to obtain the initial voxel features of the input point cloud data.

[0146] The target-level feature extraction unit 320 is used to classify the non-empty voxels corresponding to the input point cloud data into foreground and background based on the initial voxel features, and determine the offset of each foreground voxel relative to its target center point; based on the offset of each foreground voxel relative to its target center point, the foreground voxels are clustered to obtain the clustered target clusters; for any target cluster, the self-attention mechanism is used to extract features from each foreground voxel in the target cluster to obtain the target-level features of each foreground voxel in the target cluster.

[0147] The target detection unit 330 is used to perform target detection based on the target-level features of each foreground voxel and generate a target detection box.

[0148] In some embodiments, the target-level feature extraction unit 320 clusters the foreground voxels based on the offset of each foreground voxel relative to its target center point to obtain clustered target clusters, including:

[0149] For any foreground voxel, based on the offset of the foreground voxel relative to the target center point, the foreground voxel is moved to a position closer to its target center point;

[0150] Based on the distance between the moved foreground voxels, add two moved foreground voxels whose distance is less than a preset distance threshold to the same connected component;

[0151] Foreground voxels belonging to the same connected domain are grouped into the same target cluster.

[0152] In some embodiments, the target-level feature extraction unit 320 uses a self-attention mechanism to extract features from each foreground voxel within the target cluster, including:

[0153] Using the voxel features of each foreground voxel within the target cluster as the query vector, key features, and value features, a multi-head self-attention mechanism is used to extract features from each foreground voxel within the target cluster.

[0154] In some embodiments, such as Figure 4 As shown, the device further includes:

[0155] The local-level feature extraction unit 340 is used to divide the non-empty voxels corresponding to the input point cloud data into multiple different groups according to their spatial location; for any group, based on the initial voxel features of each non-empty voxel in the group, the self-attention mechanism is used to extract features of each non-empty voxel in the group to obtain the local-level features of each non-empty voxel in the group.

[0156] The target-level feature extraction unit 320, based on the initial voxel features, classifies the non-empty voxels corresponding to the input point cloud data into foreground and background, and determines the offset of each foreground voxel relative to its target center point, including:

[0157] Based on the local level features of the non-empty voxels corresponding to the input point cloud data, the non-empty voxels corresponding to the input point cloud data are classified as foreground and background, and the offset of each foreground voxel relative to its target center point is determined.

[0158] The target-level feature extraction unit 320 uses a self-attention mechanism to extract features from each foreground voxel within the target cluster, obtaining target-level features for each foreground voxel within the target cluster, including:

[0159] Based on the local level features of each foreground voxel within the target cluster, a self-attention mechanism is used to extract features from each foreground voxel within the target cluster, thereby obtaining the target level features of each foreground voxel within the target cluster.

[0160] In some embodiments, based on the initial voxel features, multiple local-level feature extractions and target-level feature extractions are performed at intervals on the foreground voxels corresponding to the input point cloud data.

[0161] In some embodiments, such as Figure 5 As shown, the device further includes:

[0162] The feature aggregation and enhancement unit 350 is used to re-cluster the foreground voxels corresponding to the input point cloud data according to the target level features of each foreground voxel to obtain an updated target cluster; for any updated target cluster, the representative features of the updated target cluster are determined according to the target level features of each foreground voxel in the updated target cluster; based on the representative features of each updated target cluster, the representative features of each updated target cluster are enhanced using a self-attention mechanism to obtain the enhanced features of each updated target cluster.

[0163] The target detection unit 330 performs target detection based on the target-level features of each foreground voxel, generating a target detection box, including:

[0164] Based on the enhanced features of each updated target cluster, target detection is performed, and target detection boxes are generated.

[0165] In some embodiments, the feature aggregation enhancement unit determines representative features of the updated target cluster based on the target-level features of each foreground voxel within the updated target cluster, including:

[0166] Max pooling is performed on the target-level features of each foreground voxel within the updated target cluster to obtain the representative features of the updated target cluster.

[0167] This application provides an electronic device including a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement the target detection method described above.

[0168] Please see Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device may include a processor 601 and a memory 602 storing machine-executable instructions. The processor 601 and the memory 602 can communicate via a system bus 603. Furthermore, by reading and executing the machine-executable instructions corresponding to the target detection logic in the memory 602, the processor 601 can execute the target detection method described above.

[0169] The memory 602 mentioned in this document can be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0170] In some embodiments, a machine-readable storage medium, such as Figure 6 The memory 602 in the machine-readable storage medium stores machine-executable instructions, which, when executed by a processor, implement the target detection method described above. For example, the storage medium may be ROM, RAM, CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0171] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0172] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A target detection method, characterized in that, include: The input point cloud data is voxelized, and the voxelized features are initially extracted to obtain the initial voxel features of the input point cloud data. Based on the initial voxel features, the non-empty voxels corresponding to the input point cloud data are classified as foreground and background, and the offset of each foreground voxel relative to its target center point is determined. For any foreground voxel, based on the offset of the foreground voxel relative to the target center point, the foreground voxel is moved to a position closer to its target center point; based on the distance between the moved foreground voxels, two moved foreground voxels with a distance less than a preset distance threshold are added to the same connected component; foreground voxels belonging to the same connected component are assigned to the same target cluster. For any target cluster, the self-attention mechanism is used to extract features from each foreground voxel in the target cluster to obtain the target-level features of each foreground voxel in the target cluster. Based on the target-level features of each foreground voxel, target detection is performed, and target detection boxes are generated.

2. The method according to claim 1, characterized in that, The feature extraction of each foreground voxel within the target cluster using a self-attention mechanism includes: Using the voxel features of each foreground voxel within the target cluster as the query vector, key features, and value features, a multi-head self-attention mechanism is used to extract features from each foreground voxel within the target cluster.

3. The method according to claim 1, characterized in that, The step of classifying the non-empty voxels corresponding to the input point cloud data into foreground and background based on the initial voxel features, and determining the offset of each foreground voxel relative to its target center point, includes: The non-empty voxels corresponding to the input point cloud data are divided into multiple different groups according to their spatial location; For any group, based on the initial voxel features of each non-empty voxel in the group, the self-attention mechanism is used to extract features of each non-empty voxel in the group, and the local level features of each non-empty voxel in the group are obtained. Based on the local level features of the non-empty voxels corresponding to the input point cloud data, the non-empty voxels corresponding to the input point cloud data are classified as foreground and background, and the offset of each foreground voxel relative to its target center point is determined. The method of extracting features from each foreground voxel within the target cluster using a self-attention mechanism to obtain target-level features of each foreground voxel within the target cluster includes: Based on the local level features of each foreground voxel within the target cluster, a self-attention mechanism is used to extract features from each foreground voxel within the target cluster, thereby obtaining the target level features of each foreground voxel within the target cluster.

4. The method according to claim 3, characterized in that, Based on the initial voxel features, multiple local-level feature extractions and target-level feature extractions are performed at intervals on the foreground voxels corresponding to the input point cloud data.

5. The method according to claim 1, characterized in that, The step of performing target detection and generating target detection boxes based on the target-level features of each foreground voxel includes: Based on the target level features of each foreground voxel, the foreground voxels corresponding to the input point cloud data are re-clustered to obtain the updated target clusters; For any updated target cluster, the representative features of the updated target cluster are determined based on the target-level features of each foreground voxel within the updated target cluster. Based on the representative features of each updated target cluster, a self-attention mechanism is used to enhance the representative features of each updated target cluster, resulting in enhanced features for each updated target cluster. Based on the enhanced features of each updated target cluster, target detection is performed, and target detection boxes are generated.

6. The method according to claim 5, characterized in that, The step of determining the representative features of the updated target cluster based on the target-level features of each foreground voxel within the updated target cluster includes: Max pooling is performed on the target-level features of each foreground voxel within the updated target cluster to obtain the representative features of the updated target cluster.

7. A target detection device, characterized in that, include: The initial feature extraction unit is used to voxelize the input point cloud data and perform preliminary voxel feature extraction to obtain the initial voxel features of the input point cloud data. The target-level feature extraction unit is used to classify the non-empty voxels corresponding to the input point cloud data into foreground and background based on the initial voxel features, and determine the offset of each foreground voxel relative to its target center point; for any foreground voxel, based on the offset of the foreground voxel relative to the target center point, move the foreground voxel to a position closer to its target center point; based on the distance between the moved foreground voxels, add two moved foreground voxels with a distance less than a preset distance threshold to the same connected component; and classify the foreground voxels belonging to the same connected component into the same target cluster. For any target cluster, the self-attention mechanism is used to extract features from each foreground voxel in the target cluster to obtain the target-level features of each foreground voxel in the target cluster. The target detection unit is used to perform target detection based on the target-level features of each foreground voxel and generate a target detection box.

8. The apparatus according to claim 7, characterized in that, The target-level feature extraction unit utilizes a self-attention mechanism to extract features from each foreground voxel within the target cluster, including: Using the voxel features of each foreground voxel within the target cluster as the query vector, key features, and value features, a multi-head self-attention mechanism is used to extract features from each foreground voxel within the target cluster. And / or, The device further includes: The local-level feature extraction unit is used to divide the non-empty voxels corresponding to the input point cloud data into multiple different groups according to their spatial location. For any group, based on the initial voxel features of each non-empty voxel in the group, the self-attention mechanism is used to extract features of each non-empty voxel in the group to obtain the local-level features of each non-empty voxel in the group. The target-level feature extraction unit, based on the initial voxel features, classifies the non-empty voxels corresponding to the input point cloud data into foreground and background, and determines the offset of each foreground voxel relative to its target center point, including: Based on the local level features of the non-empty voxels corresponding to the input point cloud data, the non-empty voxels corresponding to the input point cloud data are classified as foreground and background, and the offset of each foreground voxel relative to its target center point is determined. The target-level feature extraction unit uses a self-attention mechanism to extract features from each foreground voxel within the target cluster, obtaining target-level features for each foreground voxel within the target cluster, including: Based on the local level features of each foreground voxel within the target cluster, a self-attention mechanism is used to extract features from each foreground voxel within the target cluster, thereby obtaining the target level features of each foreground voxel within the target cluster. Among them, based on the initial voxel features, multiple local-level feature extractions and target-level feature extractions are performed at intervals on the foreground voxels corresponding to the input point cloud data; And / or, The device further includes: The feature aggregation and enhancement unit is used to re-cluster the foreground voxels corresponding to the input point cloud data based on the target level features of each foreground voxel to obtain an updated target cluster; for any updated target cluster, the representative features of the updated target cluster are determined based on the target level features of each foreground voxel in the updated target cluster; based on the representative features of each updated target cluster, the representative features of each updated target cluster are enhanced using a self-attention mechanism to obtain the enhanced features of each updated target cluster. The target detection unit performs target detection based on the target-level features of each foreground voxel, generating a target detection box, including: Based on the enhanced features of each updated target cluster, target detection is performed, and target detection boxes are generated. The feature aggregation enhancement unit determines the representative features of the updated target cluster based on the target-level features of each foreground voxel within the updated target cluster, including: Max pooling is performed on the target-level features of each foreground voxel within the updated target cluster to obtain the representative features of the updated target cluster.

9. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing machine-executable instructions that can be executed by the processor, the processor executing the machine-executable instructions to implement the method as described in any one of claims 1-6.

10. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores machine-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1-6.

Images