Target detection method, device and electronic equipment

Abstract

Embodiments of the present application provide a target detection method and device and electronic equipment, and relate to the technical field of point cloud processing. The target detection method comprises: taking point cloud data comprising four dimension information as input data, performing multi-layer reflection intensity bird's eye view coding on the point cloud data, and extracting reflection intensity bird's eye view features; performing three-dimensional voxel coding on the point cloud data, and extracting three-dimensional voxel features; after fusing the reflection intensity bird's eye view features and the three-dimensional voxel features, performing detection on the fusion result by using a first stage detection head to obtain three-dimensional region proposals; performing pooling on the three-dimensional voxel features based on the three-dimensional region proposals by using a three-dimensional voxel region of interest pooling layer to obtain region of interest features; and obtaining a detection result of the point cloud data by using a second stage detection head. The embodiments provided by the present application have the advantages of good discrimination and high processing effect.

Description

Technical Field

[0001] This invention relates to the field of point cloud processing technology, and more specifically to a target detection method, a target detection device, an electronic device, and a storage medium. Background Technology

[0002] With the rapid development of autonomous driving technology, 3D object detection, as a key task in scene understanding, has received increasing attention. 3D object detection aims to predict the position, size, orientation, and category of key objects (such as cars, pedestrians, and cyclists) in 3D space. Compared to images, LiDAR point clouds contain more geometric and semantic information and are not limited by lighting conditions, making them a popular input data source for mainstream 3D object detection algorithms.

[0003] Point cloud data is unstructured, therefore standard convolution cannot be used directly; effective encoding of the point cloud is required. Currently, the main encoding methods include: point-based methods, voxel-based methods, and point-voxel-based methods. Point-based methods require operation on each point in the point cloud and aggregate point-by-point features using permutation-invariant operators (such as multilayer perceptrons), which incurs significant computational overhead. Voxel-based methods voxelize the point cloud and then extract features through 3D convolution. Since the introduction of 3D sparse convolution has greatly improved the inference speed of networks, voxel-based methods have become the preferred mainstream algorithm. While maintaining speed, some geometric information in certain regions is lost during voxel downsampling. To compensate for the lost geometric information in voxel-based methods, point-voxel-based methods have emerged, fully utilizing the advantages of point-based methods in preserving geometric information. However, these algorithms still suffer from the high computational overhead faced by point-based methods.

[0004] Another problem that needs to be addressed is that detectors make incorrect detections of objects with similar shapes. Existing algorithms misidentify road signs, trees, and flower beds as pedestrians, billboards as cyclists, and buildings and bushes as cars. These misidentifications are due to the similarity of the objects' geometric information. In-depth research has revealed that previous algorithms focused primarily on the geometric information carried in the point cloud coordinates, but did not delve into the reflection intensity information, thus causing the algorithms to make false detections when faced with objects with similar shapes. Summary of the Invention

[0005] The purpose of this invention is to provide a target detection method and a chip using this method, so that the trained model has high efficiency and the ability to distinguish objects with similar shapes, thereby solving at least some of the problems in the background art.

[0006] To achieve the above objectives, the first aspect of the present invention provides a target detection method, the method comprising: using point cloud data including four-dimensional information as input data, the four dimensions including three-dimensional coordinate information and reflection intensity;

[0007] The point cloud data is encoded with multi-layer reflection intensity bird's-eye view, and a reflection intensity bird's-eye view feature extraction network consisting of up / downsampling modules, convolutional modules and connection layers is used to extract reflection intensity bird's-eye view features from the encoding results.

[0008] The point cloud data is 3D voxel encoded, and a 3D voxel feature extraction network consisting of an encoding layer, a convolutional module, and a transformation module is used to extract 3D voxel features from the encoding results.

[0009] After fusing the bird's-eye view features of reflection intensity and the three-dimensional voxel features, a first-stage detection head composed of a feature extraction sub-network, a multi-scale sub-network and a convolutional layer is used to detect the fusion result to obtain a three-dimensional region suggestion.

[0010] A three-dimensional voxel region of interest pooling layer is used to pool the three-dimensional voxel features based on the proposed three-dimensional region to obtain the region of interest features;

[0011] The second-stage detection head is used to perform feature transformation on the features of the region of interest, and then performs bounding box regression and confidence prediction to obtain the detection results of the point cloud data.

[0012] Preferably, the point cloud data is encoded using a multi-layer bird's-eye view of reflectance intensity, and a bird's-eye view feature extraction network consisting of upsampling / downsampling modules, convolutional modules, and connection layers is used to extract bird's-eye view features of reflectance intensity from the encoding result. This includes: slicing the reflectance intensity in the point cloud data along the height direction in a bird's-eye view manner; projecting the reflectance intensity of the point cloud into a two-dimensional pseudo-image at a preset resolution according to coordinates within each small space after slicing, as the encoding result; the encoding result is then processed through multiple downsampling modules, convolutional modules and connection layers, multiple upsampling modules, and convolutional modules to obtain the bird's-eye view features of reflectance intensity; wherein, the first downsampling module in the multiple downsampling modules consists of two convolutional layers, and subsequent downsampling modules use a RestNet residual structure; the upsampling module uses a deconvolutional layer; and the connection layer is a horizontal connection layer, set between the upsampling module and the corresponding downsampling module.

[0013] Preferably, the point cloud data is subjected to 3D voxel encoding, and a 3D voxel feature extraction network consisting of an encoding layer, a convolutional module, and a transformation module is used to extract 3D voxel features from the encoding result. This includes: uniformly subdividing the 3D space where the point cloud data is located into equally spaced voxels, encoding the four-dimensional information of the point cloud within each voxel to obtain the encoding result; the encoding result is then passed through an encoding layer, multiple convolutional modules, and a transformation module in sequence to obtain the 3D voxel features; wherein, the encoding layer is a voxel feature encoding layer; the convolutional module is a 3D sparse convolutional module, the first convolutional module among the multiple convolutional modules consists of two layers of submanifold sparse convolution, and subsequent convolutional modules consist of one layer of spatial sparse convolution with a stride of 2 and two layers of submanifold sparse convolution; the transformation module is a sparse feature conversion layer for dense BEV feature conversion.

[0014] Preferably, after fusing the bird's-eye view features of reflection intensity and the three-dimensional voxel features, a first-stage detection head composed of a feature extraction sub-network, a multi-scale sub-network, and convolutional layers is used to detect the fusion result to obtain a three-dimensional region proposal. This includes: fusing the bird's-eye view features of reflection intensity and the three-dimensional voxel features to obtain a fusion result; after the fusion result passes through a network composed of a top-down feature extraction sub-network consisting of two convolutional layers and a multi-scale sub-network for upsampling and feature fusion, the output of the network is convolved with two parallel convolutional layers to generate a three-dimensional region proposal.

[0015] Preferably, a three-dimensional voxel region of interest pooling layer is used to pool the three-dimensional voxel features based on the three-dimensional region proposal to obtain region of interest features, including: extracting aggregated features using a voxel pooling method at different stages; concatenating aggregated features of different stages and scales to obtain voxel region of interest features; the voxel pooling method includes the following steps: obtaining the index of adjacent voxels based on the three-dimensional coordinate information of the center voxel and the preset voxel position offset, and then grouping the voxels; aggregating the features of adjacent non-empty voxels to the located center point for feature extraction based on the three-dimensional region proposal; using a fully connected layer to learn the features of adjacent voxels at different distances, and then using max pooling to obtain the aggregated features of K non-empty adjacent voxels.

[0016] Preferably, the second-stage detection head performs feature transformation on the features of the region of interest and performs bounding box regression and confidence prediction respectively to obtain the detection result of the point cloud data, including: using a shared multilayer perceptron to convert the features of the region of interest into feature vectors, and inputting the feature vectors into the bounding box regression branch and the confidence prediction branch respectively to obtain the detection result of the point cloud data; the bounding box regression branch is used to predict the residual from the 3D region proposal to the ground truth, and the confidence prediction branch is used to predict the confidence score related to IoU.

[0017] Preferably, the parameter weights in the bird's-eye view feature extraction network for reflection intensity, the three-dimensional voxel feature extraction network, the first-stage detection head, and the second-stage detection head are obtained through the following training steps: constructing a loss function based on position loss and classification loss; obtaining training samples and determining training control parameters; and using the ADAM optimizer to train in an end-to-end manner based on the loss function, training samples, and training control parameters.

[0018] In a second aspect of the invention, a target detection device is also provided, the device comprising: a raw data module for using point cloud data including four-dimensional information as input data, the four dimensions including three-dimensional coordinate information and reflection intensity; a first feature module for performing multi-layer reflection intensity bird's-eye view encoding on the point cloud data, and extracting reflection intensity bird's-eye view features from the encoding result using a reflection intensity bird's-eye view feature extraction network composed of an up / downsampling module, a convolution module, and a connection layer; and a second feature module for performing three-dimensional voxel encoding on the point cloud data, and extracting three-dimensional voxel feature extraction network composed of an encoding layer, a convolution module, and a transformation module. The system extracts three-dimensional voxel features from the encoding results; a fusion detection module is used to fuse the reflection intensity bird's-eye view features and the three-dimensional voxel features, and then uses a first-stage detection head composed of a feature extraction sub-network, a multi-scale sub-network, and a convolutional layer to detect the fused results and obtain three-dimensional region proposals; a pooling module is used to pool the three-dimensional voxel features based on the three-dimensional region proposals using a three-dimensional voxel region of interest pooling layer to obtain region of interest features; and a result output module is used to perform feature transformation on the region of interest features using a second-stage detection head and then perform bounding box regression and confidence prediction to obtain the detection results of the point cloud data.

[0019] A fourth aspect of the present invention provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor implements the steps of the aforementioned target detection method by executing the instructions stored in the memory.

[0020] A fourth aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the aforementioned target detection method.

[0021] The fifth aspect of the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned target detection method.

[0022] The above technical solution has the following beneficial effects:

[0023] First, a point cloud encoding method based on bird's-eye view of reflection intensity and voxels is proposed, and a corresponding feature pyramid backbone network architecture is designed to solve the problem that current point cloud algorithms cannot effectively distinguish objects with similar shapes.

[0024] Secondly, the use of computationally inexpensive 3D sparse convolution and voxel pooling with low time complexity ensures the network's processing efficiency.

[0025] Finally, the embodiments of the present invention have better versatility and can be applied to various traffic scenarios.

[0026] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0027] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings:

[0028] Figure 1 The schematic diagram illustrates the steps of a target detection method according to an embodiment of the present invention;

[0029] Figure 2 This schematic diagram illustrates the structural framework of a target detection method according to an embodiment of the present invention.

[0030] Figure 3 The schematic diagram illustrates the structure of a bird's-eye view feature extraction network for reflection intensity according to an embodiment of the present invention;

[0031] Figure 4 The schematic diagram illustrates the structure of a three-dimensional voxel feature extraction network according to an embodiment of the present invention;

[0032] Figure 5 The schematic diagram illustrates the structure of a target detection device according to an embodiment of the present invention. Detailed Implementation

[0033] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the scope of the present invention.

[0034] Figure 1 The schematic diagram illustrates the steps of a target detection method according to an embodiment of the present invention. Figure 1 As shown, this embodiment provides a target detection method, the processing method including:

[0035] S01. Use point cloud data including information in four dimensions as input data, wherein the four dimensions include three-dimensional coordinate information and reflection intensity;

[0036] S02. Perform multi-layer reflection intensity bird's-eye view encoding on the point cloud data, and use a reflection intensity bird's-eye view feature extraction network composed of up / downsampling module, convolution module and connection layer to extract reflection intensity bird's-eye view features from the encoding result;

[0037] S03. Perform three-dimensional voxel encoding on the point cloud data, and use a three-dimensional voxel feature extraction network consisting of an encoding layer, a convolution module, and a transformation module to extract three-dimensional voxel features from the encoding result.

[0038] S04. After fusing the bird's-eye view features of the reflection intensity and the three-dimensional voxel features, a first-stage detection head composed of a feature extraction sub-network, a multi-scale sub-network and a convolutional layer is used to detect the fusion result to obtain a three-dimensional region suggestion.

[0039] S05. A three-dimensional voxel region of interest pooling layer is used to pool the three-dimensional voxel features based on the three-dimensional region proposal to obtain the region of interest features;

[0040] S06. The second-stage detection head is used to perform feature transformation on the features of the region of interest, and then the bounding box regression and confidence prediction are performed respectively to obtain the detection results of the point cloud data.

[0041] Figure 2 A schematic diagram illustrating the structural framework of a target detection method according to an embodiment of the present invention is shown. Figure 2 As shown, this method mainly includes a point cloud encoding method based on a bird's-eye view of reflection intensity and a two-stage point cloud target detection network.

[0042] The point cloud encoding method of reflection intensity bird's-eye view-voxel includes encoding the input point cloud into voxel features and reflection intensity bird's-eye view features. The voxel method can ensure the efficiency of the network, while the reflection intensity bird's-eye view can make fuller use of the information carried in the point cloud without introducing a large computational overhead.

[0043] The two-stage point cloud target detection network includes: a reflection intensity bird's-eye view feature extraction network, a 3D voxel feature extraction network, a first-stage detection head, a 3D voxel region of interest pooling layer, and a second-stage detection head.

[0044] By training the above network, it can accurately detect targets in road traffic and rail traffic, while also having real-time inference speed.

[0045] In some embodiments of the present invention, the point cloud data is encoded using a multi-layer reflection intensity bird's-eye view, and a reflection intensity bird's-eye view feature extraction network composed of an up / downsampling module, a convolutional module, and a connection layer is used to extract reflection intensity bird's-eye view features from the encoding result. This includes: slicing the reflection intensity in the point cloud data according to the height direction in a bird's-eye view manner; for example, assuming the scale of the original point cloud space is L×W×H, the entire space is divided into N segments of size L×W×H according to the height. The space. Within each small space after slicing, the reflection intensity of the point cloud is projected into a two-dimensional pseudo-image with a preset resolution according to coordinates. That is, the reflection intensity of the point cloud within the small space is projected into a two-dimensional pseudo-image with D. L ×D W The resolution is projected onto the coordinates to form a two-dimensional pseudo-image, the image size of which is... This allows us to obtain the dimension as... The reflection intensity feature map is used as the encoding result; the encoding result is passed through multiple downsampling modules, convolutional modules and connection layers, multiple upsampling modules and convolutional modules to obtain the reflection intensity bird's-eye view feature map; Figure 3 A schematic diagram illustrating the structure of a bird's-eye view feature extraction network for reflection intensity according to an embodiment of the present invention is shown. Figure 3 As shown, the first downsampling module in the plurality of downsampling modules consists of two convolutional layers, and subsequent downsampling modules adopt a RestNet residual structure; the upsampling module adopts a deconvolutional layer; the connection layer is a lateral connection layer, which is set between the upsampling module and the corresponding downsampling module. The convolutional operation parameters of the downsampling modules (Block 1 to Block 5) in the reflection intensity bird's-eye view feature extraction network are shown in Tables 1, 2, 3 and 4.

[0046] Table 1. Block 1 & Block 2 of the feature extraction network for the bird's-eye view of reflection intensity.

[0047]

[0048] Table 2. Block 3 of the feature extraction network for bird's-eye view of reflection intensity.

[0049]

[0050] Table 3. Block 4 of the feature extraction network for bird's-eye view of reflection intensity.

[0051]

[0052] Table 4. Block 5 of the feature extraction network for bird's-eye view of reflection intensity.

[0053]

[0054]

[0055] Specifically, the input size of the bird's-eye view feature of the reflection intensity is 4×800×704. After being processed by the first-layer downsampling module Block 1, the output size is 32×800×704. After being processed by the second-layer downsampling module Block 2, the output size is 96×400×352. After being processed by the third-layer downsampling module Block 3, the output size is 192×200×176. After being processed by the fourth-layer downsampling module Block 4, the output size is 256×100×88. After being processed by the fifth-layer downsampling module Block 5, the output size is 384×50×44. After being processed by a 1×1 convolutional block, the output size is 196×50×44. After being processed by the first-layer upsampling module (3×3 deconvolutional block), the output size is 128×100×88. The output features of the fourth-layer downsampling module are processed by a 1×1 convolutional block and then added to the output features of the first-layer upsampling module, resulting in a feature size of 128×100×88. After processing by the second-layer upsampling module (a 3×3 deconvolutional block), the output features are 96×200×176. The output features of the third-layer downsampling module are processed by a 1×1 convolutional block and then added to the output features of the second-layer upsampling module, resulting in a feature size of 96×200×176. Finally, after processing by a 1×1 convolutional block, the output features are 256×200×176.

[0056] In some embodiments of the present invention, the point cloud data is subjected to three-dimensional voxel encoding, and a three-dimensional voxel feature extraction network consisting of an encoding layer, a convolutional module, and a transformation module is used to extract three-dimensional voxel features from the encoding result. This includes: uniformly subdividing the three-dimensional space containing the point cloud data into equally spaced voxels, encoding the four-dimensional information of the point cloud within each voxel, and obtaining the encoding result; specifically, uniformly subdividing the three-dimensional space into equally spaced voxels, assuming the size of each voxel is v. D ×v W ×v H Then the pointwise index tensor can be calculated. The original point cloud space is then quantized into a 3D mesh with a resolution of D×W×H. The point cloud is grouped according to the voxel containing each point, which can also be understood as grouping the point cloud according to the index tensor of each point. Due to factors such as occlusion, distance, and the relative pose of objects, the point cloud exhibits a non-uniform and sparse distribution throughout the space. After grouping, the number of points contained in each voxel is inconsistent, and some voxels may even contain no points. Therefore, to further optimize the data, it is necessary to record the index of each non-empty voxel and maintain the same number of points (5) within each non-empty voxel (if the number of points is less than 5, repeated sampling is performed; if the number of points is greater than 5, 5 points are randomly retained).

[0057] The encoding result is sequentially passed through an encoding layer, multiple convolutional modules, and a transformation module to obtain the three-dimensional voxel features; Figure 4 A schematic diagram illustrating the structure of a three-dimensional voxel feature extraction network according to an embodiment of the present invention is shown. Figure 4 As shown, the encoding layer is a voxel feature encoding layer; the convolutional module is a three-dimensional sparse convolutional module. The first convolutional module consists of two layers of sub-manifold sparse convolutions, and subsequent convolutional modules consist of one layer of spatial sparse convolution with a stride of 2 and two layers of sub-manifold sparse convolutions; the transformation module is a sparse feature to dense BEV feature layer. This network consists of a voxel feature encoding layer (VFE), four three-dimensional sparse convolutional modules (Spconv Block1 to Spconv Block4), and a sparse feature to dense BEV feature layer (To BEV). The operation of the voxel feature encoding layer (VFE) is to directly average the intrinsic properties of all internal points in each voxel. By averaging the features of the points within the voxel, the dimensionality of the features can be significantly reduced, thereby reducing computational and storage overhead; this operation can also suppress the influence of noise points on the feature representation to a certain extent, thereby improving the stability of the model. The first sparse convolutional module (Spconv Block1) consists of two layers of sub-manifold sparse convolutions. The second to fourth sparse convolutional modules (Spconv Block2 to Spconv Block4) consist of one layer of spatial sparse convolution with a stride of 2 and two layers of submanifold sparse convolution. Sparse convolution can effectively process non-empty voxel features by querying a hash table, thus significantly improving computational efficiency. Using submanifold sparse convolution and spatial sparse convolution together can increase the receptive field while preserving sparsity as much as possible. After passing through the four 3D sparse convolutional modules, multi-stage feature maps with downsampling resolutions of 1×, 2×, 4×, and 8× are obtained, respectively. The operation of converting sparse features into dense BEV feature layers (To BEV) involves compressing the extracted sparse features in the Z-axis direction to form a 2D BEV feature map. The convolution operation parameters of the four 3D sparse convolutional modules in the 3D voxel feature extraction network are shown in Tables 5 and 6.

[0058] Table 5. SpConv Block1 & Block2 of the 3D voxel feature extraction network

[0059]

[0060] Table 6. SpConv Block3 & Block4 of the 3D voxel feature extraction network

[0061]

[0062] Because the density of the original point cloud input varies each time, the point cloud features are different each time. Assuming there are N non-empty voxels, the input point cloud features can be represented as [N, 5, 4] (5 represents that each non-empty voxel contains 5 points, and 4 represents that each point carries a four-dimensional feature of [x, y, z, r]). The dimension of the output features after the voxel feature encoding layer (VFE) is represented as [N, 4]. The dimension of the output features after the first 3D sparse convolutional module (Spconv Block1) is represented as [N, 16]. The dimension of the output features after the second 3D sparse convolutional module (Spconv Block2) is represented as [N / 2, 32]. The dimension of the output features after the third 3D sparse convolutional module (Spconv Block3) is represented as [N / 4, 64]. The dimension of the output features after the fourth 3D sparse convolutional module (Spconv Block4) is represented as [N / 8, 64]. The features obtained above are all sparse features. After passing through the To BEV module (sparse features are converted into dense BEV feature layers), the extracted point cloud sparse features are converted into two-dimensional features, and the size of the obtained two-dimensional features is 256×200×176.

[0063] In some embodiments of the present invention, after fusing the bird's-eye view features of reflection intensity and the three-dimensional voxel features, a first-stage detection head composed of a feature extraction subnetwork, a multi-scale subnetwork, and a convolutional layer is used to detect the fusion result to obtain a three-dimensional region proposal, including:

[0064] The reflection intensity bird's-eye view features and the 3D voxel features are fused to obtain the fusion result; preferably, the concatenate operation is used for feature fusion. The fusion result is then passed through a network consisting of a top-down feature extraction sub-network with two convolutional layers and a multi-scale sub-network for upsampling and feature fusion. The network output is then convolved with two parallel convolutional layers to generate a 3D region proposal. The parameters of the first-stage detection head are shown in Table 7. This network consists of a top-down feature extraction sub-network with two standard 3×3 convolutional layers and a multi-scale sub-network for upsampling and feature fusion. Finally, the network output is convolved with two parallel 1×1 convolutional layers to generate a 3D region proposal.

[0065] Table 7 First Stage Detection Head

[0066]

[0067]

[0068] In some embodiments of the present invention, a three-dimensional voxel region of interest pooling layer is used to pool the three-dimensional voxel features based on the proposed three-dimensional region to obtain region of interest features. This includes: extracting aggregated features using a voxel pooling method at different stages; and concatenating aggregated features of different stages and scales to obtain voxel region of interest features. The voxel pooling method includes the following steps: obtaining the index of adjacent voxels based on the three-dimensional coordinate information of the center voxel and a preset voxel position offset, and then grouping the voxels. The present invention uses a voxel pooling method, which, compared to point sampling and grid pooling methods, aggregates adjacent voxel features using neighbor perception, offering advantages such as ease of querying and lower time complexity. Specifically, the center voxel can be pooled by adding an offset (Δ). x ,Δ y ,Δ z To quickly find adjacent voxels, assume the index of the center voxel is (x C ,y C ,z C If ), then the index of a non-empty neighboring voxel can be represented as (x C +Δ x ,y C +Δ y ,z C +Δ z The distance between them can be calculated using the Manhattan distance: D = |Δ x |+|Δ y |+|Δ zSince voxels are arranged in a regular pattern in space, after setting a distance threshold, the indices of adjacent voxels can be calculated from the index of the center voxel, thus enabling rapid and effective grouping of voxels.

[0069] Based on the proposed 3D region, features of adjacent non-empty voxels are aggregated to the localized center point for feature extraction; max pooling is used to obtain aggregated features of K non-empty neighboring voxels. Specifically, given a center point voxel... K non-empty neighboring voxels are obtained based on the distance threshold. The aggregation feature can then be represented as:

[0070]

[0071] Where i represents the number of center point voxels contained in the space. Represents the relative coordinates of non-empty adjacent voxels and the center voxel. Represents non-empty adjacent voxels The voxel features are obtained, and then the features of neighboring voxels at different distances are learned using a fully connected layer (MLP). Finally, the MaxPooling operation is used to obtain the aggregated features of K non-empty neighboring voxels. Features are extracted using voxel pooling at different stages of the 3D backbone network. For each stage, a specific distance threshold is set to accommodate voxels of different scales. Then, the aggregated features from different stages and scales are concatenated to obtain voxel-based regions of interest (ROI) features.

[0072] In some embodiments of the present invention, a second-stage detection head is used to perform feature transformation on the features of the region of interest, and then performs bounding box regression and confidence prediction to obtain the detection results of the point cloud data. The parameters of the second-stage detection head are shown in Table 8.

[0073] Table 8 Second Stage Detection Head

[0074]

[0075] The network uses the fused region of interest features as input to refine the bounding boxes. This includes: using a shared multilayer perceptron to convert the region of interest features into feature vectors, and inputting the feature vectors into a bounding box regression branch and a confidence prediction branch to obtain the detection results of the point cloud data; the bounding box regression branch is used to predict the residual from the 3D region proposal to the ground truth, and the confidence prediction branch is used to predict the confidence score related to IoU.

[0076] In some embodiments of the present invention, the parameter weights in the reflection intensity bird's-eye view feature extraction network, the three-dimensional voxel feature extraction network, the first-stage detection head, and the second-stage detection head are obtained through the following training steps:

[0077] A loss function is constructed based on location loss and classification loss; specifically, the error between the predicted box and the ground truth box is represented as (Δx, Δy, Δz, Δw, Δl, Δh), and the formula for calculating each degree of freedom is as follows:

[0078]

[0079]

[0080] Δθ=sin(θ gt -θ reg ),

[0081] In the formula, The subscript gt represents the ground truth bounding box parameters, and reg represents the predicted bounding box parameters. Δx, Δy, and Δz represent the error values ​​between the center points of the predicted and ground truth bounding boxes in three dimensions, respectively; Δw, Δl, and Δh represent the error values ​​between the predicted and ground truth bounding boxes in the width, length, and height directions, respectively; and Δθ represents the error between the predicted rotation angle and the true value. Therefore, the regression loss can be expressed as:

[0082]

[0083] In the formula, SmoothL1() represents the SmoothL1 loss function. Therefore, the target position loss can be expressed as:

[0084]

[0085] In the formula, N represents the total number of predicted anchor frames; I and u are the input discriminant parameters; 1 (I≥u) means that regression loss will be calculated only if I≥u.

[0086] The classification loss can be expressed as:

[0087]

[0088] In the formula, p i FL() represents the probability predicted by the model; FL() represents the Focal loss function.

[0089] The loss function for the first-stage detection head is designed as a combination of classification loss and location loss, as follows:

[0090]

[0091] In the formula, N fg Indicates the number of foreground anchors; pI δ represents the probability of the output of the first-stage classification branch. I This represents the output of the first-stage box regression branch; g i Indicates category label; g i And 1 represents 1 (g) i ≥1), meaning that only the foreground anchor is used to calculate the regression loss.

[0092] The loss function of the second-stage detection head can be expressed as:

[0093]

[0094] In the formula, N rp p represents the number of region suggestions sampled during the training phase. II and δ II These represent the outputs of the second-stage classification and bounding box regression branches, respectively; IoU i β represents the IoU between the i-th proposal and its corresponding truth box; reg Indicates the IoU threshold; IoU i and β reg Represents 1 (IoU) i ≥θ reg (i.e., only IoU) i ≥β reg Only regions that provide specific regional recommendations will be used to calculate regression loss.

[0095] Obtain training samples and determine training control parameters; the training control parameters include: the number of training iterations is set to 80, the initial learning rate is 0.01, the weight-decay is set to 0.01, and the moment parameter is set to 0.9.

[0096] The ADAM optimizer is used for end-to-end training based on the aforementioned loss function, training samples, and training control parameters. During training, the network is trained from scratch using the ADAM optimizer in an end-to-end manner. Based on the aforementioned training control parameters, for KITTI, the original point cloud is first cropped along the X, Y, and Z axes to a space of [0, 70.4m] × [-40, 40m] × [-3, 1m], with a voxel size of (0.05, 0.05, 0.1)m. For the Waymo Open Dataset, the corresponding space range is [-75.2, 75.2m] × [-75.2, 75.2m] × [-2, 4m], with a voxel size of (0.1, 0.1, 0.15)m. The experimental environment is a server containing a single GTX 2080Ti GPU (CUDA 11.4, CUDNN 8.0.5). All experiments were conducted using Pytoch 1.8.1 and OpenPCDet 0.6.0. During inference, a non-maximum suppression (NMS) threshold of 0.01 was used to remove redundant predicted boxes. The experimental environment was the same as that used during training.

[0097] Based on the same inventive concept, this application also provides a target detection device. Figure 5 A schematic diagram of the target detection device according to an embodiment of the present invention is shown. Figure 5 As shown, the device includes:

[0098] The raw data module is used to take point cloud data including four-dimensional information as input data, the four dimensions including three-dimensional coordinate information and reflection intensity;

[0099] The first feature module is used to encode the point cloud data into a multi-layer reflection intensity bird's-eye view, and to extract the reflection intensity bird's-eye view features from the encoding result using a reflection intensity bird's-eye view feature extraction network composed of an up / downsampling module, a convolution module and a connection layer.

[0100] The second feature module is used to encode the point cloud data in three dimensions using voxels, and to extract three dimensions voxel features from the encoding result using a three dimensions voxel feature extraction network consisting of an encoding layer, a convolutional module, and a transformation module.

[0101] The fusion detection module is used to fuse the bird's-eye view features of reflection intensity and the three-dimensional voxel features, and then use a first-stage detection head composed of a feature extraction sub-network, a multi-scale sub-network and a convolutional layer to detect the fusion result and obtain a three-dimensional region suggestion.

[0102] The pooling module is used to pool the 3D voxel features based on the 3D region proposal using a 3D voxel region-of-interest pooling layer to obtain region-of-interest features; and

[0103] The result output module is used to perform feature transformation on the features of the region of interest using the second-stage detection head, and then perform bounding box regression and confidence prediction to obtain the detection results of the point cloud data.

[0104] The specific limitations of each functional module in the aforementioned target detection device can be found in the limitations of the target detection method described above, and will not be repeated here. Each module in the aforementioned device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module. It also overcomes the shortcomings of existing target detection methods, achieving high discrimination, high processing efficiency, and good adaptability.

[0105] In one embodiment of the present invention, a computer-readable storage medium is also provided, wherein the storage medium stores instructions that, when executed on a computer, cause the processor to be configured to perform the target detection method described above.

[0106] In one embodiment of the present invention, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described target detection method.

[0107] Through the above implementation methods and the cooperation of each module, the trained model can not only effectively detect targets in the original input point cloud, but also avoid the problem of long processing time.

[0108] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0109] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0110] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0111] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0112] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0113] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0114] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0115] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0116] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A target detection method, characterized in that, The method includes: Point cloud data including information in four dimensions is used as input data, the four dimensions including three-dimensional coordinate information and reflection intensity; The point cloud data undergoes multi-layer reflection intensity bird's-eye view encoding, and a reflection intensity bird's-eye view feature extraction network composed of upsampling / downsampling modules, convolutional modules, and connection layers is used to extract reflection intensity bird's-eye view features from the encoding result. This includes: slicing the reflection intensity in the point cloud data along the height direction in a bird's-eye view manner; projecting the reflection intensity of the point cloud into a two-dimensional pseudo-image at a preset resolution according to coordinates within each sliced ​​space, using this as the encoding result; the encoding result is then processed through multiple downsampling modules, convolutional modules and connection layers, multiple upsampling modules, and convolutional modules to obtain the reflection intensity bird's-eye view features; wherein, the first downsampling module in the multiple downsampling modules consists of two convolutional layers, and subsequent downsampling modules use a RestNet residual structure; the upsampling module uses deconvolutional layers; and the connection layer is a horizontal connection layer, placed between the upsampling module and the corresponding downsampling module. The point cloud data is 3D voxel encoded, and a 3D voxel feature extraction network consisting of an encoding layer, a convolutional module, and a transformation module is used to extract 3D voxel features from the encoding results. After fusing the bird's-eye view features of reflection intensity and the three-dimensional voxel features, a first-stage detection head composed of a feature extraction sub-network, a multi-scale sub-network and a convolutional layer is used to detect the fusion result to obtain a three-dimensional region suggestion. A three-dimensional voxel region of interest pooling layer is used to pool the three-dimensional voxel features based on the proposed three-dimensional region to obtain the region of interest features; The second-stage detection head is used to perform feature transformation on the features of the region of interest, and then performs bounding box regression and confidence prediction to obtain the detection results of the point cloud data.

2. The target detection method according to claim 1, characterized in that, The point cloud data is 3D voxel encoded, and a 3D voxel feature extraction network consisting of an encoding layer, a convolutional module, and a transformation module is used to extract 3D voxel features from the encoding result, including: The three-dimensional space containing the point cloud data is uniformly subdivided into equally spaced voxels, and the four-dimensional information of the point cloud is encoded in each voxel to obtain the encoding result. The encoding result is sequentially passed through an encoding layer, multiple convolutional modules, and a transformation module to obtain the three-dimensional voxel features; The encoding layer is a voxel feature encoding layer; the convolution module is a three-dimensional sparse convolution module, the first convolution module in the plurality of convolution modules consists of two layers of sub-manifold sparse convolution, and the subsequent convolution modules consist of one layer of spatial sparse convolution with a stride of 2 and two layers of sub-manifold sparse convolution; the conversion module is a sparse feature conversion layer to a dense BEV feature layer.

3. The target detection method according to claim 1, characterized in that, After fusing the bird's-eye view features of reflection intensity and the three-dimensional voxel features, a first-stage detection head composed of a feature extraction subnetwork, a multi-scale subnetwork, and convolutional layers is used to detect the fused result, obtaining three-dimensional region suggestions, including: After fusing the bird's-eye view features of the reflection intensity and the three-dimensional voxel features, the fusion result is obtained; The fusion result is passed through a network consisting of a top-down feature extraction subnetwork with two convolutional layers and a multi-scale subnetwork for upsampling and feature fusion. The output of the network is then convolved with two parallel convolutional layers to generate a 3D region proposal.

4. The target detection method according to claim 1, characterized in that, A three-dimensional voxel region-of-interest (ROI) pooling layer is used to pool the three-dimensional voxel features based on the proposed three-dimensional region, resulting in ROI features, including: Voxel pooling method is used to extract aggregation features at different stages; By splicing together the aggregated features of different stages and scales, the voxel region of interest features are obtained. The voxel pooling method includes the following steps: Based on the three-dimensional coordinate information of the center voxel and the preset voxel position offset, the index of the adjacent voxels is obtained, and then the voxels are grouped. Based on the proposed three-dimensional region, the features of adjacent non-empty voxels are aggregated to the localized center point for feature extraction. A fully connected layer is used to learn the features of neighboring voxels at different distances, and then max pooling is used to obtain the aggregated features of K non-empty neighboring voxels.

5. The target detection method according to claim 1, characterized in that, The second-stage detection head performs feature transformation on the features of the region of interest, and then performs bounding box regression and confidence prediction to obtain the detection results of the point cloud data, including: A shared multilayer perceptron is used to convert the features of the region of interest into feature vectors. The feature vectors are then input into the bounding box regression branch and the confidence prediction branch to obtain the detection results of the point cloud data. The bounding box regression branch is used to predict the residual from the 3D region proposal to the ground truth, and the confidence prediction branch is used to predict the confidence score associated with IoU.

6. The target detection method according to claim 1, characterized in that, The parameter weights in the bird's-eye view feature extraction network for reflection intensity, the three-dimensional voxel feature extraction network, the first-stage detection head, and the second-stage detection head are obtained through the following training steps: A loss function is constructed based on location loss and classification loss; Obtain training samples and determine training control parameters; The ADAM optimizer is used to train the system in an end-to-end manner based on the loss function, training samples, and training control parameters.

7. A target detection device, characterized in that, The device includes: The raw data module is used to take point cloud data including four-dimensional information as input data, the four dimensions including three-dimensional coordinate information and reflection intensity; The first feature module is used to perform multi-layer reflection intensity bird's-eye view encoding on the point cloud data, and to extract reflection intensity bird's-eye view features from the encoding result using a reflection intensity bird's-eye view feature extraction network composed of upsampling / downsampling modules, convolutional modules, and connection layers. This includes: slicing the reflection intensity in the point cloud data along the height direction in a bird's-eye view manner; projecting the reflection intensity of the point cloud into a two-dimensional pseudo-image at a preset resolution according to coordinates within each sliced ​​small space, as the encoding result; the encoding result is then processed through multiple downsampling modules, convolutional modules and connection layers, multiple upsampling modules, and convolutional modules to obtain the reflection intensity bird's-eye view features; wherein, the first downsampling module in the multiple downsampling modules consists of two convolutional layers, and subsequent downsampling modules use a RestNet residual structure; the upsampling module uses deconvolutional layers; and the connection layer is a horizontal connection layer, positioned between the upsampling module and the corresponding downsampling module. The second feature module is used to encode the point cloud data in three dimensions using voxels, and to extract three dimensions voxel features from the encoding result using a three dimensions voxel feature extraction network consisting of an encoding layer, a convolutional module, and a transformation module. The fusion detection module is used to fuse the bird's-eye view features of reflection intensity and the three-dimensional voxel features, and then use a first-stage detection head composed of a feature extraction sub-network, a multi-scale sub-network and a convolutional layer to detect the fusion result and obtain a three-dimensional region suggestion. The pooling module is used to pool the 3D voxel features based on the 3D region proposal using a 3D voxel region-of-interest pooling layer to obtain region-of-interest features; and The result output module is used to perform feature transformation on the features of the region of interest using the second-stage detection head, and then perform bounding box regression and confidence prediction to obtain the detection results of the point cloud data.

8. An electronic device, characterized in that, include: At least one processor; A memory connected to the at least one processor; The memory stores instructions executable by the at least one processor, which executes the steps of the target detection method according to any one of claims 1 to 6 by executing the instructions stored in the memory.

9. A computer-readable storage medium, characterized in that, The storage medium stores instructions that, when run on a computer, cause the computer to perform the target detection method according to any one of claims 1 to 6.

Images