Point cloud target detection method, device and equipment based on multi-scale feature fusion

Abstract

The application relates to a point cloud target detection method, device and equipment based on multi-scale feature fusion, comprising the following steps: inputting a pseudo image corresponding to original point cloud data into a backbone network, and performing deconvolution on a feature map output by a last selected feature extraction layer to obtain a plurality of groups of deconvolution feature maps; performing cascade fusion on the feature map output by the selected feature extraction layer and the corresponding deconvolution feature map to obtain a plurality of groups of initial fusion feature maps and performing division to obtain a first feature map set not including an initial fusion feature map corresponding to a first selected feature extraction layer and a second feature map set not including an initial fusion feature map corresponding to a last selected feature extraction layer; performing cascade fusion on the initial fusion feature maps in the first and second feature map sets respectively to obtain first and second fusion feature maps; and performing cascade fusion on the first and second fusion feature maps to obtain a target feature map for detection. The method can improve the accuracy of point cloud target detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of point cloud target detection, and in particular to a point cloud target detection method, device and equipment based on multi-scale feature fusion. BACKGROUND

[0002] With the continuous development of laser radar scanning technology and artificial intelligence technology, point cloud target detection has become a research hotspot in the field of robots and autonomous driving in recent years, and is one of the key technologies for obstacle detection in the environment of robots and autonomous driving. Compared with image data, point cloud data is less affected by light and weather conditions, and contains accurate depth information in three-dimensional space, which can better represent the three-dimensional spatial geometric information of the target. Therefore, it is of great significance to study target detection based on point cloud data.

[0003] Point cloud target detection is one of the most basic tasks in point cloud data processing. Current point cloud target detection methods can be divided into four categories: point cloud target detection methods based on original points, point cloud target detection methods based on voxels, point cloud target detection methods based on data dimension reduction, and point cloud target detection methods based on a mixture of point clouds and voxels. The point cloud target detection method based on original points processes and analyzes the original point cloud data to determine the target category and regress the target bounding box. The main advantage of this method is that it fully utilizes all the information of the point cloud data, so that the extracted point features can effectively represent the three-dimensional spatial structure of the target, and the detection effect is good, but the memory occupation is high and the calculation amount is large. The point cloud target detection method based on voxels divides the irregular point cloud into regular voxels of the same size in three-dimensional space, processes the voxels, and extracts voxel features for target detection. The three-dimensional spatial feature representation capability of this method is limited, and the calculation cost is small compared with the method based on original points, but the use of three-dimensional convolution and sparse convolution still requires a large amount of computing power. The point cloud target detection method based on data dimension reduction converts the point cloud into a two-dimensional image, and then uses existing mature image target detection algorithms for detection. This method has small calculation cost, fast reasoning speed and easy deployment compared with the above three methods, but the detection effect is relatively poor due to information loss in the data dimension reduction process. The point cloud target detection method based on a mixture of original point cloud and voxel uses point cloud and voxel in a network for target detection, retains point features and voxel features, and combines the advantages of the original point cloud and voxel-based methods. This method reduces the calculation cost on the premise of fully preserving the three-dimensional spatial structure, but its calculation cost is still higher than that of the data dimension reduction-based method.

[0004] In recent years, with the development of high-performance computing devices and deep learning technology, point cloud target detection technology based on deep learning has been widely studied. PointPillar is a classic point cloud target detection algorithm based on data dimension reduction. Due to its good real-time performance and deployability, it is widely studied and applied in academia and industry. However, PointPillar has three limitations: first, the detection performance is greatly affected by the size of the column. The larger the column size, the smaller the resolution of the generated pseudo image, the higher the network running speed, but the detection effect is poor. The smaller the column size, the higher the resolution of the generated pseudo image, the lower the network running speed, but the detection effect is good. Second, the pseudo image is generated by a feature encoding network, and the quality of the generated pseudo image directly affects the detection result. Third, the three feature maps output by the backbone network are all high-level features, and the high-level features are directly fed into the detection head, containing a lot of redundant information and noise. Therefore, PointPillar has poor detection effect on small targets such as pedestrians and cyclists. SUMMARY

[0005] Therefore, it is necessary to provide a point cloud target detection method, device and equipment based on multi-scale feature fusion to improve the accuracy of point cloud target detection in order to solve the above technical problems.

[0006] A point cloud target detection method based on multi-scale feature fusion, comprising:

[0007] generating a pseudo image from the original point cloud data;

[0008] After inputting the pseudo image into the backbone network, performing multiple deconvolutions on the feature map output by the last selected feature extraction layer in the backbone network to obtain a plurality of groups of deconvolution feature maps corresponding thereto; wherein the number of deconvolutions is the same as the number of layers of the selected feature extraction layer;

[0009] Cascade fusion of the feature map output by the selected feature extraction layer in the backbone network and the corresponding deconvolution feature map to obtain a plurality of groups of initial fusion feature maps;

[0010] dividing the plurality of groups of initial fusion feature maps to obtain a first feature map set including a plurality of groups of initial fusion feature maps other than the initial fusion feature map corresponding to the first selected feature extraction layer, and a second feature map set including a plurality of groups of initial fusion feature maps other than the initial fusion feature map corresponding to the last selected feature extraction layer;

[0011] Cascade fusion of the initial fusion feature maps in the first feature map set to obtain a first fusion feature map, and cascade fusion of the initial fusion feature maps in the second feature map set to obtain a second fusion feature map;

[0012] Cascade fusion of the first fusion feature map and the second fusion feature map to obtain a target feature map;

[0013] perform point cloud target detection according to the target feature map.

[0014] A point cloud target detection device based on multi-scale feature fusion comprises:

[0015] A pseudo-image generation module is configured to generate a pseudo-image according to original point cloud data.

[0016] A deconvolution module is configured to perform multiple deconvolutions on a feature map output by a last selected feature extraction layer in a backbone network after the pseudo-image is input into the backbone network, to obtain a plurality of groups of deconvolution feature maps corresponding thereto; wherein the number of deconvolutions is the same as the number of layers of the selected feature extraction layer.

[0017] A first fusion module is configured to cascade fuse the feature map output by the selected feature extraction layer in the backbone network and the corresponding deconvolution feature map, to obtain a plurality of groups of initial fusion feature maps.

[0018] A feature map division module is configured to divide the plurality of groups of initial fusion feature maps, to obtain a first feature map set comprising a plurality of groups of initial fusion feature maps other than the initial fusion feature map corresponding to the first selected feature extraction layer, and a second feature map set comprising a plurality of groups of initial fusion feature maps other than the initial fusion feature map corresponding to the last selected feature extraction layer.

[0019] A second fusion module is configured to cascade fuse the initial fusion feature maps in the first feature map set to obtain a first fusion feature map, and cascade fuse the initial fusion feature maps in the second feature map set to obtain a second fusion feature map.

[0020] A third fusion module is configured to cascade fuse the first fusion feature map and the second fusion feature map, to obtain a target feature map.

[0021] A target detection module is configured to perform point cloud target detection according to the target feature map.

[0022] A computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0023] generating a pseudo-image according to original point cloud data;

[0024] performing multiple deconvolutions on a feature map output by a last selected feature extraction layer in a backbone network after the pseudo-image is input into the backbone network, to obtain a plurality of groups of deconvolution feature maps corresponding thereto; wherein the number of deconvolutions is the same as the number of layers of the selected feature extraction layer.

[0025] cascade fusing the feature map output by the selected feature extraction layer in the backbone network and the corresponding deconvolution feature map, to obtain a plurality of groups of initial fusion feature maps.

[0026] dividing the multiple groups of initial fusion feature maps to obtain a first feature map set including other multiple groups of initial fusion feature maps except for the initial fusion feature map corresponding to the first selected feature extraction layer, and a second feature map set including other multiple groups of initial fusion feature maps except for the initial fusion feature map corresponding to the last selected feature extraction layer;

[0027] concatenating the initial fusion feature maps in the first feature map set to obtain a first fusion feature map, and concatenating the initial fusion feature maps in the second feature map set to obtain a second fusion feature map;

[0028] concatenating the first fusion feature map and the second fusion feature map to obtain a target feature map;

[0029] performing point cloud target detection according to the target feature map.

[0030] The point cloud target detection method, device and equipment based on multi-scale feature fusion have the following advantages. Firstly, the pseudo image corresponding to the original point cloud data is input into the backbone network, and the feature map output by the last selected feature extraction layer is deconvolved multiple times to obtain multiple groups of deconvolution feature maps, where the number of deconvolutions is the same as the number of layers of the selected feature extraction layer. Then, the feature maps output by each selected feature extraction layer are respectively concatenated with the corresponding deconvolution feature maps to obtain multiple groups of initial fusion feature maps. It can be known that, according to the above-mentioned concatenation, the target information in the feature map is enhanced, so that the target features in the obtained initial fusion feature maps are more obvious. Then, the multiple groups of initial fusion feature maps are divided to obtain a first feature map set including other multiple groups of initial fusion feature maps except for the initial fusion feature map corresponding to the first selected feature extraction layer, and a second feature map set including other multiple groups of initial fusion feature maps except for the initial fusion feature map corresponding to the last selected feature extraction layer. The initial fusion feature maps in the first feature map set are concatenated to obtain a first fusion feature map, the initial fusion feature maps in the second feature map set are concatenated to obtain a second fusion feature map, and the first fusion feature map and the second fusion feature map are concatenated to obtain a target feature map. In this way, the division and then the fusion make the first fusion feature map contain rich spatial information, the second fusion feature map contain rich semantic information, and the target feature map contain both rich spatial information and rich semantic information. Finally, point cloud target detection is performed according to the target feature map. In summary, the method can improve the accuracy of point cloud target detection. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 A flowchart of the point cloud target detection method based on multi-scale feature fusion in one embodiment;

[0032] Figure 2 The structure of a convolution attention module in an embodiment is shown in FIG. 1.

[0033] Figure 3 The structure of a convolution residual block in an embodiment is shown in FIG. 2.

[0034] Figure 4 The structure of a backbone network in an embodiment is shown in FIG. 3.

[0035] Figure 5 The network structure and working schematic diagram of a point cloud target detection method based on multi-scale feature fusion in an embodiment are shown in FIG. 4.

[0036] Figure 6 The structure block diagram of a point cloud target detection device based on multi-scale feature fusion in an embodiment is shown in FIG. 5.

[0037] Figure 7 The internal structure diagram of a computer device in an embodiment is shown in FIG. 6. DETAILED DESCRIPTION

[0038] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0039] In an embodiment, as shown in FIG. 1, a point cloud target detection method based on multi-scale feature fusion is provided, comprising the following steps: Figure 1

[0040] Step 102, generating a pseudo image according to original point cloud data.

[0041] The pseudo image is obtained by feature encoding after columnarization of the original point cloud data. The three-dimensional point cloud data is converted into a two-dimensional pseudo image through encoding, which can greatly reduce the amount of data to be processed subsequently.

[0042] Step 104, inputting the pseudo image into a backbone network, and performing multiple deconvolutions on the feature maps output by the last selected feature extraction layer in the backbone network to obtain corresponding multiple groups of deconvolution feature maps.

[0043] The number of deconvolutions is the same as the number of selected feature extraction layers.

[0044] Considering that the feature maps output by the last feature extraction layer contain the most abundant high-level semantic information, the same number of deconvolutions as the number of selected feature extraction layers is performed on the feature maps, and multiple groups of deconvolution feature maps are obtained.

[0045] ​Step 106, the feature maps output by the selected feature extraction layers in the backbone network are respectively concatenated and fused with the corresponding deconvolution feature maps to obtain a plurality of groups of initial fusion feature maps.

[0046] The determination of the foregoing selected feature extraction layers needs to satisfy a principle: the size of the feature maps output by the selected feature extraction layers is consistent with the size of the corresponding deconvolution feature maps, so as to perform subsequent concatenation and fusion.

[0047] The corresponding relationship between the feature maps and the deconvolution feature maps is that the order of the selected feature extraction layer numbers corresponding to the feature maps is opposite to the order of the deconvolution corresponding to the deconvolution feature maps, that is, the foregoing principle is satisfied. For example, assuming that there are four selected feature extraction layers, which correspond to R out1 , R out2 , R out3 and R out4 respectively, four deconvolutions are performed to obtain T1, T2, T3 and T4 in turn, and then R out1 and T4 are concatenated and fused to obtain S1, R out2 and T3 are concatenated and fused to obtain S2, R out3 and T2 are concatenated and fused to obtain S3, and R out4 and T1 are concatenated and fused to obtain S4, thereby obtaining the corresponding four groups of initial fusion feature maps.

[0048] According to the foregoing corresponding relationship, the deconvolution feature maps and the corresponding feature maps are concatenated and fused, which is equivalent to enhancing the target information in the feature maps, so that the target feature is more obvious.

[0049] Step 108, the plurality of groups of initial fusion feature maps are divided to obtain a first feature map set including other groups of initial fusion feature maps except for the initial fusion feature map corresponding to the first selected feature extraction layer, and a second feature map set including other groups of initial fusion feature maps except for the initial fusion feature map corresponding to the last selected feature extraction layer.

[0050] Step 110, the initial fusion feature maps in the first feature map set are concatenated and fused to obtain a first fusion feature map, and the initial fusion feature maps in the second feature map set are concatenated and fused to obtain a second fusion feature map.

[0051] Taking the case of four selected feature extraction layers in step 106 as an example, the beneficial effect of the feature map division manner in step 108 on cascade fusion is explained: S2 and S3 are public parts in the four groups of initial fusion feature maps, S1 is a low-level feature containing rich spatial information, and S4 is a high-level feature containing rich semantic information; low-level features are beneficial to small target detection, and high-level features are beneficial to large target detection. The feature map obtained by fusing S1, S2 and S3 contains more spatial information and is more beneficial to small target detection; the feature map obtained by fusing S2, S3 and S4 contains more semantic information and is more beneficial to large target detection.

[0052] Step 112, cascade fusion of the first fusion feature map and the second fusion feature map to obtain a target feature map.

[0053] Cascade fusion of the first fusion feature map containing rich spatial information and the second fusion feature map containing rich semantic information makes the target feature map contain both rich spatial information and rich semantic information.

[0054] Step 114, point cloud target detection according to the target feature map.

[0055] In the multi-scale feature fusion point cloud target detection method, first, the pseudo image corresponding to the original point cloud data is input into the backbone network, and the feature map output by the last selected feature extraction layer is deconvolved multiple times to obtain a plurality of groups of deconvolution feature maps, wherein the number of deconvolutions is the same as the number of layers of the selected feature extraction layer; then, the feature maps output by each selected feature extraction layer are respectively cascade fused with the corresponding deconvolution feature maps to obtain a plurality of groups of initial fusion feature maps. It can be known that, according to the cascade fusion according to the aforementioned correspondence relationship, the target information in the feature map is enhanced, so that the target features in the obtained initial fusion feature maps are more obvious; then, the plurality of groups of initial fusion feature maps are divided to obtain a first feature map set including other groups of initial fusion feature maps except the initial fusion feature map corresponding to the first selected feature extraction layer, and a second feature map set including other groups of initial fusion feature maps except the initial fusion feature map corresponding to the last selected feature extraction layer; the initial fusion feature maps in the first feature map set are cascade fused to obtain a first fusion feature map, the initial fusion feature maps in the second feature map set are cascade fused to obtain a second fusion feature map, and the first fusion feature map and the second fusion feature map are cascade fused to obtain a target feature map. After such division and fusion, the first fusion feature map contains rich spatial information, the second fusion feature map contains rich semantic information, and the target feature map contains both rich spatial information and rich semantic information; finally, point cloud target detection is performed according to the target feature map. In summary, the method can improve the accuracy of point cloud target detection.

[0056] In one embodiment, the pseudo image is generated according to the original point cloud data, comprising:

[0057] Obtaining original point cloud data, setting a point cloud data detection range, columnizing the original point cloud data in the point cloud data detection range on an X-Y plane to obtain columnized point cloud data, inputting the columnized point cloud data into a pre-constructed feature encoding network to obtain a pseudo image.

[0058] In one embodiment, the feature maps output by the selected feature extraction layers in the backbone network are respectively concatenated and fused with the corresponding deconvolution feature maps to obtain a plurality of groups of initial fusion feature maps, comprising:

[0059] The feature maps output by the selected feature extraction layers in the backbone network are respectively input into the corresponding convolution attention modules to obtain a plurality of groups of attention feature maps, and each group of attention feature maps is concatenated and fused with the corresponding deconvolution feature map to obtain a plurality of groups of initial fusion feature maps.

[0060] In one embodiment, the feature maps output by the selected feature extraction layers in the backbone network are respectively input into the corresponding convolution attention modules to obtain a plurality of groups of attention feature maps, comprising:

[0061] The feature maps F output by the selected feature extraction layers in the backbone network are respectively input into the corresponding convolution attention modules to obtain a plurality of groups of attention feature maps, comprising: C×H×W Channel splitting is performed to obtain a corresponding first channel feature map F1 and a second channel feature map F2. (C / 2)×H×W (C / 2)×H×W ;

[0062] The first channel feature map F1 is fed into an average pooling layer to obtain an average pooling feature map F1'. (C / 2)×H×W (C / 2)×1×1 The average pooling feature map F1' is input into a 1x1 convolution layer and then into a sigmoid function to output a first weight value β of the average pooling feature map, and the first channel feature map F1 is multiplied by the first weight value β to obtain a first channel initial attention map F1". (C / 2)×1×1 (C / 2)×H×W (C / 2)×H×W The first channel initial attention map F1" is added to the first channel feature map F1 to obtain a first channel attention map F1. out :

[0063] F1 out = S(Conv(Avgpool(F1))) x F1 + F1

[0064] The second channel feature map F2 is fed into a maximum pooling layer to obtain a maximum pooling feature map F2'. (C / 2)×H×W (C ​​​​​ / 2)×1×1 ; the maximum pooling feature map F2' e R (C / 2)×1×1 After inputting the sigmoid function after 1x1 convolution, the second weight value a of the maximum pooling feature map is output, and the second channel feature map F2 e R (C / 2)×H×W is multiplied by the second weight value a to obtain the second channel initial attention map F2'', and the second channel initial attention map F2'' is added to the second channel feature map F2 e R (C / 2)×H×W to obtain the second channel attention map F2 out :

[0065] F2 out = S(Conv(Maxpool(F2))) x F2 + F2

[0066] After adding the first channel attention map F1 out and the second channel attention map F2 out , 1x1 convolution is performed to obtain a corresponding group of attention feature maps:

[0067]

[0068] Where S(·) represents the sigmoid function, Conv(·) represents 1x1 convolution, Avgpool(·) represents average pooling, and Maxpool(·) represents maximum pooling.

[0069] Convolutional neural networks generate a batch of multi-dimensional feature maps containing rich detailed information at each layer, but the information represented in each feature map is different, and the importance is also different. Some features are more important for target detection and recognition. Therefore, in order to make full use of important features in the network and suppress redundant features irrelevant to the task, the aforementioned convolutional attention module is designed, and its structure is shown in Figure 2 .

[0070] In one embodiment, the feature extraction network in the backbone network is a convolutional residual block; the steps of feature extraction through each convolutional residual block include:

[0071] Obtain the feature map input to the current convolutional residual block, and sequentially perform 1x1 convolution, 3x3 convolution with a step of t, and 1x1 convolution on the feature map to obtain a first branch feature map; pass the feature map through a 3x3 convolution to obtain a second branch feature map; after cascading and fusing the first branch feature map and the second branch feature map, perform 1x1 convolution to obtain a feature map output by the current convolutional residual block.

[0072] In this embodiment, the pseudo image is fed into the backbone network constructed by the convolutional residual block to extract low-level spatial information and high-level semantic information in the feature map. The structure of the convolutional residual block is as follows Figure 3As shown, it contains two 3×3 convolutions and three 1×1 convolutions, each followed by a Batch Normalization (BN) layer and a ReLU layer. For a residual block, assume the input feature map is F∈R. C ×H×W The output feature map is F out The first branch consists of one 3×3 convolution and two 1×1 convolutions. F is first reduced to dimensionality F1'∈R by a 1×1 convolution. (C / 2)×H×W F1' is processed by a 3×3 convolution with a stride of t to output the feature F1″∈R. (C / 2)×(H / t)×(W / t) F1″ is augmented with a 1×1 convolution to make the data dimension F1″′∈R. C×(H / t)×(W / t) The second branch consists of a single 3×3 convolution, and its output feature F2'∈R C×(H / t)×(W / t) Concatenate the outputs of the first branch F1″′ and the second branch F2' to obtain F3∈R 2C×(H / t)×(W / t) Then, F3 is processed through a 1×1 convolution to output F. out ∈R C×(H / t)×(W / t) Therefore, the convolutional residual block in this embodiment can be represented as:

[0073]

[0074] F1″=f1(f3(f1(F)));

[0075] F2' = f3(F);

[0076] Where f1(·) and f3(·) represent 1×1 convolution and 3×3 convolution, respectively.

[0077] In one embodiment, the backbone network comprises 17 convolutional residual blocks, with the structure as follows: Figure 4 As shown. Taking the KITTI dataset as an example, the number of feature extraction layers is selected as 4, and the feature map M∈R is input to the backbone network. 496×432×64 M is processed through two convolutional residual blocks with a stride of 1 to generate feature map R. out1 ∈R 496×432×64 ;R out1 The feature map R is generated after passing through one convolutional residual block with a stride of 2 and two convolutional residual blocks with a stride of 1. out2 ∈R 248×216×64 ;R out2 After passing through one convolutional residual block with a stride of 2 and five convolutional residual blocks with a stride of 1, a feature map R is generated. out3 ∈R 124×108×128 ;R out3 After passing through one convolutional residual block with a stride of 2 and five convolutional residual blocks with a stride of 1, a feature map R is generated. out4 ∈R 62×54×256 R out1R out2 R out3 and R out4 are four different size feature maps output by the backbone network.

[0078] In one embodiment, the first fusion feature map and the second fusion feature map are concatenated to obtain a target feature map, comprising:

[0079] The first fusion feature map is input into a first convolutional attention module to obtain a first fusion attention feature map, and the second fusion feature map is input into a second convolutional attention module to obtain a second fusion attention feature map; the first fusion attention feature map and the second fusion attention feature map are up-sampled and then concatenated to obtain an initial target feature map; the initial target feature map is input into a third convolutional attention module to obtain the target feature map.

[0080] In one embodiment, before point cloud target detection is performed according to the target feature map, comprising:

[0081] obtaining a pre-constructed target detection model;

[0082] inputting a training point cloud dataset into the target detection model, calculating and optimizing a loss function to obtain a trained target detection model; the model loss function includes a bounding box regression loss function and a classification loss function.

[0083] The bounding box regression loss function is:

[0084]

[0085] L θ = SmoothL1(sin(Δθ-θ'));

[0086]

[0087]

[0088] Δθ=θ gt -θ;

[0089]

[0090] wherein, L loc is the bounding box regression loss function, SmoothL1 is the SmoothL1 loss function, (x, y, z, w, l, h, θ) is the three-dimensional prior box of the point cloud target, x, y, z are the three-dimensional prior center coordinates of the point cloud target, w, l, h are the width, length and height of the three-dimensional prior box respectively, θ is the prior rotation angle of the point cloud target, (x gt , y gt , z gt , wgt ,l gt ,h gt ,θ gt ) is a real bounding box of the point cloud target, x gt , y gt , z gt are real center coordinates of the point cloud target, w gt , l gt , h gt are width, length and height of the three-dimensional prior box respectively, θ gt is a real deflection angle of the point cloud target, (x', y', z', w', l', h', θ') is a predicted bounding box of the point cloud target, x', y', z' are predicted center coordinates of the point cloud target, w', l', h' are width, length and height of the predicted bounding box respectively, θ' is a predicted deflection angle of the target;

[0091] The classification loss function is:

[0092] L cls =-α a (1-p a ) γ logp a ;

[0093] wherein, L cls is the classification loss function, α a is a weight factor of the classification loss function, used to control the class imbalance problem between samples, p a is the class probability of the anchor box, and γ is an adjustable factor used to distinguish difficult samples;

[0094] The model loss function is:

[0095]

[0096] wherein, L is the model loss function, N pos is the number of positive anchor boxes, β loc is the weight of the bounding box regression loss function, β cls is the weight of the classification loss function, and β dir is the weight of the deflection angle loss function, L dir is the deflection angle loss function.

[0097] Taking the backbone network including 4 feature extraction layers as an example, the network structure and working schematic diagram of the point cloud target detection method based on multi-scale feature fusion are shown in Figure 5 The input of the detection head is the 4 groups of feature maps R out1 , R out2 , R out3 and R out4, the output is the bounding box position, size, deflection angle and target category. First, R out4 Four groups of feature maps are generated through four deconvolutions, and their sizes are 62x54x256, 124x108x128, 248x216x64 and 496x432x64 respectively. R out1 , R out2 , R out3 and R out4 After convolution attention, four groups of feature maps S1, S2, S3 and S4 are generated by cascading fusion respectively. S1, S2 and S3 are cascaded to generate feature map I1, and S2, S3 and S4 are cascaded to generate feature map I2. Then I1 and I2 are cascaded after passing through the convolution attention module to generate the final feature map I. I is fed into two 1x1 convolution to perform target classification and bounding box regression respectively.

[0098] Next, the effect of the present scheme is proved by experimental data:

[0099] Experimental setup: In order to facilitate subsequent data processing of each module, the original point cloud data needs to be preprocessed first. First, for the KITTI dataset, set the point cloud data detection range as x∈[0,69.12m], y∈[-39.68m,39.68m], z∈[-3m,1m]; for the DAIR-V2X-I dataset, set the point cloud data detection range as x∈[0,99.84m], y∈[-39.68m,39.68m], z∈[-3m,1m]. The present application regularly columnizes the original point cloud data P in the set data range on the X-Y plane to obtain columnized point cloud P', and the size of each column body is set as [0.16m, 0.16m, 4m]. The maximum number of column bodies contained in the entire three-dimensional space is 12000, and the maximum number of points contained in each column body is 100.

[0100] Evaluation index: For the car class, this paper sets the detection correct when IoU≥0.7; for the pedestrian and cyclist classes, this paper sets the detection correct when IoU≥0.5. At the same time, according to the official standard of KITTI, the detection difficulty of the three types of targets to be detected is divided into three types: easy, medium and difficult according to the size, occlusion and truncation of the target. The algorithm performance is evaluated under the three different detection difficulties.

[0101] 1. Quantitative analysis

[0102] KITTI validation set, Table 2 is the three-dimensional detection AOS value on the KITTI validation set, Table 3 is the three-dimensional detection AP value on the DAIR-V2X-I validation set, and Table 4 is the three-dimensional detection AOS value on the DAIR-V2X-I validation set.

[0103] Table 1 mAP of three-dimensional detection on KITTI validation set

[0104]

[0105] Table 2 mAOS of three-dimensional detection on KITTI validation set

[0106]

[0107] Table 3 mAP of three-dimensional detection on DAIR-V2X-I validation set

[0108]

[0109] Table 4 mAOS of three-dimensional detection on DAIR-V2X-I validation set

[0110]

[0111] According to the experimental results in Tables 1-4, it can be seen that on the experimental data set in this paper, the detection performance of Pillar-FFNet for cars, pedestrians and cyclists is higher than that of PointPillar under three different detection difficulties. On the KITTI validation set, the average precision of car, pedestrian and cyclist detection is increased by 0.84%, 1.81%, and 4.02% under simple detection difficulty, by 0.62%, 2.13%, and 2.39% under medium detection difficulty, and by 0.8%, 1.16%, and 1.58% under difficult detection difficulty; On the DAIR-V2X-I validation set, the average precision of car, pedestrian and cyclist detection is increased by 0.33%, 2.09%, and 4.71% under simple detection difficulty, by 0.17%, 0.17%, and 1.84% under medium detection difficulty, and by 0.17%, 0.17%, and 1.82% under difficult detection difficulty.

[0112] In point cloud data, the line human target is small, and the points can be used to represent pedestrians, so it is difficult to detect. Compared with PointPillar, the Pillar-FFNet proposed in the application can effectively improve the detection performance of small targets such as pedestrians and cyclists without affecting the detection performance of the car category. The main reasons are three points: first, the convolution attention module designed in the application can effectively suppress the redundant information in the network and enhance the effective information in the network; second, high-level features are beneficial to large target detection, and low-level features are beneficial to small target detection. The detection head designed based on the multi-scale feature fusion strategy in the application fully fuses the semantic information and spatial information of each different scale feature. Finally, the residual-based backbone network can effectively extract high-level fine-grained features. In summary, the Pillar-FFNet can effectively improve the detection performance of pedestrians and cyclists without affecting the detection performance of cars.

[0113] 2. Ablation experiment

[0114] This subsection uses different convolution attention and different feature fusion strategies to conduct experimental verification on the KITTI dataset.

[0115] First, five groups of ablation experiments are conducted to test the influence of different fusion methods of the detection head on the detection performance: experiment one replaces the detection head in Pillar-FFNet with the SSD structure detection head in the original PointPillar; experiment two fuses S1 and S2 in the detection head designed in this paper, and fuses S3 and S4; experiment three directly fuses S1, S2, S3, and S4 in the detection head designed in this paper; experiment four fuses S1, S3, and S4 in the detection head designed in this paper, and fuses S2, S3, and S4; experiment five replaces the detection head in this paper with an FPN structure. Experimental verification is conducted on the KITTI validation set, and the experimental results are shown in Tables 5 and 6.

[0116] Table 5 mAP of Pillar-FFNet with different detection heads on KITTI validation set

[0117] Method Car Pedestrian Bicyclist mAP The method 78.17 51.44 65.55 65.05 Experiment 1 77.30 52.83 61.69 63.94 Experiment 2 78.21 52.76 62.98 64.65 Experiment 3 78.07 48.08 61.23 62.46 Experiment 4 74.13 46.86 52.34 57.78 Experiment 5 77.31 49.19 62.83 63.24

[0118] Table 6 mAOS of Pillar-FFNet with different detection heads on KITTI validation set

[0119] Method Car Pedestrian Bicyclist mAOS The method 89.87 55.07 71.89 72.28 Experiment 1 89.48 57.29 69.44 72.07 Experiment 2 89.79 56.91 70.95 72.55 Experiment 3 89.72 53.43 72.22 71.79 Experiment 4 86.38 52.85 59.85 66.36 Experiment 5 89.44 53.43 72.30 71.48

[0120] Secondly, five groups of experiments are carried out to test the influence of the attention module on the detection performance: experiment six replaces the convolution attention module designed in this paper with an SE module; experiment seven replaces the convolution attention module designed in this paper with a CBAM module; experiment eight replaces the convolution attention module designed in this paper with an ECA module; experiment nine replaces the convolution attention module designed in this paper with a 3x3 convolution. The experimental results on the KITTI validation set are shown in Tables 7 and 8.

[0121] Table 7 mAP of Pillar-FFNet with different attention modules on KITTI validation set

[0122] Method Car Pedestrian Bicyclist mAP Experiment 6 77.98 49.86 60.59 62.81 Experiment 7 77.73 51.35 63.63 64.23 Experiment 8 77.75 47.04 63.21 62.67 Experiment 9 77.53 51.41 63.86 64.27

[0123] Table 8 mAOS of Pillar-FFNet with different attention modules on KITTI validation set

[0124] Method Car Pedestrian Bicyclist mAOS Experiment 6 89.77 51.49 71.79 71.02 Experiment 7 89.64 50.67 73.87 71.39 Experiment 8 89.69 52.88 70.06 70.88 Experiment 9 89.66 49.29 70.31 69.75

[0125] From Tables 5 to 8, it can be seen that the most effective one is designed in this application for different fusion strategies and different convolution attention mechanisms in the detection head.

[0126] It should be understood that, although Figure 1 the steps in the flowchart are shown in order according to the arrows, these steps are not necessarily executed in order according to the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps in the flowchart can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or sub-steps or stages of other steps.

[0127] In one embodiment, as Figure 6 shown, a point cloud target detection device based on multi-scale feature fusion is provided, comprising:

[0128] a pseudo-image generation module for generating a pseudo-image according to original point cloud data;

[0129] a deconvolution module for inputting the pseudo-image into a backbone network, and performing multiple deconvolutions on the feature maps output by the last selected feature extraction layer in the backbone network to obtain corresponding multiple groups of deconvolution feature maps; wherein the number of deconvolutions is the same as the number of layers of the selected feature extraction layer;

[0130] The first fusion module is configured to perform cascade fusion on the feature maps output by the selected feature extraction layers in the backbone network and the corresponding deconvolution feature maps respectively, to obtain a plurality of groups of initial fusion feature maps;

[0131] The feature map division module is configured to divide the plurality of groups of initial fusion feature maps to obtain a first feature map set including a plurality of groups of initial fusion feature maps except for the initial fusion feature map corresponding to the first selected feature extraction layer, and a second feature map set including a plurality of groups of initial fusion feature maps except for the initial fusion feature map corresponding to the last selected feature extraction layer;

[0132] The second fusion module is configured to perform cascade fusion on the initial fusion feature maps in the first feature map set to obtain a first fusion feature map, and perform cascade fusion on the initial fusion feature maps in the second feature map set to obtain a second fusion feature map;

[0133] The third fusion module is configured to perform cascade fusion on the first fusion feature map and the second fusion feature map to obtain a target feature map.

[0134] The target detection module is configured to perform point cloud target detection according to the target feature map.

[0135] The specific limitations of the point cloud target detection device based on multi-scale feature fusion can be referred to the limitations of the point cloud target detection method based on multi-scale feature fusion in the foregoing, which will not be described here. Each module in the point cloud target detection device based on multi-scale feature fusion can be realized by software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0136] In one embodiment, a computer device, which can be a server, is provided, and an internal structure diagram of the computer device can be as shown in Figure 7 The computer device includes a processor, a memory, a network interface, and a database connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is configured to store point cloud and other data. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement the point cloud target detection method based on multi-scale feature fusion.

[0137] Those skilled in the art can understand that,Figure 7 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0138] In an embodiment, a computer device is provided, comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps of the method in the above embodiments when executing the computer program.

[0139] A person of ordinary skill in the art can understand that all or part of the processes in the above embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synch link) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0140] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not contradict, they should be considered within the scope of the present application.

[0141] The above embodiments only express several implementation manners of the present application, and the description is specific and detailed, but it should not be understood as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the patent protection scope of the present application should be subject to the appended claims.

Claims

1. A point cloud target detection method based on multi-scale feature fusion, characterized in that, The method comprises: generating a pseudo image according to original point cloud data; after inputting the pseudo image into a backbone network, performing multiple times of deconvolution on a feature map output by a last selected feature extraction layer in the backbone network to obtain corresponding multiple groups of deconvolution feature maps; wherein the number of times of deconvolution is the same as the number of layers of the selected feature extraction layer; concatenating and fusing the feature map output by the selected feature extraction layer in the backbone network with corresponding deconvolution feature maps to obtain multiple groups of initial fusion feature maps; dividing the multiple groups of initial fusion feature maps to obtain a first feature map set including multiple groups of initial fusion feature maps except for the initial fusion feature map corresponding to the first selected feature extraction layer, and a second feature map set including multiple groups of initial fusion feature maps except for the initial fusion feature map corresponding to the last selected feature extraction layer; concatenating and fusing the initial fusion feature maps in the first feature map set to obtain a first fusion feature map, and concatenating and fusing the initial fusion feature maps in the second feature map set to obtain a second fusion feature map; concatenating and fusing the first fusion feature map and the second fusion feature map to obtain a target feature map; performing point cloud target detection according to the target feature map; concatenating and fusing the feature map output by the selected feature extraction layer in the backbone network with corresponding deconvolution feature maps to obtain multiple groups of initial fusion feature maps, comprising: inputting the feature map output by the selected feature extraction layer in the backbone network into a corresponding convolution attention module to obtain corresponding multiple groups of attention feature maps; concatenating and fusing each group of attention feature maps with corresponding deconvolution feature maps to obtain multiple groups of initial fusion feature maps; inputting the feature map output by the selected feature extraction layer in the backbone network into a corresponding convolution attention module to obtain corresponding multiple groups of attention feature maps, comprising: performing channel splitting on the feature map output by the selected feature extraction layer in the backbone network to obtain corresponding first channel feature maps and second channel feature maps; feeding the first channel feature maps into an average pooling layer to obtain an average pooling feature map, inputting the average pooling feature map into a sigmoid function after 1×1 convolution to output a first weight value of the average pooling feature map, multiplying the first channel feature maps with the first weight value to obtain a first channel initial attention map, and adding the first channel initial attention map and the first channel feature map to obtain a first channel attention map; feeding the second channel feature map into a max-pooling layer to obtain a max-pooling feature map, inputting the max-pooling feature map into a 1*1 convolution after which a sigmoid function is inputted, outputting a second weight value of the max-pooling feature map, multiplying the second channel feature map and the second weight value to obtain a second channel initial attention map, and adding the second channel initial attention map and the second channel feature map to obtain a second channel attention map feeding the second channel feature map into a max-pooling layer to obtain a max-pooling feature map, inputting the max-pooling feature map into a 1*1 convolution after which a sigmoid function is inputted, outputting a second weight value of the max-pooling feature map, multiplying the second channel feature map and the second weight value to obtain a second channel initial attention map, and adding the second channel initial attention map and the second channel feature map to obtain a second channel attention map adding the first channel attention map and the second channel attention map and then performing 1×1 convolution to obtain corresponding multiple groups of attention feature maps.

2. The method of claim 1, wherein, concatenating and fusing the first fusion feature map and the second fusion feature map to obtain a target feature map, comprising: inputting the first fusion feature map into a first convolution attention module to obtain a first fusion attention feature map, and inputting the second fusion feature map into a second convolution attention module to obtain a second fusion attention feature map; performing upsampling on the first fusion attention feature map and the second fusion attention feature map and then concatenating and fusing to obtain an initial target feature map; Input the initial target feature map into a third convolutional attention module to obtain a target feature map.

3. The method of claim 1, wherein, The feature extraction network in the backbone network is a convolutional residual block; The feature extraction step through each convolutional residual block comprises: obtaining a feature map input by a current convolutional residual block; performing 1 1x1 convolution, 1 3x3 convolution with a step of t and 1 1x1 convolution on the feature map in sequence to obtain a first branch feature map; performing 1 3x3 convolution on the feature map to obtain a second branch feature map; performing 1 1x1 convolution on the first branch feature map and the second branch feature map after cascaded fusion to obtain a feature map output by the current convolutional residual block.

4. The method of claim 3, wherein, The number of layers of the selected feature extraction layer is 4; The step of outputting a feature map by the selected feature extraction layer in the backbone network comprises: the pseudo image passes through 2 convolutional residual blocks with a step of 1 to generate a first selected feature map; the first selected feature map passes through 1 convolutional residual block with a step of 2 and 2 convolutional residual blocks with a step of 1 in sequence to generate a second selected feature map; the second selected feature map passes through 1 convolutional residual block with a step of 2 and 5 convolutional residual blocks with a step of 1 in sequence to generate a third selected feature map; the third selected feature map passes through 1 convolutional residual block with a step of 2 and 5 convolutional residual blocks with a step of 1 in sequence to generate a fourth selected feature map.

5. The method according to any one of claims 1 to 4, characterized in that, Generating a pseudo image according to original point cloud data comprises: obtaining original point cloud data and setting a point cloud data detection range; performing columnarization on the original point cloud data in the point cloud data detection range on an X-Y plane to obtain columnarized point cloud; inputting the columnarized point cloud into a pre-constructed feature encoding network to obtain a pseudo image.

6. The method of claim 5, wherein, Before performing point cloud target detection according to the target feature map, comprising: obtaining a pre-constructed target detection model; inputting a training point cloud data set into the target detection model, calculating and optimizing a loss function to obtain a trained target detection model; the model loss function comprises a bounding box regression loss function and a classification loss function; the bounding box regression loss function is: ； ； ； ； ； ； wherein, is a bounding box regression loss function, is a loss function, is a three-dimensional prior box of a point cloud target, is a three-dimensional prior center coordinate of a point cloud target, are width, length and height of the three-dimensional prior box, respectively, is a prior yaw angle of a point cloud target, is a real bounding box of a point cloud target, is a real center coordinate of a point cloud target, are width, length and height of the three-dimensional prior box, respectively, is a real yaw angle of a point cloud target, is a predicted bounding box of a point cloud target, is a predicted center coordinate of a point cloud target, are width, length and height of the predicted bounding box, respectively, is a predicted yaw angle of the target; the classification loss function is: ； wherein, is a classification loss function, is a weight factor for the classification loss function, is a class probability of an anchor box, is an adjustable factor; the model loss function is: wherein, is a model loss function, is a number of positive anchor boxes, is a weight of a bounding box regression loss function, is a weight of a classification loss function, is a weight of a yaw angle loss function, is a yaw angle loss function.

7. A point cloud object detection device based on multi-scale feature fusion, implemented by the method of any one of claims 1-6, characterized in that, the device comprises: a pseudo image generation module for generating a pseudo image according to original point cloud data; a deconvolution module for inputting the pseudo image into a backbone network and performing multiple deconvolutions on a feature map output by a last selected feature extraction layer in the backbone network to obtain corresponding multiple groups of deconvolution feature maps; the number of deconvolutions is the same as the number of layers of the selected feature extraction layer; a first fusion module for performing cascaded fusion of the feature map output by the selected feature extraction layer in the backbone network and the corresponding deconvolution feature map to obtain multiple groups of initial fusion feature maps; a feature map division module for dividing the multiple groups of initial fusion feature maps to obtain a first feature map set comprising other multiple groups of initial fusion feature maps except for the initial fusion feature map corresponding to the first selected feature extraction layer, and a second feature map set comprising other multiple groups of initial fusion feature maps except for the initial fusion feature map corresponding to the last selected feature extraction layer. a second fusion module, configured to perform cascade fusion on initial fusion feature maps in the first set of feature maps to obtain a first fusion feature map, and perform cascade fusion on initial fusion feature maps in the second set of feature maps to obtain a second fusion feature map; a third fusion module, configured to perform cascade fusion on the first fusion feature map and the second fusion feature map to obtain a target feature map; a target detection module, configured to perform point cloud target detection according to the target feature map. 8.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-7. The processor implements the steps of the method in any one of claims 1 to 6 when executing the computer program.

Images