3D point cloud object detection method based on attention mechanism and image feature fusion

By adopting an image feature fusion method based on attention mechanism in 3D object detection, the sparsity and disorder of point cloud data are solved, the accuracy and real-time nature of 3D object detection are improved, and the distance is achieved is achieved and small object detection effect.

CN115115917BInactive Publication Date: 2024-11-26SOUTHEAST UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210659670.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-13
Publication Date
2024-11-26
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing 3D object detection methods have poor adaptability in unknown scenes. The sparsity and disorder of point clouds lead to difficulty in feature fusion, and the inability to effectively predict long-distance and small objects, large calculations, and difficulty in real-time detection.

Method used

Using a 3D point cloud object detection method based on the fusion of attention mechanism and image features, data enhancement of RGB images and point cloud data is used, and features are extracted in combination with CBAM and coordinate attention mechanisms. After the pseudo-image and image features are stitched, the feature pyramid is input to the feature Fusion feature extraction, and 2D and 3D detection frame prediction are performed.

Benefits of technology

It improves the accuracy of 3D object detection, and can use image and point cloud information at the same time to achieve faster and real-time detection, improves the accuracy of long-distance and small object detection, and reduces the amount of calculation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115115917B_ABST
    Figure CN115115917B_ABST
Patent Text Reader

Abstract

The present invention discloses a 3D point cloud target detection method based on attention mechanism and image feature fusion, relates to the field of computer vision technology, solves the technical problem that the 3D target detection algorithm is not accurate enough, and the key points of its technical solution are to use image data and laser point cloud data as input, respectively input into the image feature extraction network and the point cloud feature encoder, learn important information based on the attention mechanism in the image feature extraction backbone network, and then perform feature fusion of the extracted feature map and the point cloud pseudo image. Finally, the generated fusion feature is sent to the feature pyramid, and the final fusion feature map is used as the input of the 2D-3D detector, the 2D and 3D frame coordinate values ​​are predicted, and then the 2D and 3D frame regression is completed through non-maximum suppression and data conversion to obtain the final result. The present invention makes up for the defects of point cloud information by using the image feature extraction network based on the attention mechanism, and improves the accuracy of 3D target detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a 3D point cloud target detection method based on attention mechanism and image feature fusion. Background Art

[0002] As one of the important tasks in computer vision, object detection is a technical branch with important research value in the artificial intelligence system. Its main task is to accurately and efficiently identify and locate a large number of object instances of predefined categories from images and give the bounding box of each object. In recent years, with the development of deep learning, the research on object detection in 3D images has made great breakthroughs. Compared with 2D object detection, 3D object detection combines depth information and can provide spatial scene information such as the position, direction and size of the target. It has developed rapidly in the fields of autonomous driving and robotics.

[0003] Point cloud data in 3D vision provides accurate structure and depth information, but it has disadvantages such as sparsity, disorder, and sensitivity to noise. Therefore, CNN, which has been well studied in 2D detection, cannot be directly used to process point clouds. In addition, the representation of point clouds directly affects the performance of the model, which brings great challenges to target detection. According to the type of input data, 3D target detection frameworks can be divided into three categories:

[0004] (1) 3D object detection methods based on monocular cameras: RGB images captured by monocular cameras contain rich texture information and high resolution, but lack depth information, so 2D object detection cannot provide all the information needed to perceive the environment. Since there is no depth information available, most methods first detect 2D candidate objects and then use neural networks, geometric constraints, or 3D model matching to predict the 3D bounding box containing the object. 3DOP extends FastR-CNN in the network structure, adding a context branch and direction regression loss to the last convolutional layer to jointly learn the position and direction of the object. This work considers depth images to generate proposals in a similar framework. The Mono3D model only uses monocular images and slightly improves the performance obtained using depth images. It uses context, semantics, hand-designed shape features, and position priors to project each candidate box into the image and score it. The lack of depth information in image features reduces the accuracy of detection and localization, especially for distant and occluded objects, as well as sensitivity to light and weather conditions, which limits the use of these methods during the day.

[0005] (2) Point cloud-based 3D object detection method: The point cloud data collected by LiDAR is an important source for autonomous vehicles to obtain information about the surrounding scene. It can generate accurate depth estimates in scenes such as nighttime, which can improve the safety of autonomous vehicles. However, the sparsity, disorder, local sensitivity and large amount of computation of point clouds make it difficult to apply in practice. PointNet can directly process point cloud data. The structure proposes a spatial transformation network to solve the rotation problem and applies maximum pooling to solve the disorder problem. The segmented 3DPCLs are used as input to perform object classification and part segmentation. The network uses a fully connected layer to perform point-by-point transformation and aggregates global features through a maximum pooling layer to ensure the independence of point order. VoxelNet proposes to divide the point cloud into voxels in space, and then use the PointNet structure (MLP, maxpooling, concat) for each voxel multiple times. Then, the convolution operation of 3DCNN is used to obtain the interaction between sparse voxels, and finally the RPN network is added to perform the 3D object detection task. In applications such as autonomous driving, the vertical direction (z-axis) of the point cloud data in the 3D scene has lower information content than the other two axes. Based on this, PointPillars divides the grid on the xy plane, and then presses the point cloud in the vertical direction onto the grid, and uses the 2DCNN method for object detection.

[0006] (3) 3D target detection method based on fusion information: Different types of sensors have their own advantages and disadvantages. A single sensor cannot achieve accurate and efficient detection. For example, the advantage of image features is that they contain rich texture information and high resolution, but the disadvantage is that they lack depth information. Point cloud data provides accurate structure and depth information, but has disadvantages such as sparsity, disorder and local sensitivity. To this end, fusing multiple sensors with complementary characteristics to enhance perception capabilities has become an emerging research topic. At present, the commonly used fusion data in the field of 3D target detection are images and point clouds. In order to fuse RGB image and point cloud data information, the MVX-Net network first uses a two-dimensional detection network to extract features, pre-trains on ImageNet, and then fine-tunes the 2D target detection task to encode semantic information as prior knowledge to help infer the existence of the target. For point clouds, features are extracted through VoxelNet, and image features and point cloud features are fused to obtain the final detection results.

[0007] The above existing 3D object detection methods have poor adaptability to unknown scenes. Since point clouds are irregular and disordered, while images are regular and ordered, there is a huge difference in the processing methods of images and point clouds, making feature fusion difficult. In addition, the sparsity and disorder of point clouds also make existing methods unable to predict distant and small objects well, and the amount of calculation is large, making real-time detection difficult. Summary of the invention

[0008] This application provides a 3D point cloud target detection method based on attention mechanism and image feature fusion. Its technical purpose is to improve the accuracy of 3D target detection algorithm so that 3D detection can perceive color, semantics and texture information, etc.

[0009] The above technical objectives of this application are achieved through the following technical solutions:

[0010] A 3D point cloud target detection method based on attention mechanism and image feature fusion, comprising:

[0011] S1: Data enhancement of RGB images and point cloud data in 3D object detection datasets;

[0012] S2: inputting the enhanced RGB image into an image feature extraction network, wherein the image feature extraction network extracts features of the enhanced RGB image in combination with an attention mechanism to obtain a first feature representing image information;

[0013] S3: input the enhanced point cloud data into the point cloud feature encoder for feature extraction to obtain point cloud features, and process the point cloud features into a pseudo image;

[0014] S4: splicing the first feature and the pseudo image in the channel dimension and inputting them into a feature pyramid for feature extraction to obtain a fusion feature;

[0015] S5: Predicting the 2D detection frame and the 3D detection frame of the fusion feature to obtain a prediction result, performing data conversion and indicator analysis on the prediction result to obtain a final 3D object detection result.

[0016] Furthermore, the backbone of the image feature extraction network is a residual structure, including stacked downsampling layers, an attention mechanism module and an SPPF module arranged in sequence; the attention mechanism module is based on the CBAM attention mechanism and the coordinate attention mechanism.

[0017] Furthermore, the point cloud feature encoder performs feature extraction through point cloud features based on PointPillars.

[0018] Furthermore, the data enhancement of the RGB image includes HSV hue transformation, affine transformation and mosaic operation.

[0019] The beneficial effects of the present application are as follows: the 3D point cloud target detection method described in the present application can simultaneously utilize image information and point cloud information, thereby achieving better results in 3D target detection; at the same time, feature extraction is performed on RGB images through the attention mechanism to obtain a feature map representing the image information, thereby making the detection speed faster and the real-time performance better. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A flowchart of the method described in this application;

[0021] Figure 2 This is the overall network architecture diagram of the method described in this application;

[0022] Figure 3 Schematic diagram of the CBAM attention mechanism structure used in this application method;

[0023] Figure 4 Schematic diagram of the coordinate attention mechanism structure used in this application method;

[0024] Figure 5 A schematic diagram of the structure of the feature pyramid used in the method of this application;

[0025] Figure 6 The AP obtained by the present application method and the existing method BEV With AP 3D Schematic diagram of comparison results;

[0026] Figure 7 The AP obtained by this application method and the method using only point cloud features in the Cyclists category BEV With AP 3D Schematic diagram of the comparison results. DETAILED DESCRIPTION

[0027] The technical solution of the present application will be described in detail below with reference to the accompanying drawings.

[0028] like Figure 1 and Figure 2 As shown, the 3D point cloud target detection method described in this application specifically includes the following steps:

[0029] Step 1: The RGB images in the 3D object detection dataset are resized to 1024×1024 after data enhancement methods such as HSV hue transformation, affine transformation, and mosaic operation, and used as the input of the image feature extraction network.

[0030] Specifically, the 3D object detection dataset of the KITTI dataset is selected. The KITTI dataset contains a total of 7481 training images and 7518 test images, as well as corresponding point clouds and calibration parameters; it includes 80256 2D and 3D labeled objects of three types, namely cars, pedestrians and bicycles. According to the degree of occlusion and truncation of the object, each 3D ground truth is divided into three difficulty categories: easy, medium and difficult. Here we divide the 7481 training images into a dataset consisting of 3712 training sets and 3769 validation sets, and the size of the input images is 375×1242.

[0031] Step 2: After random sampling, rotation, translation, flipping along the x-axis, and random scaling, the lidar point cloud data in the 3D object detection dataset is sent to the point cloud feature encoder.

[0032] Here, we ensure that the training set and validation set of the point cloud data are divided into one-to-one correspondence with the image data. The dimension of each point cloud data is (x, y, z, r), (x, y, z) represents the coordinates of the point, r represents the reflection intensity of the point (related to the object material and laser incident angle, etc.), and the point cloud range captured by the KITTI dataset is [0, -39.68, -3, 69.12, 39.68, 1]. The point cloud feature encoder divides a group of up to 32 points into pillars with a size of [0.16, 0.16, 4]. When performing data enhancement, the points in each pillar are increased to 6 dimensions of data, expressed as: (x c ,y c ,z c ,x p ,y p ,z p ), the subscript c represents the offset of each point cloud to the average value of all points in the pillar corresponding to the point, and the subscript p represents the offset of the point from the center point of the pillar on the x-axis, y-axis, and z-axis. At this time, the point cloud is represented as a tensor of size [D, P, N], where D represents the 10 features of each point cloud, P represents all non-empty pillars, and N represents the maximum number of points in each pillar. The features of the points are then extracted through a 1×1 convolutional layer, a one-dimensional batch normalization layer, and a ReLu activation function. In order to simplify the calculation, the maximum pooling operation is used to extract the point in each pillar that best represents the pillar.

[0033] Step 3: The image feature extraction network combines the attention mechanism to extract the input image data into a feature map (i.e., the first feature) representing the image information.

[0034] The framework of the image feature extraction network includes stacked downsampling layers + attention mechanism modules + SPPF modules. The attention mechanism is based on the CBAM attention mechanism (such as Figure 3 ) and coordinate attention mechanisms (as shown Figure 4 As shown in the figure, the backbone of the image feature extraction network is a residual structure. After extracting features, spatial pyramid pooling is performed to obtain a feature map representing image information. Its input is 3-channel image data with a size of 1024×1024, and its output is a feature map with 256 channels and a size of 192×192.

[0035] Step 4: After the point cloud feature encoder extracts the input point cloud features, it repositions the features back to their original coordinate distribution according to their indices to form a pseudo image representation.

[0036] The point cloud feature encoder is based on the point cloud feature extraction part of PointPillars.

[0037] Step 5: splice the image information feature map (first feature) obtained in step 3 and the pseudo image obtained in step 4 in the channel dimension and input them into the deep neural network to obtain the final fusion feature.

[0038] The image information feature map is adjusted to 496×432 size through transposed convolution, and the number of channels is adjusted to the same as the point cloud information represented by the pseudo image after 1×1 convolution; then the two are spliced ​​in the channel dimension and input into the feature pyramid for feature extraction to obtain fusion features. The fusion feature extraction network is a 3-layer feature pyramid, and its structure is as follows Figure 5 shown.

[0039] Step 6: Predict the 2D detection box and 3D detection box of the final fusion feature, perform data conversion and indicator analysis on the prediction results, and output the final 3D object detection result.

[0040] As a specific embodiment, each annotation box contains seven parameters (x, y, z, ω, l, h, θ), where (x, y, z) represents the predicted position of the center coordinates of a annotation box in the point cloud, ω, l, h respectively predict the length, width, and height data of a annotation box, and θ predicts the rotation angle of the annotation box. The regression residual of the positioning task is expressed as:

[0041]

[0042]

[0043] where x gt Indicates the x length of the annotation box, x a Indicates the length information of the prior frame, d a represents the diagonal distance between the length and width of the prior box, and y gt 、z gt ,ω gt , l gt 、h gt ,θ gt Both represent the parameters of the annotation box; a 、z a 、h a ,ω a , l a ,θ a They all represent the parameters of the prior box.

[0044] The total regression loss is thus expressed as:

[0045] L loc =∑ b∈(x,y,x,z,ω,l,h,θ) SmoothL1(Δb)

[0046] For the object category classification of each prior box, focal loss is used to adjust the positive and negative sample balance, and the category loss is recorded as L cls , expressed as:

[0047] L cls =-α a (1-p a ) γ logp a ; where α = 0.25, γ = 2, p a Represents the classification probability of the prior box in the three categories of car, pedestrians, and cyclists.

[0048] Since it is impossible to completely distinguish two prediction boxes with completely opposite directions during angle regression, the Softmax function is used to predict the direction of the prior box during implementation:

[0049] where β loc =2,β cls =1,β dir =0.2;

[0050] L dir represents the direction loss, N ops Indicates the number of predicted positive sample annotation boxes.

[0051] In order to illustrate the effect of this application, the following experimental comparison is provided:

[0052] The experiment was evaluated under an IoU (intersection over union, reflecting the detection effect of the predicted detection box and the real detection box) threshold of 0.7, the learning rate was selected as 0.003, which decayed with training, the weight of the momentum method was 0.9, the optimizer selected the Adam algorithm, the batch size was 4, and the training was performed for 80 rounds.

[0053] AP (Average precision) stands for average precision and is an evaluation indicator of mainstream target detection models. BEV Indicates the accuracy of the detection box under the BEV view; AP 3D It indicates the accuracy of the 3D detection frame and is calculated using the AP|R11 standard. The larger the AP value, the higher the accuracy of target detection. 3D Value pairs such as Figure 6The proposed method and the method using point cloud features are improved by 2.13, 1.51 and 1.47 respectively in the three difficulty levels of Cyclists, namely, simple, medium and difficult. Figure 7 The experimental results show that the image feature extraction network with the attention mechanism described in this application can effectively improve the accuracy of 3D object detection.

[0054] Obviously, the above embodiments are only some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

Claims

1. A 3D point cloud object detection method based on attention mechanism and image feature fusion, It is characterized in that include: S1: Data enhancement of RGB images and point cloud data in 3D object detection datasets; S2: inputting the enhanced RGB image into an image feature extraction network, wherein the image feature extraction network extracts features of the enhanced RGB image in combination with an attention mechanism to obtain a first feature representing image information; S3: input the enhanced point cloud data into the point cloud feature encoder for feature extraction to obtain point cloud features, and process the point cloud features into a pseudo image; S4: splicing the first feature and the pseudo image in the channel dimension and inputting them into a feature pyramid for feature extraction to obtain a fusion feature; S5: Predicting the 2D detection frame and the 3D detection frame of the fusion feature to obtain a prediction result, performing data conversion and indicator analysis on the prediction result to obtain a final 3D object detection result.

2. The 3D point cloud target detection method according to claim 1, It is characterized in that The backbone of the image feature extraction network is a residual structure, including stacked downsampling layers, an attention mechanism module and an SPPF module arranged in sequence; the attention mechanism module is based on the CBAM attention mechanism and the coordinate attention mechanism.

3. The 3D point cloud target detection method according to claim 1, It is characterized in that The point cloud feature encoder performs feature extraction through point cloud features based on PointPillars.

4. The 3D point cloud target detection method according to claim 1, It is characterized in that The data enhancement of the RGB image includes HSV hue transformation, affine transformation and mosaic operation.