Three-dimensional point cloud feature extraction method based on multi-modal attention driving

Abstract

The invention discloses a three-dimensional point cloud feature extraction method based on multi-modal attention driving, and the method comprises the steps: collecting corresponding 2D image data and 3D point cloud data in a 3D space of a to-be-detected object, and marking a 3D bounding box of the to-be-detected object and the type of the to-be-detected object in the 3D point cloud data; constructing a feature extraction network; inputting the 2D image data and the 3D point cloud data into a feature extraction network for training until the VoteNet loss function converges; and inputting the collected corresponding 2D image data and 3D point cloud data in the 3D space of the to-be-detected object into the trained feature extraction network, and outputting the 3D bounding box of the to-be-detected object and the category information of the to-be-detected object. According to the 3D point cloud feature extraction method, the relationship between the 2D image data and the 3D point cloud data is established by using the attention weight matrix, so that data of different modalities can act on extraction of 3D point cloud features, and the 3D point cloud feature extraction effect is further improved.

Description

technical field [0001] The invention relates to the technical field of 3D point cloud data processing, in particular to a method for extracting three-dimensional point cloud features based on multimodal attention drive. Background technique [0002] With the rapid development of 3D scanning equipment, 3D point cloud data has been widely used in various fields of computer vision and computer graphics. However, in the process of data acquisition, due to occlusion, illumination and other reasons, the obtained 3D point cloud data often has defects. Using such data to calculate the visual algorithm, the results obtained often have problems of low precision and poor effect. To this end, there are many research works, starting to consider integrating multi-modal information such as text and 2D images into 3D vision algorithms to further improve the effect of point cloud-based 3D vision algorithms. [0003] At present, it is the current mainstream practice to consider 2D images as ...

AI Technical Summary

Problems solved by technology

The first is to simply compress the depth features of the 2D image into a vector, then copy the vector into a matrix of the same size as the 3D point cloud feature, and then directly stitch it to the point cloud feature. This method is simple to operate, but cannot Establishing a direct connection between the starting point cloud features and image features does not significantly improve the algorithm

The second is to establish the relationship between 2D image features and 3D point cloud features, and then realize the fusion of features based on this relationship. The disadvantage of this method is that the relationship between 2D image features and 3D point cloud features is Difficult to determine, usually requires a lot of delicate operations, so it also limits the application of this method

For the fusion of 2D image features and 3D point cloud features, and it is difficult to further improve the effect of 3D vision algorithm based on point cloud, no simple and effective solution has been proposed yet

Images