Three-dimensional scene motion trend estimation method based on scattered point cloud cross-attention learning

Abstract

The application discloses a three-dimensional scene motion trend estimation method based on a scattered point cloud cross-attention learning, relates to the field of deep learning and three-dimensional point cloud scene understanding, and aims at three-dimensional scene motion trend estimation. On the basis of extracting point cloud features of front and rear frames by using a convolutional neural network, the network model captures joint information between the point clouds of the front and rear frames by constructing a correlation feature extraction module, so that the richness of the point cloud features is improved. After the point cloud motion trend is initially obtained, the calibration module of the three-dimensional scene motion trend is constructed to fine-tune, so that the accuracy and reliability of the three-dimensional scene motion trend estimation are further improved.

Description

Technical Field

[0001] This invention relates to the fields of deep learning and 3D point cloud scene understanding, and particularly to a method for estimating the motion trend of 3D scenes based on cross-attention learning of scattered point clouds. Background Technology

[0002] Point clouds are points representing the surface features and spatial distribution of a target object in a common reference coordinate system. They contain rich information, including 3D coordinates, color, intensity values, time, and more. In recent years, with the widespread use of 3D scanning equipment such as depth cameras and LiDAR, the analysis and research of 3D point cloud data has advanced. As a compact form of 3D data representation, point clouds can preserve the original geometric information of 3D space and are widely used in various fields such as robot navigation, autonomous driving, augmented reality, and healthcare.

[0003] Scene understanding is the process of perceiving, analyzing, and interpreting observed 3D dynamic scenes, typically in real-time. This process primarily involves matching signals from sensors sensing the environment with models humans use to understand the scene. Target detection, classification, and recognition based on 3D point cloud data are currently the main technologies for scene understanding. In dynamic 3D scenes, such as autonomous driving, the motion and trends of point clouds are crucial for scene understanding. Past 3D scene motion estimation methods were mainly based on 2D input features, extending optical flow estimation methods to stereo or RGBD images. With the development of point cloud research technology, methods have gradually emerged in recent years that estimate object motion trends by directly processing 3D point clouds. These methods select two consecutive frames of point clouds from the same scene as input, use deep neural networks to extract features from the two frames respectively, and then embed the point cloud feature input stream into a layer to estimate the point cloud motion trend. These methods directly estimate point cloud motion in an end-to-end manner, which is innovative and has achieved good results, but some problems still exist: On the one hand, the two frames of point clouds as input belong to different times of the same scene, and their feature information should be related and similar. When using deep learning networks to extract feature information from the input point clouds, these methods only process the two frames of point clouds separately without considering the feature correlation information between the two point clouds, which affects the accuracy of motion estimation in subsequent steps; on the other hand, the motion of point clouds in adjacent positions should be similar. Traditional methods do not consider the interrelationship of the motion of point clouds in adjacent positions after estimating the motion trend of the point clouds, resulting in a high error in the final estimation result.

[0004] The attention mechanism, proposed by Bengio's team in 2014, has been widely applied in various fields of deep learning in recent years, such as capturing receptive fields in images in computer vision and locating key features in natural language processing (NLP). In 2017, Google's machine translation team proposed a Transformer-based attention mechanism, abandoning traditional convolutional neural networks (CNNs) and recurrent neural networks (RNNs). The entire network structure is composed entirely of self-attention and feedforward neural networks. In 2021, Jia Jiaya et al. proposed a Transformer structure for 3D point clouds, achieving excellent results in point cloud classification and semantic segmentation tasks. Point clouds are essentially collections of embedded metric spaces, exhibiting unordered and sparsity. Compared to language or image processing, the Transformer is more suitable for point cloud processing. Summary of the Invention

[0005] To address the aforementioned shortcomings, this application provides a method for estimating the motion trend of a 3D scene based on cross-attention learning of scattered point clouds. This method can estimate the motion trend of point clouds more accurately and effectively.

[0006] To achieve the above-mentioned technical objectives, the technical solution of the present invention is as follows:

[0007] A method for estimating motion trends in 3D scenes based on scattered point cloud cross-attention learning, comprising the following steps:

[0008] Step 1: Use the nearest neighbor algorithm (KNN) to find nearest neighbor points for the first frame point cloud X and the second frame point cloud Y respectively, and obtain the nearest location information (Gragh) of each point;

[0009] Step 2: Input the acquired proximity information (Gragh) into a multi-layer convolutional network to extract multi-layer point cloud feature vectors Fx and Fy;

[0010] Step 3: Connect the feature vectors of the multi-layer point cloud and input them into a one-dimensional convolutional network for feature aggregation to obtain new point cloud feature vectors Fx and Fy. The feature information obtained in this way contains both global features and can also capture the local features of the point cloud.

[0011] Step 4: Input the point cloud feature vectors Fx and Fy into the associated feature extraction module, and output a new point cloud feature vector. The resulting new point cloud feature vector Adding these features to Fx and Fy yields the final two frames of point cloud features;

[0012] Step 5: Based on the features of the two point clouds, calculate the weights (w) of each point in point cloud X and each point in point cloud Y as corresponding points using the Sinkhorn algorithm;

[0013] Step 6: Calculate the motion vector (flow) of each point in the point cloud X based on the weights (w) obtained in the previous step;

[0014] Step 7: Input the motion vector (flow) obtained in the previous step and the neighborhood information (Gragh) of point cloud X into the calibration module to obtain the estimated motion trend of the point cloud.

[0015] A further improvement to this method is that the calibration module includes three two-dimensional convolutional layers and one linear layer.

[0016] A further improvement to this method is that in step 4, the point cloud features are input into the associated feature extraction module to capture the association information between point clouds.

[0017] A further improvement to this method is that, after initially obtaining the motion vector of the point cloud, the motion vector (flow) and the neighboring position information (Gragh) of point cloud X are input into the calibration module, and the motion vector is fine-tuned according to the positional relationship between the point clouds.

[0018] The technical advancements achieved by this invention due to the adoption of the above technical solutions are as follows: 1) After extracting point cloud features using a deep neural network, this method uses a Transformer based on a cross-attention mechanism as the associated feature extraction module, enabling the point cloud features to perceive and capture the positional encoding information of another frame of point cloud while retaining their own information; furthermore, by using the Sinkhorn algorithm, based on the feature similarity of the point clouds, the corresponding points of the first frame of point cloud are estimated in the global scope of the second frame of point cloud, thereby estimating the point cloud motion. Finally, the motion trend of the point cloud is fine-tuned through a motion correction network. Compared with previous methods, the method in this invention can estimate the point cloud motion trend more accurately and effectively.

[0019] 2) The point cloud feature extraction method was improved by capturing the joint features between two frames of point clouds through the correlation feature extraction module and by using the calibration module, which improved the computational accuracy of the network. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1This is a schematic diagram of the overall network structure of the present invention;

[0022] Figure 2 This is a schematic diagram of the structure of the correlation feature extraction module used in the network. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] This invention discloses a method for estimating motion trends in a 3D scene based on scattered point cloud cross-attention learning, comprising the following steps:

[0025] Step 1: Perform KNN on the first frame point cloud X and the second frame point cloud Y respectively to find the nearest neighbor points and obtain the neighborhood information (Gragh) of each point.

[0026] Select the K points {x1, x2, ..., x3} closest to point x. k ,}, and set its coordinates (x i ,x j ,x k ) and (x ki ,x kj ,x kk ) connect as (x i ,x j ,x k ,x ki ,x kj ,x kk This allows us to obtain the neighborhood information (Gragh) of each point.

[0027] Step 2: Input the acquired proximity information (Gregor) into a multi-layer convolutional network to extract point cloud features.

[0028] The proximity information (Gregor) is input into a one-dimensional convolutional network with layers of 32, 32, 64, and 128 in sequence. After each layer is computed, a batch normalization layer and an activation function layer (ReLU) are added.

[0029] Step 3: Concatenate the output tensors of each network layer and input them into a one-dimensional convolutional network for feature aggregation to obtain a new point cloud feature vector F. x F y The feature information obtained in this way includes both global features and can also capture the local features of the point cloud very well.

[0030] The outputs of each convolutional network layer are concatenated to form a 256-layer feature vector, which is then input into a one-dimensional convolutional network for feature fusion, ultimately yielding a 256-dimensional two-point cloud feature F. x F y .

[0031] Step 4: Extract the feature vector F of the point cloud x F y The input is processed by the cross-attention-based correlation feature extraction module, which outputs new point cloud features. These new feature vectors are then compared with F... x F y Add them together to obtain the final point cloud features.

[0032]

[0033]

[0034] Wherein, φ:R N×P ×R N×P →R N×P φ is the asymmetric mapping function learned by the Transformer, where N is the number of points in the point cloud and P represents the feature dimension. φ is considered a residual term, which can be based on F... x F y The input order provides it with an additional change φ(F) x ,F y ). and F x In comparison, besides including the feature information of point cloud X itself, We also learned about the structural information related to X in the point cloud Y.

[0035] Step 5: After obtaining the point cloud features, use the Sinkhorn algorithm to calculate the weights of each point in point cloud X and each point in point cloud Y.

[0036] First, based on point cloud features Calculate the transportation cost matrix from point cloud X to point cloud Y:

[0037]

[0038] in:

[0039]

[0040] When the distance between two points is greater than the set threshold value d max At that time, its transportation cost C ij Set it to infinity.

[0041] The more similar the point cloud features of two points, i.e., the smaller the feature distance, the lower the transportation cost C. ijThe smaller the value, the better. After obtaining the transportation cost of each corresponding point, the Sinkhorn algorithm is used to calculate the weight matrix W of the point clouds for the two frames.

[0042] Step 6: Calculate the motion vector (flow) of each point in the first frame point cloud X based on the weights (w) obtained in the previous step.

[0043] The coordinate differences between point x and point Y in the first frame of the point cloud are multiplied by their corresponding weights, and then weighted to obtain the motion vector f of point x. x .

[0044] The calculation formula is:

[0045]

[0046] Step 7: Input the motion vector (flow) obtained in the previous step and the nearest neighbor location information (Gregor) of the input point cloud X into the motion correction network, and output the corrected motion vector (f). out ).

[0047] In a point cloud, adjacent points should have similar motion trends. Therefore, after initially obtaining the motion vector of the point cloud X, it is fed into a convolutional neural network containing Gragh information about nearest neighbors. This network consists of three fully connected layers and a linear transformation layer. The motion vector is further adjusted to obtain the change Δ(f), which is then compared with f. x Adding them together gives f out .

[0048] f out =f x +Δ(f) (6)

[0049] Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. Furthermore, those skilled in the art will recognize that, based on the ideas of this invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A method for estimating motion trends in 3D scenes based on scattered point cloud cross-attention learning, characterized in that, Includes the following steps: Step 1: Use the nearest neighbor algorithm to find the nearest neighbor points for the first frame point cloud X and the second frame point cloud Y respectively, and obtain the neighbor location information of each point; Step 2: Input the acquired proximity location information into a multi-layer convolutional network to extract multi-layer point cloud feature vectors Fx and Fy; Step 3: Connect the feature vectors of the multi-layer point cloud and input them into a one-dimensional convolutional network for feature aggregation to obtain new point cloud feature vectors Fx and Fy; Step 4: Extract the feature vectors of the point cloud , The input is processed by the cross-attention-based correlation feature extraction module, which outputs new point cloud features. These new feature vectors are then combined with... , Add them together to obtain the final point cloud features. , ; (1) (2) in, : It is the asymmetric mapping function learned by Transformer, where N is the number of points in the point cloud and P represents the feature dimension. Treated as a residual term, based on , The input order provides it with an additional option for modification. ;and In comparison, besides including the feature information of point cloud X itself, We also learned about the structural information related to X in the point cloud Y; Step 5: Based on the features of the two point clouds, calculate the weights of each point in point cloud X and each point in point cloud Y as corresponding points using the Sinkhorn algorithm; Step Six: Calculate the motion vector of each point in point cloud X based on the weights obtained in the previous step. Multiply the coordinate difference between point x in the first frame of point cloud and each point in point cloud Y by its corresponding weight, and sum the weights to obtain the motion vector of point x. ; Step 7: Input the motion vector obtained in the previous step and the neighboring location information of point cloud X into the calibration module to obtain the estimated motion trend of point cloud.

2. The method for estimating the motion trend of a 3D scene based on scattered point cloud cross-attention learning according to claim 1, characterized in that, The calibration module includes three two-dimensional convolutional layers and one linear layer.

3. The method for estimating the motion trend of a 3D scene based on scattered point cloud cross-attention learning according to claim 1, characterized in that: After obtaining the motion vectors of the point cloud, the motion vectors and the neighboring position information of point cloud X are input into the calibration module, and the motion vectors are fine-tuned according to the positional relationship between the point clouds.

Images