A point cloud long-term prediction method for autonomous driving scene flow

Abstract

The present invention provides a method for long-term point cloud prediction for autonomous driving scene flows in the field of autonomous driving technology. The method comprises the following steps: S1, converting 3D point cloud data into multi-views; S2, extracting multi-view point cloud features; S3, encoding the multi-view point cloud features using a vector quantization variational autoencoder; S4, establishing a point cloud diffusion model, and training the point cloud diffusion model based on the encoded multi-view point cloud features and a loss function; S5, performing long-term point cloud prediction based on the trained point cloud diffusion model. Through the method of the present invention, the purpose of long-term point cloud prediction can be achieved without losing the spatial structure information, semantic information, boundary continuity, etc. of the point cloud, and the generalization ability is good. The future long-term motion trajectory and change trend of objects in the autonomous driving scene can be effectively predicted, thereby adjusting the driving strategy, reducing the risk of collision, and improving the safety and reliability of the autonomous driving system.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a point cloud long-term prediction method for autonomous driving scene flow. Background Art

[0002] LiDAR (LiDAR) sensors used in autonomous driving systems use laser beams for scanning and ranging. They emit laser beams and measure the time it takes for them to reflect back, acquiring three-dimensional information about the vehicle's surroundings, including distance, height, and shape. LiDAR offers advantages such as high precision, long-range detection, and all-weather operation. The point cloud generated by LiDAR is crucial for accurate and reliable environmental perception, as well as high-precision positioning and mapping, in autonomous driving systems. LiDAR point cloud-based technologies for understanding the current driving scene, such as object detection, recognition and tracking, semantic segmentation, and positioning and mapping, have made significant progress. Currently, there is an increasing need to perform high-precision simulation and prediction of future driving scenarios based on LiDAR point clouds to enhance the situational awareness and decision-making capabilities of autonomous driving systems.

[0003] Currently, there are two main approaches for predicting future point cloud scene flow for autonomous driving. One is discriminative point cloud scene flow prediction based on 3D convolutional neural networks. This involves converting point cloud data into a two-dimensional range pseudo-image, then using a 3D convolutional neural network to extract features and employ a decoder to predict point cloud scene flow. Representative works include ATPPNet by Kaustab Pal et al., PCPNet by Zhen Luo et al., and TCNet by B. Mersch et al. While these approaches have achieved excellent performance in point cloud scene flow prediction, the conversion of point cloud data into two-dimensional range images can lead to problems such as loss of spatial structure information, boundary discontinuities, noise amplification, and loss of semantic information. In particular, these approaches only achieve short-term prediction of point cloud scene flow up to 0.5 seconds in the future, failing to achieve long-term prediction. This makes it impossible to accurately simulate and predict future driving scenarios over long periods of time. Another type of method is a generative point cloud scene flow prediction method based on a diffusion model. These methods first voxelize the point cloud data and extract features, then convert it into a BEV volume column. These methods generate the point cloud scene flow based on the diffusion model, such as the Copilot4D model proposed by Lunjun Zhang et al. While these generative methods based on the diffusion model achieve long-term prediction of point cloud scene flow, they incorporate future prediction data into the training model during the prediction process, resulting in poor generalization and difficulty in applying the prediction model to unknown datasets. Summary of the Invention

[0004] In view of this, the present invention provides a long-term point cloud prediction method for autonomous driving scene flow to solve the technical problems that existing point cloud prediction methods are prone to lose point cloud spatial structure information, semantic information, boundary continuity and other information, cannot perform long-term prediction of point cloud scene flow, and have poor generalization ability of the prediction model.

[0005] The technical solution adopted by the present invention to solve its technical problem is:

[0006] A method for long-term point cloud prediction of autonomous driving scene flow includes the following steps:

[0007] S1, convert 3D point cloud data into multiple views;

[0008] S2, extract multi-view point cloud features;

[0009] S3, apply vector quantization variational autoencoder to encode multi-view point cloud features;

[0010] S4. Establish a point cloud diffusion model and train the point cloud diffusion model based on the encoded multi-view point cloud features and loss function;

[0011] S5. Perform long-term point cloud prediction based on the trained point cloud diffusion model.

[0012] Preferably, in step S1, converting the 3D point cloud data into multiple views specifically includes the following steps:

[0013] S11, performing coordinate conversion on 3D point cloud data;

[0014] S12, projecting the obtained transformed coordinates onto the bird's-eye view plane to convert the 3D point cloud data into a bird's-eye view;

[0015] S13, projecting the obtained transformed coordinates onto the left view plane to convert the 3D point cloud data into a left view;

[0016] S14. Project the obtained transformed coordinates onto the front view plane to transform the 3D point cloud data into a front view.

[0017] Preferably, in step S2, extracting multi-view point cloud features specifically includes the following steps:

[0018] S21, extracting point cloud features of each view in the multi-view;

[0019] S22. Extracting global features of the point clouds of multiple views based on the point cloud features of each view.

[0020] Preferably, in step S21, extracting point cloud features of each view in the multiple views specifically includes the following steps:

[0021] S211, independently extracting features of the points in each view to obtain a feature vector for each point;

[0022] S212, using a maximum pooling operation to aggregate the feature vector features of each point into a global feature vector of the view;

[0023] S213 , aggregate the global feature vector of each view to obtain a global feature vector of multiple views.

[0024] Preferably, in step S22, extracting global features of point clouds from multiple views specifically includes the following steps:

[0025] S221, preprocessing and dividing the global feature vectors of the multiple views into blocks to obtain embedded feature vectors;

[0026] S222, reshape the embedded feature vector into a feature map, and divide the feature map into a number of non-overlapping windows;

[0027] S223, perform window self-attention calculation;

[0028] S224. Introduce a moving window mechanism and combine it with window self-attention to perform hierarchical feature extraction and layered feature extraction on the feature map to obtain the global features of the point cloud of multiple views.

[0029] Preferably, in step S3, encoding the multi-view point cloud features using a vector quantization variational autoencoder specifically includes the following steps:

[0030] S31, input the multi-view point cloud features into the encoder to obtain a continuous potential vector;

[0031] S32. Replace the continuous latent vector with a discrete vector to obtain the encoded multi-view point cloud features.

[0032] Preferably, in step S4, establishing a point cloud diffusion model and training the point cloud diffusion model based on the encoded multi-view point cloud features and the loss function specifically includes the following steps:

[0033] S41, define the noise matrix;

[0034] S42, based on the noise matrix, gradually adding noise to the encoded multi-view point cloud features in the input data to train the forward diffusion process of the point cloud diffusion model;

[0035] S43, generating raw 3D point cloud data through the U-Net decoder to train the reverse denoising process of the point cloud diffusion model;

[0036] S44. Optimize the reverse denoising process of the point cloud diffusion model based on the loss function to obtain a trained point cloud diffusion model.

[0037] Preferably, the loss function is:

[0038]

[0039] Among them, q(z t-1 |z t ,z0) is the forward diffusion process of the point cloud diffusion model, p θ (z t-1 |z t ) is the reverse denoising process of the point cloud diffusion model; KL represents the Kullback-Leibler divergence, which is used to measure the difference between the forward diffusion process and the reverse denoising process.

[0040] Preferably, in step S5, performing long-term point cloud prediction based on the trained point cloud diffusion model includes: inputting the point cloud of the current frame and the five previous frames into the trained point cloud diffusion model, calling the reverse denoising process of the point cloud diffusion model and applying the time embedding method to perform point cloud prediction for the next 1 second.

[0041] Preferably, the time embedding method includes:

[0042] 1) Calculate embedding frequency

[0043]

[0044] where i∈{0,1,…,half dim -1},

[0045] 2) Time step embedding

[0046] The time steps are embedded using sine and cosine functions:

[0047] emb(T)=[sin(T·freqs),cos(T·freqs)]

[0048] 3) Zero padding

[0049] pad(emb)=[emb(T),0].

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] This method converts 3D point cloud data into multiple views to better capture the spatial structure, semantic information, and boundary continuity of the point cloud data. It then extracts multi-view point cloud features, discretizes them using a vector quantized variational autoencoder, and uses the encoded multi-view point cloud features as input to train a point cloud diffusion model to improve its generalization capabilities. Finally, based on the trained point cloud diffusion model, it implements long-term point cloud prediction, effectively predicting the future long-term motion and change trends of objects in autonomous driving scenarios. This provides early warning for the autonomous driving system, optimizes driving decisions, adjusts driving strategies, reduces collision risks, and improves the safety and reliability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A flow chart of a method of the present invention.

[0053] Figure 2 This is another method flow chart of the present invention.

[0054] Figure 3 This is the converted bird's-eye view.

[0055] Figure 4 It is the point cloud at the current moment.

[0056] Figure 5 To predict the point cloud image 1 second later.

[0057] Figure 6 To predict the point cloud image 3 seconds later. DETAILED DESCRIPTION

[0058] The technical solutions and technical effects of the embodiments of the present invention are further elaborated in detail below with reference to the accompanying drawings of the present invention.

[0059] Please also see Figure 1 and Figure 2 , a point cloud long-term prediction method for autonomous driving scene flow, including the following steps:

[0060] S1, convert 3D point cloud data into multiple views;

[0061] S2, extract multi-view point cloud features;

[0062] S3, apply vector quantization variational autoencoder to encode multi-view point cloud features;

[0063] S4. Establish a point cloud diffusion model and train the point cloud diffusion model based on the encoded multi-view point cloud features and loss function;

[0064] S5. Perform long-term point cloud prediction based on the trained point cloud diffusion model.

[0065] This method converts 3D point cloud data into multiple views to better capture the spatial structure, semantic information, and boundary continuity of the point cloud data. It then extracts multi-view point cloud features, discretizes them using a vector quantized variational autoencoder, and uses the encoded multi-view point cloud features as input to train a point cloud diffusion model to improve its generalization capabilities. Finally, based on the trained point cloud diffusion model, it implements long-term point cloud prediction, effectively predicting the future long-term motion and change trends of objects in autonomous driving scenarios. This provides early warning for the autonomous driving system, optimizes driving decisions, adjusts driving strategies, reduces collision risks, and improves the safety and reliability of the system.

[0066] In some embodiments, the present invention takes the KITTI dataset as an example. For given data, a point cloud diffusion model is established with 0 to 7 as training data and the diffusion model is trained based on a designed loss function. Long-term prediction of point cloud data is performed with 8 to 10 as test data.

[0067] Furthermore, in step S1, the 3D point cloud data is converted into multiple views, which specifically includes the following steps:

[0068] S11, performing coordinate conversion on 3D point cloud data;

[0069] According to the direction angle and coordinate information given in the data, for any point P in the 3D point cloud s =(x s ,y s ,z s ), and coordinate transformation is performed:

[0070]

[0071] In formula (1), T=(T x ,T y ,T z ) is the translation vector, and R is the rotation matrix; the rotation matrix R is constructed by three Euler angles (rotation angles around the x-axis, y-axis, and z-axis) α, β, γ, where:

[0072]

[0073] Then the rotation matrix R can be obtained as:

[0074] R=R x (α)·R y (β)·R z (γ) (5).

[0075] S12, projecting the obtained transformed coordinates onto the bird's-eye view plane to convert the 3D point cloud data into a bird's-eye view;

[0076] Specifically, the obtained transformed coordinates P w =(x w ,y w ,z w ) onto the bird's-eye view plane, that is, onto the XY plane, retaining the Z coordinate to represent the height.

[0077] Assume that the grid resolution is Δx and Δy, the resolution in the height direction is Δz, and the coverage area is [x min ,x max ] and [y min ,y max ], the height range is [z min ,z max ],but:

[0078]

[0079] Where W, H, and D represent the number of grids in the X, Y, and Z directions, respectively.

[0080] Then point P w =(x w ,y w ,z w ) is projected onto the bird's-eye view grid as:

[0081]

[0082] S13, projecting the obtained transformed coordinates onto the left view plane to convert the 3D point cloud data into a left view;

[0083] Specifically, the obtained transformed coordinates P w =(x w ,y w ,z w ) onto the left view plane, that is, onto the YZ plane, retaining the X coordinate to represent the height.

[0084] Assume that the grid resolution is Δy and Δz, the resolution in the left direction is Δx, and the coverage area is [y min ,y max ] and [z min ,z max ], the height range is [x min ,x max ], the grid size is:

[0085]

[0086] Then point P w =(x w ,y w ,zw ) is projected onto the left view grid at:

[0087]

[0088] S14. Project the obtained transformed coordinates onto the front view plane to transform the 3D point cloud data into a front view.

[0089] Specifically, the obtained transformed coordinates P w =(x w ,y w ,z w ) onto the front view plane, that is, onto the XZ plane, retaining the Y coordinate to represent the height.

[0090] Assume that the grid resolution is Δx and Δz, the resolution in the forward direction is Δy, and the coverage area is [x min ,x max ] and [z min ,z max ], the height range is [y min ,y max ], the grid size is:

[0091]

[0092] Then point P w =(x w ,y w ,z w ) is projected onto the left view grid at:

[0093]

[0094] By performing coordinate transformation on the 3D point cloud data according to the given direction angle and coordinate information in the data, and projecting the transformed coordinates, we can obtain a bird's-eye view, a left view, and a front view, so as to better capture the scene's spatial structure information, semantic information, and boundary continuity.

[0095] Further, see Figure 2 Due to the disorder and rotation invariance of point clouds, in order to better perform long-term prediction of point clouds, the spatial structure features of point clouds are extracted. In step S2, multi-view point cloud features are extracted, which specifically includes the following steps:

[0096] S21, extracting point cloud features of each view in the multi-view;

[0097] S22. Extract global features of the point clouds of multiple views based on the point cloud features of each view. Specifically, extract point cloud features from the bird's eye view, front view, and left view converted in step S1, aggregate the extracted features, and extract global features from the aggregated data.

[0098] Furthermore, in step S21, point cloud features of each view in the multi-view are extracted, which specifically includes the following steps:

[0099] S211, independently extracting features of the points in each view to obtain a feature vector for each point;

[0100] S212, using a maximum pooling operation to aggregate the feature vector features of each point into a global feature vector of the view;

[0101] S213, aggregate the global feature vectors of each view to obtain the global feature vectors of multiple views. Specifically, for each point in each view, record it as in Represents point x i The three-dimensional coordinates of . Next, the features of the points in each view are extracted independently, as shown below:

[0102] F(u i )=MLP(x i ) (twenty four)

[0103] In formula (24), F(u i ) is the feature vector, MLP is the fully connected neural network, which is further expanded as follows:

[0104] F(u i )=W k (σ(…(σ(W2(σ(W1x i +b1))+b2)…)+b k )) (25)

[0105] where W1, W2, …, W k are the weight matrices of different layers, b1, b2, ..., b k is the bias vector of each layer and σ is the activation function.

[0106] After obtaining the feature vector of each point, the maximum pooling operation is used to aggregate these features into the global feature vector of the view

[0107]

[0108] Where max represents the element-by-element maximum value operation for each feature dimension.

[0109] After obtaining the global feature vector of each view through the above operation, the global feature vector of each view is again To polymerize

[0110]

[0111] Here γ i represents the global feature vector of each view, Represents the global feature vector of multiple views.

[0112] Furthermore, in step S22, global features of the point clouds of multiple views are extracted, which specifically includes the following steps:

[0113] S221, preprocessing and dividing the global feature vectors of the multiple views into blocks to obtain embedded feature vectors;

[0114] S222, reshape the embedded feature vector into a feature map, and divide the feature map into a number of non-overlapping windows;

[0115] S223, perform window self-attention calculation;

[0116] S224. Introduce a moving window mechanism and combine it with window self-attention to perform hierarchical feature extraction and layered feature extraction on the feature map to obtain the global features of the point cloud of multiple views.

[0117] extract The global features (obtained by aggregating the global feature vectors of each view) are as follows:

[0118] 1) Input feature preprocessing and segmentation

[0119] First, Reshape into a feature map of H×W×C, where H is the height, W is the width, and C is the number of channels; then perform a block operation to divide the feature map into small blocks of size P×P, each with P×P×C values. After the block operation, the input feature map is a combination of a series of small blocks, which can be expressed as N×(P 2 C) shape, where is the total number of blocks. Finally, each small block is flattened into a vector and linearly transformed to obtain an embedded feature vector with dimension d. The input feature is represented as an N×d two-dimensional feature sequence.

[0120] 2) Window division

[0121] The two-dimensional feature sequence obtained above is reshaped into a feature map, that is, its size is restored to The feature map is then divided into several non-overlapping windows, each with a size of M×M, resulting in k windows. The feature size of each window is M×M×d, which can be expressed as:

[0122] {W1,W2,…,W n} (28)

[0123] in

[0124] 3) Window self-attention calculation

[0125] Use the multi-head attention mechanism to capture each window W i Dependencies between locations within a file. The process is as follows:

[0126] First, the window W i The features are linearly transformed to obtain the query matrix Q, key matrix K and value matrix V:

[0127] Q=W i ×W Q , K=W i ×W K , V=W i ×W V (29)

[0128] in d k is the dimension of each attention head.

[0129] The attention weights are then calculated using the scaled dot product formula, which represents the correlation between positions within the window:

[0130]

[0131] Finally, in order to enhance the expressive power of the model, a multi-head attention mechanism is used. Assume that h is used for each head. i The results are stitched together and passed through an output projection matrix

[0132] MultiHead(W i )=Concat(head1,…,head h )W O (31)

[0133] In formula (31), is the linear transformation matrix of the i-th head.

[0134] 4) Moving window mechanism

[0135] In order to better capture a wider range of contextual information, a moving window mechanism is introduced, which includes window translation and cross-window information interaction. That is, before the self-attention operation of each layer of window, the window division is translated to the right or downward by a fixed step size to form a new window division; due to the offset of the window position, part of the area of ​​adjacent windows will be included in the same window, thereby realizing cross-window information interaction.

[0136] 5) Hierarchical feature extraction

[0137] In order to further extract features and increase the nonlinear expression ability of the model, the window self-attention layer and multi-layer perceptron (MLP) are introduced to enable the model to learn more complex features. First, the input features are normalized, and then the window self-attention layer is applied.

[0138]

[0139] In formula (32), W MSA =MultiHead(W i ) represents the window multi-head attention, LN is the normalization operation, is the output of the self-attention layer.

[0140] Then Normalized and passed through a fully connected layer MLP network.

[0141]

[0142] is the output of the MLP network.

[0143] 6) Hierarchical feature extraction

[0144] The multi-scale features are extracted step by step through the hierarchical structure, and the input feature map is convolved and pooled to reduce the feature map resolution by half, that is, the feature map of H×W×C is reduced to The features extracted at each layer contain a gradually larger receptive field, thereby obtaining broader global information from fine-grained local information. After multiple feature extractions, the final output is an H t ,W t ,C t feature map, that is, the global features of the point cloud from multiple views.

[0145] Furthermore, in step S3, a vector quantization variational autoencoder is applied to encode the multi-view point cloud features, which specifically includes the following steps:

[0146] S31, input the multi-view point cloud features into the encoder to obtain a continuous potential vector;

[0147] S32. Replace the continuous latent vector with a discrete vector to obtain the encoded multi-view point cloud features.

[0148] Since point cloud data is discrete, the present invention uses a vector quantized variational autoencoder to discretely encode multi-view point cloud features, thereby obtaining a more compact data representation to improve the long-term prediction of the subsequent point cloud diffusion model. The vector quantized variational autoencoder (VQ-VAE) is used for encoding, as follows:

[0149] The H t ,W t ,C t Feature map, represented as First, the feature map is input into the encoder to obtain an H t ,W t ,d-shaped potential continuous feature vector d is the dimension of the latent space.

[0150] Define the latent embedding space e∈R K×d , where K is the size of the latent embedding space, which contains K embedding vectors, and d is the dimension of each vector. For each latent vector z x (h, w), in order to replace each continuous latent vector with its closest discrete vector, find the closest vector in the latent embedding space

[0151]

[0152] Then the continuous potential vector is replaced by a discrete vector. This process is a quantization process. The quantized potential feature map Each z x (h,w) is replaced by the vector in the closest latent embedding space

[0153]

[0154] It is for H t ,W t ,C t The result of discrete encoding of the input feature map.

[0155] Furthermore, in step S4, a point cloud diffusion model is established and trained based on the encoded multi-view point cloud features and the loss function, specifically including the following steps:

[0156] S41, define the noise matrix;

[0157] S42, based on the noise matrix, gradually adding noise to the encoded multi-view point cloud features in the input data to train the forward diffusion process of the point cloud diffusion model;

[0158] S43, generating raw 3D point cloud data through the U-Net decoder to train the reverse denoising process of the point cloud diffusion model;

[0159] S44. Optimize the reverse denoising process of the point cloud diffusion model based on the loss function to obtain a trained point cloud diffusion model.

[0160] 1) Define the noise matrix

[0161] In order to better perform long-term point cloud prediction, the present invention designs a point cloud diffusion model for point cloud data; at the same time, in order to capture the feature changes in three-dimensional space and improve the quality of point cloud prediction, the present invention uses a tensor-based noise injection method instead of the traditional Gaussian noise method.

[0162] At each time step t, the noise matrix is ​​defined as The spatial feature size is denoted as where Q t Depend on and Together they form.

[0163]

[0164] m, n, p are tensor identifiers used to define Q t Medium 3D subtensor The index of , i, j, k is the tensor subscript, and the value range is as follows:

[0165]

[0166] When α t =1-γ t , represents each labeled discretized feature vector have The probability of remaining unchanged, and the probability of transitioning to other K-1 geometric categories are β t , where α t and β t Both are related to the time step t and the total time step t.

[0167] 2) Forward diffusion process of point cloud diffusion model

[0168] The forward diffusion process of the point cloud diffusion model is Noise is gradually added to the three-dimensional geometric input, that is, the state z at each time step t t From the state z of the previous step t-1 t-1 The joint probability distribution of its forward diffusion is:

[0169]

[0170] q(z0) is the prior distribution of the initial state, and its single-step noise addition formula is:

[0171]

[0172] where Q t is the noise matrix.

[0173] And for the conditional distribution from the initial state z0 to any time step t:

[0174] q(z t |z0)=Q1·Q2·…·Q t ·z0 (41)

[0175] where Q1, Q2, ..., Q t is the noise matrix at each time step.

[0176] 3) Reverse denoising process of point cloud diffusion model

[0177] The purpose of the reverse denoising process of the point cloud diffusion model is to transform the noise state z t Restore to the original state z0. Noise is added to the data during the forward diffusion process of the point cloud diffusion model. In the reverse denoising process of the point cloud diffusion model, a denoiser is required to gradually restore the original data in the reverse process. The present invention uses a U-Net network to predict the change from the current discretized data state to a state closer to the original point cloud from each diffusion step, thereby gradually denoising and reconstructing the point cloud. The conditional probability distribution of reverse denoising can be obtained by Bayes' theorem, and p is defined as θ (z t-1 |z t ) is from z t to z t-1 The conditional probability of , so we have:

[0178] p θ (z t-1 =i|z t =j)∝q(z t =j|z t-1 =i)p θ (z t-1 =i) (42)

[0179] Further expanded to:

[0180]

[0181] where q(z t |z t-1 ) is the transition probability of the forward process, p θ (z t-1 ) is the prior distribution obtained by the reverse process, and the denominator is a normalization constant to ensure that the sum of the probabilities is 1.

[0182] The joint probability distribution of inverse denoising is:

[0183]

[0184] where p(z T ) represents the final noise state.

[0185] The U-Net network passes the conditional probability p θ (z t-1 |z t ) Learn from z t to z t-1 The original 3D point cloud data is gradually restored through mapping.

[0186] 4) Loss function of point cloud diffusion model

[0187] To train the reverse denoising model p θ (z t-1 |z t ), using the Evidence Lower Bound loss function (ELBO), which aims to maximize the log-likelihood of the data,

[0188]

[0189] KL stands for Kullback-Leibler divergence, which measures the difference between the forward diffusion process and the reverse denoising process. By minimizing this loss function, the present invention can optimize the reverse denoising model, gradually recovering the original data from the noise at each time step.

[0190] 5) Training the diffusion model based on the designed loss function

[0191] The model training process involves encoding the extracted point cloud features using a vector quantized variational autoencoder (VVAE). Tensor-based noise is then added to the forward pass of the point cloud diffusion model. The VVAE-encoded features are then predicted using a backward diffusion model. Finally, the 3D point cloud is restored using a U-Net decoder. The point cloud diffusion loss function from sub-step 4 is used as the loss function. Weight information during training is saved in real time, and the optimal weights are selected for subsequent point cloud prediction.

[0192] Furthermore, in step S5, long-term point cloud prediction is performed based on the trained point cloud diffusion model, including: inputting the point cloud of the current frame and the previous five frames into the trained point cloud diffusion model, calling the reverse denoising process of the point cloud diffusion model and applying the time embedding method to perform point cloud prediction for the next 1 second.

[0193] Specifically, after establishing and training the point cloud diffusion model, the present invention uses 10Hz lidar point cloud data for long-term point cloud prediction. Initially, the optimal weights of the model saved above are used to input the point cloud data of frames t-5, t-4, t-3, t-2, t-1, and t into the point cloud test model to predict the point cloud for the next 1 second. The chamfer distance is used to evaluate the quality of the point cloud prediction. The above time steps are collectively referred to as T. Time embedding is used to map the time steps to a high-dimensional embedding space, as follows:

[0194] 1) Calculate embedding frequency

[0195]

[0196] where i∈{0,1,…,half dim -1},

[0197] 2) Time step embedding

[0198] The time steps are embedded using sine and cosine functions:

[0199] emb(T)=[sin(T·freqs),cos(T·freqs)] (47)

[0200] 3) Zero padding

[0201] pad(emb)=[emb(T),0] (48).

[0202] Through time embedding, the temporal relationship between the current frame and its previous five frames can be better represented, thereby helping the model better capture temporal dependencies and sequence characteristics and improving the model's prediction effect.

[0203] As an example, the following is a specific operational flow of the point cloud long-term prediction method for autonomous driving scene flow of the present invention. The present invention takes the KITTI dataset as an example. The KITTI dataset includes multiple groups of point cloud data. For given data, 0 to 7 groups are used as training data to establish a point cloud diffusion model and the diffusion model is trained based on the designed loss function. 8 to 10 groups are used as test data to perform long-term prediction of point cloud data. First, step S1 is performed: converting 3D point cloud data into multiple views. According to the direction angle and coordinate information given in the data, the 3D point cloud data is coordinate transformed, and the transformed coordinates are projected to obtain a bird's-eye view, a left view, and a front view. Among them, the visualization after the KITTI dataset is converted to a bird's-eye view is as follows Figure 3 shown.

[0204] For the converted view, the spatial structure features of the point cloud are extracted. The specific network architecture of step S2 for extracting multi-view point cloud features is as follows:

[0205] Sub-step (1): Calculate the spatial transformation matrix of the point cloud

[0206] 1) Pass the input data x to the first convolutional layer, which has 3 input channels, 64 output channels, and a convolution kernel size of 1. Batch normalization is used on the convolved data, and the ReLU activation function is applied to the normalized data. The batch_size is set to 32. The input data x is one of the multiple views.

[0207] 2) Pass the output of step (1) to the second convolutional layer, which has 64 input channels, 128 output channels, and a convolution kernel size of 1. Batch normalization is used on the convolved data, and the ReLU activation function is applied to the normalized data.

[0208] 3) Pass the output of step (2) to the third convolutional layer, which has 128 input channels, 1024 output channels, a convolution kernel size of 1, batch normalization of the convolved data, and a ReLU activation function applied to the normalized data.

[0209] 4) Perform the maximum pooling operation on the channel dimension of the output x of step (3), retaining the maximum value while keeping the dimension unchanged.

[0210] 5) Pass the output of step (4) to the first fully connected layer with a size of (1024,512), then apply batch normalization and apply the ReLU activation function to the normalized data.

[0211] 6) Pass the output of step (5) to the second fully connected layer with a size of (512, 256), then apply batch normalization and apply the ReLU activation function to the normalized data.

[0212] 7) Pass the output of step (6) to the third fully connected layer of size (256, 9), add the output to the unit matrix of dimension (32, 9), and adjust the matrix shape to (32, 3, 3).

[0213] Sub-step (2): Extract point cloud features of each view

[0214] 1) Apply the spatial transformation matrix calculated in substep (1) to each point, transforming the data x to a dimension of (32, 3, n_pts), where n_pts is the number of points, which is a dynamic transformation. Then use the first convolution layer with 3 input channels, 64 output channels, and a convolution kernel size of 1. Batch normalize the convolved data, apply the ReLU activation function to the normalized data, and save the current feature map as a local feature.

[0215] 2) Pass the output of step (1) to the second convolutional layer, which has 64 input channels, 128 output channels, and a convolution kernel size of 1. Batch normalization is used on the convolved data, and the ReLU activation function is applied to the normalized data.

[0216] 3) Pass the output of step (2) to the third convolutional layer, which has 128 input channels, 1024 output channels, a convolution kernel size of 1, batch normalization of the convolved data, and a ReLU activation function applied to the normalized data.

[0217] 4) Perform a global maximum pooling operation on the output of step (3) in the channel dimension, and finally obtain a tensor of (32, 1, 1024, 1024), which is the extracted global feature vector of the view.

[0218] Repeat sub-steps (1) and (2) to extract the point cloud features of the bird's-eye view, left view, and front view respectively, and aggregate them into a tensor. The shape of the aggregated tensor is (32, 1, 1024, 1024), which is the global feature vector of the multi-view.

[0219] Sub-step (3): extracting the global features of the point cloud after aggregation of feature vectors

[0220] 1) Input feature processing and segmentation

[0221] The tensor x obtained in sub-step (2) is reshaped into (1, 1024, 1024) and passed to the convolutional layer for block operation. The block size is 8. The number of input channels of this convolutional layer is 512, the embedding dimension is 96, the stride is 8, and the convolution kernel size is 8. Each block is converted from the original channel space to the embedding space through this convolutional layer, and the final embedding vector is (8, 8, 96).

[0222] 2) Window division and self-attention calculation

[0223] The vector obtained in step (1) is reshaped into a feature map of (128, 128, 96), and then the feature map is divided into 44 windows, resulting in a total of 32 windows, and the batch_size is set to 16.

[0224] Before performing windowed self-attention calculations, the input x is resized to (512, 16, 96). After passing through a linear layer, it is rearranged into a multi-head attention format and the tensor is split into query, key, and value components. The query vector is scaled, and the dot product and transposition of the scaled query and key vectors are performed. The result is the attention weight matrix. The relative position deviation is obtained based on the relative position index and added to the attention weight matrix. The final output is (512, 16, 96) after passing through a linear layer again.

[0225] 3) Hierarchical feature extraction, layered feature extraction, and moving window mechanism

[0226] The input feature map is normalized and padded, and then the self-attention mechanism in step (2) is used to remove the previous feature map padding, apply residual connection and normalization, and finally obtain a (32,32,96) feature map.

[0227] In order to increase the computation speed, reduce the computation cost, and improve the efficiency and effect of the subsequent point cloud diffusion model, a vector quantization variational autoencoder is used for discrete encoding to obtain a more compact data representation. In step S3, the vector quantization variational autoencoder is used to encode the multi-view point cloud features as follows:

[0228] 1) Pass the input data (32,32,96) to the first convolutional layer with 96 input channels, 128 output channels, a kernel size of 4, a stride of 2, and an edge padding of 1. Apply the ReLU activation function to the convolved data.

[0229] 2) Pass the output of step (1) to the second convolutional layer, which has 128 input channels, 256 output channels, a kernel size of 4, a stride of 2, and an edge padding of 1. Batch normalization is used on the convolved data, and the ReLU activation function is applied to the normalized data.

[0230] 3) Pass the output of step (2) to the third convolutional layer, which has 256 input channels, 256 output channels, a kernel size of 3, a stride of 1, and an edge padding of 1. Batch normalization is used on the convolved data, and the ReLU activation function is applied to the normalized data.

[0231] 4) Use residual blocks and residual stacks to solve the existing gradient vanishing problem and improve model performance.

[0232] The present invention designs a point cloud diffusion model for point cloud data to achieve the purpose of long-term point cloud prediction. At the same time, in order to improve the quality of long-term point cloud prediction, the present invention uses a tensor-based noise injection method instead of the traditional Gaussian noise method. In step S4, a point cloud diffusion model is established and trained based on the encoded multi-view point cloud features and loss function, as follows:

[0233] Sub-step (1), define the noise matrix

[0234] In order to capture the feature changes in three-dimensional space and improve the quality of point cloud prediction, the present invention uses a tensor-based noise injection method instead of the traditional Gaussian noise method.

[0235] At each time step t, the noise matrix is ​​defined as The spatial feature size is denoted as where Q t Depend on and Together they form.

[0236]

[0237] m, n, p are tensor identifiers used to define Q t Medium 3D subtensor The index of , i, j, k is the tensor subscript, and the value range is as follows:

[0238]

[0239] When α t =1-γ t , represents each labeled discretized feature vector have The probability of remaining unchanged, and the probability of transitioning to other K-1 geometric categories are β t , where α t and β t Both are related to the time step t and the total time step t.

[0240] Sub-step (2), point cloud diffusion model forward process

[0241] Since the noise used in the present invention is based on the tensor-based noise injection method, the β in sub-step (1) is defined here. t =0.6,α t =0.4, r t =0.4, from which the tensor noise matrix Q is calculated t , posterior variance, posterior variance coefficient, etc., and gradually add noise to the input data to the point cloud data encoded by the vector quantized variational autoencoder, and perform the forward process output without a classifier based on the conditional and unconditional input calculations.

[0242] Sub-step (3), reverse process of point cloud diffusion model

[0243] The reverse process of the point cloud diffusion model is used to gradually recover the original data from the noisy data and is the core of the entire diffusion model. First, a time step tensor is created and set to the current time step. The multi-step diffusion process is controlled by the diffusion model multi-step solver. The entire diffusion process is continuously optimized using the evidence lower bound, gradually recovering high-quality data from the noise. At each time step, the current noise is calculated using the classifier-free forward process, where the classifier-free forward process calls the Unet network to generate the original 3D point cloud.

[0244] Sub-step (4), loss function of point cloud diffusion model

[0245] To train the reverse denoising model p θ (zt-1 |z t ), using the Evidence Lower Bound loss function (ELBO), which aims to maximize the log-likelihood of the data,

[0246]

[0247] KL stands for Kullback-Leibler divergence, which measures the difference between the forward diffusion process and the reverse denoising process. By minimizing this loss function, the present invention can optimize the reverse denoising model, gradually recovering the original data from the noise at each time step.

[0248] Sub-step (5): training the diffusion model based on the designed loss function

[0249] The model training process is to first encode the extracted point cloud features through the vector quantization variational autoencoder for the input data, then add tensor-based noise through the forward process of the point cloud diffusion model and predict the features encoded by the vector quantization variational autoencoder through the backward diffusion model, and finally restore the 3D point cloud through the U-Net decoder. The loss function uses the point cloud diffusion loss function of sub-step (4). The weight information during the training process is saved in real time, and the best weight is selected for subsequent point cloud prediction.

[0250] In step S5, long-term point cloud prediction is performed based on the trained point cloud diffusion model, as follows:

[0251] The point cloud data of the current frame and the five previous frames are input into the trained point cloud diffusion model. By repeating the point cloud and adding random noise, a partial point cloud is generated and converted into a sparse tensor form. Then, the point cloud diffusion model is reversed and the time embedding method is applied to generate the point cloud of the next 1 second and 3 seconds. Among them, the original point cloud, that is, the point cloud of the current frame is as follows Figure 4 As shown, the point cloud is predicted to be as follows after 1 second Figure 5 As shown, the point cloud predicted 3 seconds later is as follows Figure 6 shown.

[0252] The long-term point cloud prediction method for autonomous driving scene flow of the present invention can achieve the purpose of long-term point cloud prediction without losing the spatial structure information, semantic information, boundary continuity, etc. of the point cloud, and has good generalization ability, so that the autonomous driving system can know the possible movement trajectory and change trend of the object in advance, thereby adjusting the driving strategy of the vehicle to avoid collision.

[0253] The above disclosure is only a preferred embodiment of the present invention, and it is certainly not intended to limit the scope of the present invention. A person skilled in the art can understand that all or part of the processes of the above embodiment and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A point cloud long-term prediction method for autonomous driving scene flow, characterized by: The following steps are involved: S1, convert 3D point cloud data into multiple views; S2, extract multi-view point cloud features; S3, apply vector quantization variational autoencoder to encode multi-view point cloud features; S4. Establish a point cloud diffusion model and train the point cloud diffusion model based on the encoded multi-view point cloud features and loss function; S5. Perform long-term point cloud prediction based on the trained point cloud diffusion model; In step S2, the extraction of multi-view point cloud features specifically includes the following steps: S21, extracting point cloud features of each view in the multi-view; S22, extracting global features of the point clouds of multiple views based on the point cloud features of each view; In step S21, extracting point cloud features of each view in the multi-view specifically includes the following steps: S211, independently extracting features of the points in each view to obtain a feature vector for each point; S212, using a maximum pooling operation to aggregate the feature vector features of each point into a global feature vector of the view; S213, aggregating the global feature vector of each view to obtain a global feature vector of multiple views; In step S22, extracting global features of point clouds from multiple views specifically includes the following steps: S221, preprocessing and dividing the global feature vectors of the multiple views into blocks to obtain embedded feature vectors; S222, reshape the embedded feature vector into a feature map, and divide the feature map into a number of non-overlapping windows; S223, perform window self-attention calculation; S224. Introduce a moving window mechanism and combine it with window self-attention to perform hierarchical feature extraction and layered feature extraction on the feature map to obtain the global features of the point cloud of multiple views.

2. The point cloud long-term prediction method for autonomous driving scene flow according to claim 1 is characterized in that: In step S1, the 3D point cloud data is converted into multiple views, specifically including the following steps: S11, performing coordinate conversion on 3D point cloud data; S12, projecting the obtained transformed coordinates onto the bird's-eye view plane to convert the 3D point cloud data into a bird's-eye view; S13, projecting the obtained transformed coordinates onto the left view plane to convert the 3D point cloud data into a left view; S14. Project the obtained transformed coordinates onto the front view plane to transform the 3D point cloud data into a front view.

3. The point cloud long-term prediction method for autonomous driving scene flow according to claim 1, characterized in that: In step S3, the application of the vector quantization variational autoencoder to encode the multi-view point cloud features specifically includes the following steps: S31, input the multi-view point cloud features into the encoder to obtain a continuous potential vector; S32. Replace the continuous latent vector with a discrete vector to obtain the encoded multi-view point cloud features.

4. The point cloud long-term prediction method for autonomous driving scene flow according to claim 1, characterized in that: In step S4, the point cloud diffusion model is established and trained based on the encoded multi-view point cloud features and the loss function, specifically including the following steps: S41, define the noise matrix; S42, based on the noise matrix, gradually adding noise to the encoded multi-view point cloud features in the input data to train the forward diffusion process of the point cloud diffusion model; S43, generating raw 3D point cloud data through the U-Net decoder to train the reverse denoising process of the point cloud diffusion model; S44. Optimize the reverse denoising process of the point cloud diffusion model based on the loss function to obtain a trained point cloud diffusion model.

5. The point cloud long-term prediction method for autonomous driving scene flow according to claim 4, characterized in that: The loss function is: Among them, q(z t-1 |z t ,z0) is the forward diffusion process of the point cloud diffusion model, p θ (z t-1 |z t ) is the reverse denoising process of the point cloud diffusion model; KL represents the Kullback-Leibler divergence, which is used to measure the difference between the forward diffusion process and the reverse denoising process.

6. The point cloud long-term prediction method for autonomous driving scene flow according to claim 5, characterized in that: In step S5, long-term point cloud prediction based on the trained point cloud diffusion model includes: inputting the point cloud of the current frame and the previous five frames into the trained point cloud diffusion model, calling the reverse denoising process of the point cloud diffusion model and applying the time embedding method to predict the point cloud for the next 1 second.

7. The point cloud long-term prediction method for autonomous driving scene flow according to claim 6, characterized in that: The time embedding method includes: 1) Calculate embedding frequency where i∈{0,1,…,half_dim-1}, 2) Time step embedding The time steps are embedded using sine and cosine functions: emb(T)=[sin(T·freqs),cos(T·freqs)] 3) Zero padding pad(emb)=[emb(T),0].

Images