Point cloud long-time prediction method for automatic driving scene flow

Abstract

The invention provides an automatic driving scene flow-oriented point cloud long-time prediction method in the technical field of automatic driving. Comprising the following steps: S1, converting 3D point cloud data into multiple views; s2, extracting multi-view point cloud features; s3, encoding the multi-view point cloud features by using a vector quantization variational auto-encoder; s4, establishing a point cloud diffusion model, and training the point cloud diffusion model based on the encoded multi-view point cloud features and the loss function; and S5, performing point cloud long-time prediction based on the trained point cloud diffusion model. Through the method provided by the invention, the purpose of point cloud long-time prediction can be realized on the basis of not losing point cloud space structure information, semantic information, boundary continuity and the like, the generalization ability is good, and the future long-time motion trail and change trend of the object in the automatic driving scene can be effectively predicted, so that the driving strategy is adjusted, the collision risk is reduced, and the driving experience is improved. And the safety and the reliability of the automatic driving system are improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a point cloud long-term prediction method for autonomous driving scene flow. Background Art

[0002] The on-board LiDAR in the autonomous driving system is a sensor that uses laser beams for scanning and ranging. It obtains three-dimensional information about the vehicle's surroundings, including distance, height, shape, etc., by emitting laser beams and measuring the time it takes for them to reflect back. LiDAR has the advantages of high precision, long-distance detection, and all-weather operation. The point cloud generated by LiDAR is crucial for the accurate and reliable environmental perception and high-precision positioning and mapping of the autonomous driving system. Target detection, recognition and tracking, semantic segmentation, positioning and mapping based on LiDAR point clouds have made great progress in understanding the driving scene at the current moment. At present, it is more urgent to perform high-precision simulation and prediction of driving scenes in the future based on LiDAR point clouds to enhance the situational awareness and decision-making capabilities of the autonomous driving system.

[0003] At present, there are two main methods for predicting point cloud scene flow at future moments for autonomous driving. One is the discriminative point cloud scene flow prediction based on 3D convolutional neural network, which converts point cloud data into two-dimensional distance pseudo-images, then uses 3D convolutional neural network to extract features and uses decoders to realize point cloud scene flow prediction. The main representative works include ATPPNet by Kaustab Pal et al., PCPNet by Zhen Luo et al., TCNet by B. Mersch et al., etc. Although this type of work has achieved excellent performance in the accuracy of point cloud scene flow prediction, when converting point cloud data into two-dimensional distance images, it will lead to problems such as loss of spatial structure information of point cloud data, boundary discontinuity, noise amplification, and loss of semantic information. In particular, it only realizes short-term prediction of point cloud scene flow for 0.5 seconds in the future, and does not realize long-term prediction of point cloud scene flow, so it is impossible to simulate and predict future driving scenes with high precision for a long period of time. The other is a generative point cloud scene flow prediction method based on a diffusion model, which first voxelizes the point cloud data and extracts features, then converts it into a BEV body column, and generates a point cloud scene flow based on a diffusion model, such as the Copilot4D model proposed by Lunjun Zhang et al. Although the generative method based on the diffusion model achieves the effect of long-term prediction of point cloud scene flow, the future prediction data is also added to the training model during the prediction process, resulting in poor generalization ability of the prediction model and difficulty in applying it to unknown data sets. Summary of the invention

[0004] In view of this, the present invention provides a point cloud long-term prediction method for autonomous driving scene flow, so as to solve the technical problems that the existing point cloud prediction method is prone to lose point cloud spatial structure information, semantic information, boundary continuity and other information, is unable to perform long-term prediction of point cloud scene flow, and the generalization ability of the prediction model is poor.

[0005] The technical solution adopted by the present invention to solve its technical problem is:

[0006] A point cloud long-term prediction method for autonomous driving scene flow includes the following steps:

[0007] S1, converting 3D point cloud data into multiple views;

[0008] S2, extract multi-view point cloud features;

[0009] S3, apply vector quantization variational autoencoder to encode multi-view point cloud features;

[0010] S4, establishing a point cloud diffusion model, and training the point cloud diffusion model based on the encoded multi-view point cloud features and loss function;

[0011] S5. Perform long-term point cloud prediction based on the trained point cloud diffusion model.

[0012] Preferably, in step S1, converting the 3D point cloud data into multiple views specifically comprises the following steps:

[0013] S11, coordinate conversion of 3D point cloud data;

[0014] S12, projecting the obtained transformed coordinates onto the bird's-eye view plane to convert the 3D point cloud data into a bird's-eye view;

[0015] S13, projecting the obtained transformed coordinates onto the left view plane to convert the 3D point cloud data into a left view;

[0016] S14. Project the obtained transformed coordinates onto the front view plane to convert the 3D point cloud data into a front view.

[0017] Preferably, in step S2, extracting multi-view point cloud features specifically includes the following steps:

[0018] S21, extracting point cloud features of each view in the multi-view;

[0019] S22. Extracting global features of point clouds of multiple views based on point cloud features of each view.

[0020] Preferably, in step S21, extracting point cloud features of each view in the multiple views specifically includes the following steps:

[0021] S211, independently extracting features of the points in each view to obtain a feature vector of each point;

[0022] S212, using a maximum pooling operation to aggregate the feature vector features of each point into a global feature vector of the view;

[0023] S213: Aggregate the global feature vector of each view to obtain the global feature vector of multiple views.

[0024] Preferably, in step S22, extracting the point cloud global features of multiple views specifically includes the following steps:

[0025] S221, preprocessing and dividing the global feature vectors of multiple views to obtain embedded feature vectors;

[0026] S222, reshape the embedded feature vector into a feature map, and divide the feature map into a plurality of non-overlapping windows;

[0027] S223, performing window self-attention calculation;

[0028] S224. Introduce a moving window mechanism, and combine window self-attention to perform hierarchical feature extraction and layered feature extraction on the feature map to obtain the global features of the point cloud of multiple views.

[0029] Preferably, in step S3, the applying of a vector quantization variational autoencoder to encode the multi-view point cloud features specifically comprises the following steps:

[0030] S31, inputting the multi-view point cloud features into the encoder to obtain a continuous potential vector;

[0031] S32. Replace the continuous latent vector with a discrete vector to obtain the encoded multi-view point cloud features.

[0032] Preferably, in step S4, establishing a point cloud diffusion model and training the point cloud diffusion model based on the encoded multi-view point cloud features and the loss function specifically includes the following steps:

[0033] S41, define a noise matrix;

[0034] S42, based on the noise matrix, gradually adding noise to the encoded multi-view point cloud features in the input data to train the forward diffusion process of the point cloud diffusion model;

[0035] S43, generating raw 3D point cloud data through the U-Net decoder to train the reverse denoising process of the point cloud diffusion model;

[0036] S44. Based on the loss function, the reverse denoising process of the point cloud diffusion model is optimized to obtain a trained point cloud diffusion model.

[0037] Preferably, the loss function is:

[0038]

[0039] Among them, q(z t-1 |z t ,z0) is the forward diffusion process of the point cloud diffusion model, p θ (z t-1 |z t ) is the reverse denoising process of the point cloud diffusion model; KL represents the Kullback-Leibler divergence, which is used to measure the difference between the forward diffusion process and the reverse denoising process.

[0040] Preferably, in step S5, performing long-term prediction of point clouds based on the trained point cloud diffusion model includes: inputting the point clouds of the current frame and the five previous frames into the trained point cloud diffusion model, calling the reverse denoising process of the point cloud diffusion model and applying the time embedding method to perform point cloud prediction for the next 1 second.

[0041] Preferably, the time embedding method comprises:

[0042] 1) Calculate embedding frequency

[0043]

[0044] where i∈{0,1,…,half dim -1},

[0045] 2) Time step embedding

[0046] The time steps are embedded using sine and cosine functions:

[0047] emb(T)=[sin(T·freqs),cos(T·freqs)]

[0048] 3) Zero padding

[0049] pad(emb)=[emb(T),0].

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] The present invention converts 3D point cloud data into multi-views to better capture the spatial structure information, semantic information and boundary continuity of point cloud data; then extracts multi-view point cloud features, and discretely encodes the extracted multi-view point cloud features through vector quantization variational autoencoder, and uses the encoded multi-view point cloud features as the input of the point cloud diffusion model to train the point cloud diffusion model to improve the generalization ability of the point cloud diffusion model. Finally, based on the trained point cloud diffusion model, the point cloud long-term prediction is realized, thereby effectively predicting the future long-term movement and change trend of objects in the autonomous driving scene, providing early warning for the autonomous driving system, optimizing driving decisions, adjusting driving strategies, reducing collision risks, and improving the safety and reliability of the autonomous driving system. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A flow chart of a method of the present invention.

[0053] Figure 2 This is another method flow chart of the present invention.

[0054] Figure 3 This is the converted bird's-eye view.

[0055] Figure 4 It is the point cloud at the current moment.

[0056] Figure 5 To predict the point cloud image 1 second later.

[0057] Figure 6 To predict the point cloud image 3 seconds later. DETAILED DESCRIPTION

[0058] The technical solutions and technical effects of the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings of the present invention.

[0059] Please also see Figure 1 and Figure 2 , a point cloud long-term prediction method for autonomous driving scene flow, comprising the following steps:

[0060] S1, converting 3D point cloud data into multiple views;

[0061] S2, extract multi-view point cloud features;

[0062] S3, apply vector quantization variational autoencoder to encode multi-view point cloud features;

[0063] S4, establishing a point cloud diffusion model, and training the point cloud diffusion model based on the encoded multi-view point cloud features and loss function;

[0064] S5. Perform long-term point cloud prediction based on the trained point cloud diffusion model.

[0065] The present invention converts 3D point cloud data into multi-views to better capture the spatial structure information, semantic information and boundary continuity of point cloud data; then extracts multi-view point cloud features, and discretely encodes the extracted multi-view point cloud features through vector quantization variational autoencoder, and uses the encoded multi-view point cloud features as the input of the point cloud diffusion model to train the point cloud diffusion model to improve the generalization ability of the point cloud diffusion model. Finally, based on the trained point cloud diffusion model, the point cloud long-term prediction is realized, thereby effectively predicting the future long-term movement and change trend of objects in the autonomous driving scene, providing early warning for the autonomous driving system, optimizing driving decisions, adjusting driving strategies, reducing collision risks, and improving the safety and reliability of the autonomous driving system.

[0066] In some embodiments, the present invention takes the KITTI dataset as an example. For given data, a point cloud diffusion model is established with 0 to 7 as training data and the diffusion model is trained based on a designed loss function. Long-term prediction of point cloud data is performed with 8 to 10 as test data.

[0067] Furthermore, in step S1, the 3D point cloud data is converted into multiple views, which specifically includes the following steps:

[0068] S11, coordinate conversion of 3D point cloud data;

[0069] According to the direction angle and coordinate information given in the data, for any point P in the 3D point cloud s =(x s ,y s ,z s ), and coordinate transformation is performed:

[0070]

[0071] In formula (1), T = (T x ,T y ,T z ) is the translation vector, R is the rotation matrix; the rotation matrix R is constructed by three Euler angles (rotation angles around the x-axis, y-axis and z-axis) α, β, γ, where:

[0072]

[0073] Then the rotation matrix R can be obtained as:

[0074] R=R x (α)·R y (β)·R z (γ) (5).

[0075] S12, projecting the obtained transformed coordinates onto the bird's-eye view plane to convert the 3D point cloud data into a bird's-eye view;

[0076] Specifically, the obtained transformed coordinates P w =(x w ,y w ,z w ) onto the bird's-eye view plane, that is, onto the XY plane, retaining the Z coordinate to represent the height.

[0077] Assume that the grid resolution is Δx and Δy, the resolution in the height direction is Δz, and the coverage area is [x min ,x max ] and [y min ,y max ], the height range is [z min ,z max ],but:

[0078]

[0079] Where W, H, and D represent the number of grids in the X, Y, and Z directions, respectively.

[0080] Then point P w =(x w ,y w ,z w ) is projected onto the bird’s-eye view grid as:

[0081]

[0082] S13, projecting the obtained transformed coordinates onto the left view plane to convert the 3D point cloud data into a left view;

[0083] Specifically, the obtained transformed coordinates P w =(x w ,y w ,z w ) onto the left view plane, that is, onto the YZ plane, retaining the X coordinate to represent the height.

[0084] Assume that the grid resolution is Δy and Δz, the resolution in the left direction is Δx, and the coverage area is [y min ,y max ] and [z min ,z max ], the height range is [x min ,x max ], the grid size is:

[0085]

[0086] Then point P w =(x w ,y w ,zw ) is projected onto the left view grid at:

[0087]

[0088] S14. Project the obtained transformed coordinates onto the front view plane to convert the 3D point cloud data into a front view.

[0089] Specifically, the obtained transformed coordinates P w =(x w ,y w ,z w ) onto the front view plane, that is, onto the XZ plane, retaining the Y coordinate to represent the height.

[0090] Assume that the grid resolution is Δx and Δz, the resolution in the forward direction is Δy, and the coverage area is [x min ,x max ] and [z min ,z max ], the height range is [y min ,y max ], the grid size is:

[0091]

[0092] Then point P w =(x w ,y w ,z w ) is projected onto the left view grid at:

[0093]

[0094] By performing coordinate transformation on the 3D point cloud data according to the direction angle and coordinate information given in the data, and projecting the transformed coordinates, we can obtain a bird's-eye view, a left view, and a front view, so as to better capture the scene's spatial structure information, semantic information, and boundary continuity.

[0095] For further information, see Figure 2 Due to the disorder and rotation invariance of the point cloud, in order to better perform long-term prediction of the point cloud, the spatial structure features of the point cloud are extracted. In step S2, the multi-view point cloud features are extracted, which specifically includes the following steps:

[0096] S21, extracting point cloud features of each view in the multi-view;

[0097] S22, extracting global features of point clouds of multiple views based on point cloud features of each view. Specifically, extracting point cloud features from the bird's eye view, front view, and left view converted in step S1, respectively, and then aggregating the extracted features and extracting global features of the aggregated data.

[0098] Furthermore, in step S21, point cloud features of each view in the multiple views are extracted, which specifically includes the following steps:

[0099] S211, independently extracting features of the points in each view to obtain a feature vector of each point;

[0100] S212, using a maximum pooling operation to aggregate the feature vector features of each point into a global feature vector of the view;

[0101] S213, aggregate the global feature vector of each view to obtain the global feature vector of multiple views. Specifically, for each point in each view, record it as in Represents point x i The three-dimensional coordinates of . Next, the features of the points in each view are extracted independently, as shown below:

[0102] F(u i )=MLP(x i ) (twenty four)

[0103] In formula (24), F(u i ) is the feature vector, MLP is the fully connected neural network, which is further expanded as follows:

[0104] F(u i )=W k (σ(…(σ(W2(σ(W1x i +b1))+b2)…)+b k )) (25)

[0105] where W1, W2, …, W k are the weight matrices of different layers, b1,b2,…,b k is the bias vector of each layer and σ is the activation function.

[0106] After obtaining the feature vector of each point, the maximum pooling operation is used to aggregate these features into the global feature vector of the view.

[0107]

[0108] Where max represents the element-by-element maximum value operation for each feature dimension.

[0109] After obtaining the global feature vector of each view through the above operation, the global feature vector of each view is To polymerize

[0110]

[0111] Here is the gamma i represents the global feature vector for each view, Represents the global feature vector of multiple views.

[0112] Furthermore, in step S22, the global features of the point clouds of multiple views are extracted, which specifically includes the following steps:

[0113] S221, preprocessing and dividing the global feature vectors of multiple views to obtain embedded feature vectors;

[0114] S222, reshape the embedded feature vector into a feature map, and divide the feature map into a plurality of non-overlapping windows;

[0115] S223, performing window self-attention calculation;

[0116] S224. Introduce a moving window mechanism, and combine window self-attention to perform hierarchical feature extraction and layered feature extraction on the feature map to obtain the global features of the point cloud of multiple views.

[0117] extract The global features (obtained by aggregating the global feature vectors of each view) are as follows:

[0118] 1) Input feature preprocessing and segmentation

[0119] First, Reshape into a feature map of H×W×C, where H is the height, W is the width, and C is the number of channels; then perform a block operation to divide the feature map into small blocks of size P×P, each with P×P×C values. After the block operation, the input feature map is a combination of a series of small blocks, which can be expressed as N×(P 2 C) shape, where is the total number of blocks. Finally, each small block is flattened into a vector, and an embedded feature vector with a dimension of d is obtained through linear transformation. The input feature is represented as an N×d two-dimensional feature sequence.

[0120] 2) Window division

[0121] The two-dimensional feature sequence obtained above is reshaped into a feature map, that is, its size is restored to Then divide the feature map into several non-overlapping windows, each with a size of M×M, and get k windows. The feature size of each window is M×M×d, which can be expressed as:

[0122] {W1,W2,…,W n} (28)

[0123] in

[0124] 3) Window self-attention calculation

[0125] Use the multi-head attention mechanism to capture each window W i Dependencies between locations within a file. The process is as follows:

[0126] First, the window W i The features are linearly transformed to obtain the query matrix Q, key matrix K and value matrix V:

[0127] Q=W i ×W Q , K=W i ×W K , V=W i ×W V (29)

[0128] in d k is the dimension of each attention head.

[0129] The attention weights are then calculated using the scaled dot product formula, a weight matrix that represents the correlation between positions within the window:

[0130]

[0131] Finally, in order to enhance the expressiveness of the model, a multi-head attention mechanism is used. Assume that h is used for each head i The results are stitched together and passed through an output projection matrix

[0132] MultiHead(W i )=Concat(head1,…,head h )W O (31)

[0133] In formula (31), is the linear transformation matrix of the ith head.

[0134] 4) Moving window mechanism

[0135] In order to better capture a wider range of contextual information, a moving window mechanism is introduced, which includes window translation and cross-window information interaction. That is, before the self-attention operation of each layer of windows, the window division is translated to the right or downward by a fixed step size to form a new window division; since the window position is offset, part of the adjacent windows will be included in the same window, thereby realizing cross-window information interaction.

[0136] 5) Hierarchical feature extraction

[0137] In order to further extract features and increase the nonlinear expression ability of the model, the window self-attention layer and multi-layer perceptron (MLP) are introduced to enable the model to learn more complex features. First, the input features are normalized, and then the window self-attention layer is applied.

[0138]

[0139] In formula (32), W MSA =MultiHead(W i ) represents the window multi-head attention, LN is the normalization operation, is the output of the self-attention layer.

[0140] Then Normalized and passed through a fully connected MLP network.

[0141]

[0142] is the output of the MLP network.

[0143] 6) Hierarchical feature extraction

[0144] The multi-scale features are gradually extracted through the hierarchical structure, and the input feature map is convolved and pooled to halve the feature map resolution, that is, the H×W×C feature map is reduced to The features extracted at each layer contain a gradually increasing receptive field, thereby obtaining broader global information from fine-grained local information. After multiple feature extractions, the final output is an H t ,W t ,C t feature map, that is, the global features of the point cloud from multiple views.

[0145] Further, in step S3, a vector quantization variational autoencoder is applied to encode the multi-view point cloud features, which specifically includes the following steps:

[0146] S31, inputting the multi-view point cloud features into the encoder to obtain a continuous potential vector;

[0147] S32. Replace the continuous latent vector with a discrete vector to obtain the encoded multi-view point cloud features.

[0148] Since point cloud data is discrete data, the present invention uses a vector quantization variational autoencoder to discretely encode multi-view point cloud features, thereby obtaining a more compact data representation to improve the long-term prediction of the subsequent point cloud diffusion model. The vector quantization variational autoencoder (VQ-VAE) is used for encoding, as follows:

[0149] The H t ,W t ,C t The feature map is represented as First, the feature map is input into the encoder to obtain an H t ,W t ,d-shaped latent continuous feature vector d is the dimension of the latent space.

[0150] Define the latent embedding space e∈R K×d , where K is the size of the latent embedding space, which contains K embedding vectors, and d is the dimension of each vector. For each latent vector z x (h, w), in order to replace each continuous latent vector with its closest discrete vector, find the closest vector in the latent embedding space

[0151]

[0152] Then the continuous potential vector is replaced by a discrete vector. This process is a quantization process. The quantized potential feature map Each z x (h,w) is replaced by the vector in the closest latent embedding space

[0153]

[0154] That is to H t ,W t ,C t The result of discrete encoding of the input feature map.

[0155] Furthermore, in step S4, a point cloud diffusion model is established, and the point cloud diffusion model is trained based on the encoded multi-view point cloud features and the loss function, specifically including the following steps:

[0156] S41, define a noise matrix;

[0157] S42, based on the noise matrix, gradually adding noise to the encoded multi-view point cloud features in the input data to train the forward diffusion process of the point cloud diffusion model;

[0158] S43, generating raw 3D point cloud data through the U-Net decoder to train the reverse denoising process of the point cloud diffusion model;

[0159] S44. Based on the loss function, the reverse denoising process of the point cloud diffusion model is optimized to obtain a trained point cloud diffusion model.

[0160] 1) Define the noise matrix

[0161] In order to better perform long-term prediction of point clouds, the present invention designs a point cloud diffusion model for point cloud data; at the same time, in order to capture the feature changes in three-dimensional space and improve the quality of point cloud prediction, the present invention uses a tensor-based noise injection method instead of the traditional Gaussian noise method.

[0162] At each time step t, define the noise matrix The spatial feature size is denoted as Where Q t Depend on and Together they form.

[0163]

[0164] m, n, p are tensor identifiers used to define Q t Medium 3D subtensor The index of , i, j, k are tensor subscripts, and the value range is as follows:

[0165]

[0166] When α t =1-γ t , which means that each labeled discretized feature vector have The probability of remaining unchanged, and the probability of transitioning to other K-1 geometric categories is β t , where α t and β t Both are related to the time step t and the total time step t.

[0167] 2) Forward diffusion process of point cloud diffusion model

[0168] The forward diffusion process of the point cloud diffusion model is The noise is gradually added to the three-dimensional geometric input, that is, the state z at each time step t t From the state z of step t-1 in the previous period of time t-1 The joint probability distribution of its forward diffusion is:

[0169]

[0170] q(z0) is the prior distribution of the initial state, and its single-step noise addition formula is:

[0171]

[0172] Where Q t is the noise matrix.

[0173] And for the conditional distribution from the initial state z0 to any time step t:

[0174] q(z t |z0)=Q1·Q2·…·Q t ·z0 (41)

[0175] where Q1, Q2, …, Q t is the noise matrix at each time step.

[0176] 3) Reverse denoising process of point cloud diffusion model

[0177] The purpose of the reverse denoising process of the point cloud diffusion model is to transform the noise state z t Restore to the original state z0. In the forward diffusion process of the point cloud diffusion model, noise is added to the data. In the reverse denoising process of the point cloud diffusion model, a denoiser is required to gradually restore the original data in the reverse process. The present invention uses a U-Net network to predict the change from the current discretized data state to a state closer to the original point cloud from each diffusion step, thereby gradually denoising and reconstructing the point cloud. The conditional probability distribution of reverse denoising can be obtained by Bayes' theorem, and p is defined θ (z t-1 |z t ) is from z t to z t-1 The conditional probability of , so we have:

[0178] p θ (z t-1 =i|z t =j)∝q(z t =j|z t-1 =i)p θ (z t-1 =i) (42)

[0179] Further expanded to:

[0180]

[0181] where q(z t |z t-1 ) is the transition probability of the forward process, p θ (z t-1 ) is the prior distribution obtained by the reverse process, and the denominator is a normalization constant to ensure that the sum of the probabilities is 1.

[0182] The joint probability distribution of inverse denoising is:

[0183]

[0184] Where p(z T ) represents the final noise state.

[0185] The U-Net network passes through the conditional probability p θ (z t-1 |z t ) Learn from z t to z t-1 The original 3D point cloud data is gradually restored through mapping.

[0186] 4) Loss function of point cloud diffusion model

[0187] To train the reverse denoising model p θ (z t-1 |z t ), using the evidence lower bound loss function (ELBO), which aims to maximize the log-likelihood of the data,

[0188]

[0189] KL stands for Kullback-Leibler divergence, which is used to measure the difference between the forward diffusion process and the reverse denoising process. By minimizing this loss function, the present invention can optimize the reverse denoising model so that it can gradually recover the original data from the noise at each time step.

[0190] 5) Training diffusion model based on designed loss function

[0191] The model training process is to first encode the extracted point cloud features through the vector quantization variational autoencoder for the input data, add tensor-based noise through the forward process of the point cloud diffusion model, and predict the features encoded by the vector quantization variational autoencoder through the reverse diffusion model, and finally restore the 3D point cloud through the U-Net decoder. The loss function uses the point cloud diffusion loss function of sub-step 4. The weight information during the training process is saved in real time, and the weight with the best result is selected for subsequent point cloud prediction.

[0192] Furthermore, in step S5, long-term prediction of point clouds based on the trained point cloud diffusion model includes: inputting the point clouds of the current frame and the five previous frames into the trained point cloud diffusion model, calling the reverse denoising process of the point cloud diffusion model and applying the time embedding method to predict the point clouds for the next 1 second.

[0193] Specifically, after the point cloud diffusion model is established and trained, the present invention uses 10Hz laser radar point cloud data for long-term prediction of point clouds. Initially, the best weights of the model saved above are used to input t-5, t-4, t-3, t-2, t-1, and t frame point cloud data into the point cloud test model to predict the point cloud in the next 1 second, and the chamfer distance is used to evaluate the point cloud prediction quality. The above time steps are collectively referred to as T, and time embedding is used to map the time steps to a high-dimensional embedding space, as follows:

[0194] 1) Calculate embedding frequency

[0195]

[0196] where i∈{0,1,…,half dim -1},

[0197] 2) Time step embedding

[0198] The time steps are embedded using sine and cosine functions:

[0199] emb(T)=[sin(T·freqs),cos(T·freqs)] (47)

[0200] 3) Zero padding

[0201] pad(emb)=[emb(T),0] (48).

[0202] Through time embedding, the temporal relationship between the current frame and its previous five frames can be better represented, which helps the model better capture temporal dependencies and sequence characteristics and improves the prediction effect of the model.

[0203] As an example, the following is a specific operation flow of the point cloud long-term prediction method for autonomous driving scene flow of the present invention. The present invention takes the KITTI dataset as an example. The KITTI dataset includes multiple groups of point cloud data. For the given data, 0 to 7 groups are used as training data to establish a point cloud diffusion model and the diffusion model is trained based on the designed loss function. 8 to 10 groups are used as test data to perform long-term prediction of the point cloud data. First, step S1 is performed: converting the 3D point cloud data into multiple views. According to the direction angle and coordinate information given in the data, the 3D point cloud data is coordinate transformed, and the transformed coordinates are projected to obtain a bird's-eye view, a left view, and a front view. Among them, the visualization after the KITTI dataset is converted to a bird's-eye view is as follows Figure 3 shown.

[0204] For the converted view, the spatial structure features of the point cloud are extracted. The specific network architecture of step S2 for extracting multi-view point cloud features is as follows:

[0205] Sub-step (1): Calculate the spatial transformation matrix of the point cloud

[0206] 1) The input data x is passed to the first convolutional layer, which has 3 input channels, 64 output channels, and a convolution kernel size of 1. Batch normalization is used on the convolved data, and the Relu activation function is applied to the normalized data. The batch_size is set to 32. The input data x is one of the multiple views.

[0207] 2) The output of step (1) is passed to the second convolutional layer, which has 64 input channels, 128 output channels, a convolution kernel size of 1, batch normalization of the convolved data, and a ReLU activation function applied to the normalized data.

[0208] 3) The output of step (2) is passed to the third convolutional layer, which has an input channel number of 128, an output channel number of 1024, a convolution kernel size of 1, batch normalization of the convolved data, and a ReLU activation function applied to the normalized data.

[0209] 4) Perform the maximum pooling operation on the channel dimension of the output x of step (3), retaining the maximum value while keeping the dimension unchanged.

[0210] 5) Pass the output of step (4) to the first fully connected layer with a size of (1024,512), then apply batch normalization and apply the ReLU activation function to the normalized data.

[0211] 6) The output of step (5) is passed to the second fully connected layer with a size of (512, 256), and then batch normalization is applied and the ReLU activation function is applied to the normalized data.

[0212] 7) Pass the output of step (6) to the third fully connected layer, which has a size of (256, 9), add the output to the unit matrix of dimension (32, 9), and adjust the matrix shape to (32, 3, 3).

[0213] Sub-step (2): Extract point cloud features of each view

[0214] 1) Apply the spatial transformation matrix calculated in substep (1) to each point, and transform the data x to a dimension of (32, 3, n_pts), where n_pts is the number of points, which is a dynamic transformation. Then use the first convolution layer, which has 3 input channels, 64 output channels, and a convolution kernel size of 1. Use batch normalization on the convolved data, apply the Relu activation function to the normalized data, and save the current feature map as a local feature.

[0215] 2) The output of step (1) is passed to the second convolutional layer, which has 64 input channels, 128 output channels, a convolution kernel size of 1, batch normalization of the convolved data, and a ReLU activation function applied to the normalized data.

[0216] 3) The output of step (2) is passed to the third convolutional layer, which has an input channel number of 128, an output channel number of 1024, a convolution kernel size of 1, batch normalization of the convolved data, and a ReLU activation function applied to the normalized data.

[0217] 4) Perform a global maximum pooling operation on the output of step (3) in the channel dimension, and finally obtain a tensor of (32, 1, 1024, 1024), which is the extracted global feature vector of the view.

[0218] Repeat the operations of substeps (1) and (2) to extract the point cloud features of the bird's-eye view, left view, and front view respectively, and aggregate them into a tensor. The shape of the aggregated tensor is (32, 1, 1024, 1024), which is the global feature vector of multiple views.

[0219] Sub-step (3): extracting the global features of the point cloud from the aggregated feature vector

[0220] 1) Input feature processing and segmentation

[0221] The tensor x obtained in substep (2) is reshaped into (1, 1024, 1024) and passed to the convolutional layer for block operation. The block size is 8. The number of input channels of this convolutional layer is 512, the embedding dimension is 96, the step size is 8, and the convolution kernel size is 8. Each block is converted from the original channel space to the embedding space through this convolutional layer. The final embedded vector is (8, 8, 96).

[0222] 2) Window division and self-attention calculation

[0223] The vector obtained in step (1) is reshaped into a feature map of (128, 128, 96), and then the feature map is divided into windows of 44, resulting in a total of 32 windows, and the batch_size is set to 16.

[0224] Before calculating the window self-attention, the dimension of input x is adjusted to (512, 16, 96), and after passing through the linear layer, it is rearranged into a multi-head attention form, and the tensor is split into three parts: query, key, and value; the query vector is scaled, and the scaled query vector and key vector are dot-producted and transposed. The result of the operation is the attention weight matrix. The relative position deviation is obtained according to the relative position index, added to the attention weight matrix, and the final output is (512, 16, 96) after passing through the linear layer again.

[0225] 3) Hierarchical feature extraction, layered feature extraction, moving window mechanism

[0226] The input feature map is normalized and padded, and then the self-attention mechanism in step (2) is used to remove the previous feature map padding, apply residual connection and normalization, and finally obtain a (32,32,96) feature map.

[0227] In order to improve the calculation speed, reduce the calculation cost, and improve the efficiency and effect of the subsequent point cloud diffusion model, a vector quantization variational autoencoder is used for discrete encoding to obtain a more compact data representation. In step S3, the vector quantization variational autoencoder is used to encode the multi-view point cloud features, as follows:

[0228] 1) Pass the input data (32,32,96) to the first convolutional layer with 96 input channels, 128 output channels, 4 kernel size, 2 stride, 1 edge padding, and apply the Relu activation function to the convolved data.

[0229] 2) Pass the output of step (1) to the second convolutional layer, which has an input channel number of 128, an output channel number of 256, a convolution kernel size of 4, a stride of 2, an edge padding of 1, batch normalization of the convolved data, and a ReLU activation function applied to the normalized data.

[0230] 3) Pass the output of step (2) to the third convolutional layer, which has an input channel number of 256, an output channel number of 256, a convolution kernel size of 3, a stride of 1, an edge padding of 1, batch normalization of the convolved data, and a ReLU activation function applied to the normalized data.

[0231] 4) Use residual blocks and residual stacks to solve the existing gradient vanishing problem and improve model performance.

[0232] The present invention designs a point cloud diffusion model for point cloud data to achieve the purpose of long-term prediction of point clouds; at the same time, in order to improve the quality of long-term prediction of point clouds, the present invention uses a tensor-based noise injection method instead of a traditional Gaussian noise method, wherein a point cloud diffusion model is established in step S4, and the point cloud diffusion model is trained based on the encoded multi-view point cloud features and the loss function, as follows:

[0233] Sub-step (1): define the noise matrix

[0234] In order to capture the feature changes in three-dimensional space and improve the quality of point cloud prediction, the present invention uses a tensor-based noise injection method instead of the traditional Gaussian noise method.

[0235] At each time step t, define the noise matrix The spatial feature size is denoted as Where Q t Depend on and Together they form.

[0236]

[0237] m, n, p are tensor identifiers used to define Q t Medium 3D subtensor The index of , i, j, k are tensor subscripts, and the value range is as follows:

[0238]

[0239] When α t =1-γ t , which means that each labeled discretized feature vector have The probability of remaining unchanged, and the probability of transitioning to other K-1 geometric categories is β t , where α t and β t Both are related to the time step t and the total time step t.

[0240] Sub-step (2): forward process of point cloud diffusion model

[0241] Since the noise used in the present invention is a tensor-based noise injection method, the β in sub-step (1) is defined here t =0.6,α t =0.4, r t = 0.4, from which the tensor noise matrix Q is calculated t , posterior variance, posterior variance coefficient, etc., and gradually add noise to the input data to the point cloud data encoded by the vector quantized variational autoencoder, and perform the forward process output without a classifier based on the conditional and unconditional input calculations.

[0242] Sub-step (3): Reverse process of point cloud diffusion model

[0243] The reverse process of the point cloud diffusion model is used to gradually recover the original data from the noise data, which is the core part of the entire diffusion model. First, create a time step tensor and set it to the current time step. The multi-step diffusion process is controlled by the diffusion model multi-step solver. The entire diffusion process is continuously optimized using the evidence lower bound, so as to gradually recover high-quality data from the noise. At each time step, the current noise is calculated using the forward process without a classifier, where the forward process without a classifier calls the Unet network to generate the original 3D point cloud.

[0244] Sub-step (4), loss function of point cloud diffusion model

[0245] To train the reverse denoising model p θ (zt-1 |z t ), using the evidence lower bound loss function (ELBO), which aims to maximize the log-likelihood of the data,

[0246]

[0247] KL stands for Kullback-Leibler divergence, which is used to measure the difference between the forward diffusion process and the reverse denoising process. By minimizing this loss function, the present invention can optimize the reverse denoising model so that it can gradually recover the original data from the noise at each time step.

[0248] Sub-step (5): training the diffusion model based on the designed loss function

[0249] The model training process is to first encode the extracted point cloud features through the vector quantization variational autoencoder for the input data, add tensor-based noise through the forward process of the point cloud diffusion model, and predict the features encoded by the vector quantization variational autoencoder through the reverse diffusion model, and finally restore the 3D point cloud through the U-Net decoder. The loss function uses the point cloud diffusion loss function of substep (4). The weight information during the training process is saved in real time, and the best weight is selected for subsequent point cloud prediction.

[0250] In step S5, long-term prediction of point cloud is performed based on the trained point cloud diffusion model, as follows:

[0251] The point cloud data of the current frame and the five previous frames are input into the trained point cloud diffusion model. By repeating the point cloud and adding random noise, a partial point cloud is generated and converted into a sparse tensor form. Then the point cloud diffusion model is called in reverse process and the time embedding method is applied to generate the point cloud of the next 1 second and 3 seconds. Among them, the original point cloud, that is, the point cloud of the current frame, is as follows Figure 4 As shown, the point cloud is predicted to be as follows after 1 second Figure 5 As shown, the predicted point cloud after 3 seconds is as follows Figure 6 shown.

[0252] The point cloud long-term prediction method for the autonomous driving scene flow of the present invention can achieve the purpose of point cloud long-term prediction without losing the point cloud spatial structure information, semantic information, boundary continuity, etc., and has good generalization ability, so that the autonomous driving system can know the possible movement trajectory and change trend of the object in advance, thereby adjusting the driving strategy of the vehicle to avoid collision.

[0253] What is disclosed above is only a preferred embodiment of the present invention, which certainly cannot be used to limit the scope of rights of the present invention. A person skilled in the art can understand that all or part of the processes of the above embodiments and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. A point cloud long-term prediction method for autonomous driving scene flow, characterized in that: The following steps are involved: S1, converting 3D point cloud data into multiple views; S2, extract multi-view point cloud features; S3, apply vector quantization variational autoencoder to encode multi-view point cloud features; S4, establishing a point cloud diffusion model, and training the point cloud diffusion model based on the encoded multi-view point cloud features and loss function; S5. Perform long-term point cloud prediction based on the trained point cloud diffusion model.

2. The point cloud long-term prediction method for autonomous driving scene flow according to claim 1, characterized in that: In step S1, the 3D point cloud data is converted into multiple views, specifically including the following steps: S11, coordinate conversion of 3D point cloud data; S12, projecting the obtained transformed coordinates onto the bird's-eye view plane to convert the 3D point cloud data into a bird's-eye view; S13, projecting the obtained transformed coordinates onto the left view plane to convert the 3D point cloud data into a left view; S14. Project the obtained transformed coordinates onto the front view plane to convert the 3D point cloud data into a front view.

3. The point cloud long-term prediction method for autonomous driving scene flow according to claim 1, characterized in that: In step S2, the extraction of multi-view point cloud features specifically includes the following steps: S21, extracting point cloud features of each view in the multi-view; S22. Extracting global features of point clouds of multiple views based on point cloud features of each view.

4. The point cloud long-term prediction method for autonomous driving scene flow according to claim 3 is characterized in that: In step S21, extracting point cloud features of each view in the multiple views specifically includes the following steps: S211, independently extracting features of the points in each view to obtain a feature vector of each point; S212, using a maximum pooling operation to aggregate the feature vector features of each point into a global feature vector of the view; S213: Aggregate the global feature vector of each view to obtain the global feature vector of multiple views.

5. The point cloud long-term prediction method for autonomous driving scene flow according to claim 4, characterized in that: In step S22, extracting the global features of the point cloud of multiple views specifically includes the following steps: S221, preprocessing and dividing the global feature vectors of multiple views to obtain embedded feature vectors; S222, reshape the embedded feature vector into a feature map, and divide the feature map into a plurality of non-overlapping windows; S223, performing window self-attention calculation; S224. Introduce a moving window mechanism, and combine window self-attention to perform hierarchical feature extraction and layered feature extraction on the feature map to obtain the global features of the point cloud of multiple views.

6. The point cloud long-term prediction method for autonomous driving scene flow according to claim 1, characterized in that: In step S3, the application of the vector quantization variational autoencoder to encode the multi-view point cloud features specifically includes the following steps: S31, inputting the multi-view point cloud features into the encoder to obtain a continuous potential vector; S32. Replace the continuous latent vector with a discrete vector to obtain the encoded multi-view point cloud features.

7. The point cloud long-term prediction method for autonomous driving scene flow according to claim 1, characterized in that: In step S4, the point cloud diffusion model is established, and the point cloud diffusion model is trained based on the encoded multi-view point cloud features and the loss function, which specifically includes the following steps: S41, define a noise matrix; S42, based on the noise matrix, gradually adding noise to the encoded multi-view point cloud features in the input data to train the forward diffusion process of the point cloud diffusion model; S43, generating raw 3D point cloud data through the U-Net decoder to train the reverse denoising process of the point cloud diffusion model; S44. Based on the loss function, the reverse denoising process of the point cloud diffusion model is optimized to obtain a trained point cloud diffusion model.

8. The point cloud long-term prediction method for autonomous driving scene flow according to claim 7, characterized in that: The loss function is: Among them, q(z t-1 |z t ,z0) is the forward diffusion process of the point cloud diffusion model, p θ (z t-1 |z t ) is the reverse denoising process of the point cloud diffusion model; KL represents the Kullback-Leibler divergence, which is used to measure the difference between the forward diffusion process and the reverse denoising process.

9. The point cloud long-term prediction method for autonomous driving scene flow according to claim 8, characterized in that: In step S5, long-term prediction of point clouds based on the trained point cloud diffusion model includes: inputting the point clouds of the current frame and the five previous frames into the trained point cloud diffusion model, calling the reverse denoising process of the point cloud diffusion model and applying the time embedding method to predict the point clouds for the next 1 second.

10. The point cloud long-term prediction method for autonomous driving scene flow according to claim 9, characterized in that: The time embedding method comprises: 1) Calculate embedding frequency where i∈{0,1,…,half dim -1}, 2) Time step embedding The time steps are embedded using sine and cosine functions: emb(T)=[sin(T·freqs),cos(T·freqs)] 3) Zero padding pad(emb)=[emb(T),0].

Images