Point cloud prediction method based on three-dimensional pyramid optical flow

Abstract

The invention belongs to the field of automatic driving processing, and discloses a three-dimensional pyramid optical flow-based point cloud prediction method, which comprises the following steps of: firstly, converting a three-dimensional point cloud into a range view through spherical projection so as to obtain dense and ordered input data; secondly, considering that the position and the size of the object and the distance to the vehicle are important, the range view sequence prediction problem is decomposed into prediction of the change of the spatial position and the gray level (representing the distance to the vehicle) in the scene; specifically, pyramid multi-scale features of channels under a range view are constructed using three-dimensional convolution to increase receptive fields and reduce model parameters. In order to predict a spatial position in a scene, a three-dimensional exchange layer is used to process spatial position changes, and the influence of spatial object scale changes is integrated at the same time. In addition, the invention also designs a novel convolutional long and short-term memory network layer to capture the gray scale change of the range view, thereby improving the prediction precision. Experimental results show that future point cloud data which are detailed and accurate in structure can be predicted through the point cloud data at the current moment, and the method can be used in the fields of automatic driving, cultural relic protection and the like.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving processing technology, and in particular relates to a point cloud prediction method based on three-dimensional pyramid optical flow. Background Art

[0002] In autonomous driving, vehicles should be able to perceive and understand the surrounding environment and make appropriate decisions in advance to achieve safe driving, such as planning paths and avoiding potential dangers. Predicting the future surrounding environment is crucial to making timely and appropriate decisions. It is necessary to accurately and efficiently estimate the location of objects and the distance between the vehicle and the objects, while striking a balance between accuracy, efficiency and the number of adjustments to model parameters. This brings additional challenges.

[0003] Point cloud prediction, i.e. predicting future 3D point cloud sequences based on past 3D point cloud sequences, has attracted increasing attention in the field of autonomous driving because: first, 3D point clouds are high-fidelity and cost-effective; in addition, compared with RGB images and videos, point clouds can better describe the 3D surrounding environment and provide more information for predicting the surrounding environment; point cloud prediction is usually considered a self-supervised task and does not require additional manual annotations, because the ground truth data is provided by subsequent laser scanning.

[0004] Among the existing point cloud prediction methods, predicting the motion of each point on the LiDAR (Light Detection and Ranging) point cloud is a common solution, called point-based methods. Satisfactory results have been obtained, and fewer parameters are required to predict the motion of each point. However, point-based methods lead to a large amount of inference time, and even the latest state-of-the-art method RPPNet cannot reach the prediction rate of LiDAR (10 Hz). Another method is to predict the range view of the point cloud, denoted as range view-based methods, such as TCNet and PCPNet, which greatly reduces the inference time and obtains higher prediction accuracy, although the number of model parameters is five times that of point-based methods. Due to the rapid movement of autonomous vehicles, the position, size, and distance of objects in the surrounding environment to the vehicle vary greatly in consecutive frames. Therefore, it is very important to consider this information for predicting the surrounding environment point cloud. In addition, both efficiency and accuracy are crucial for point cloud prediction, and fewer parameters are more conducive to model deployment and save storage space. Therefore, to the best of our knowledge, less inference time, higher prediction accuracy, and fewer model parameters are all considered desirable, but existing methods have not yet balanced these.

[0005] To alleviate these challenges, the present invention proposes a three-dimensional pyramid optical flow point cloud prediction network. Similar to previous work based on range views, the present invention first uses spherical projection to convert the point cloud sequence into a two-dimensional range view sequence, which greatly improves the reasoning efficiency. Secondly, considering that the position, size and distance to the vehicle of the object are all important, the present invention decomposes the range view sequence prediction problem into the prediction of the changes in spatial position and grayscale (representing the distance to the vehicle) in the scene. Specifically, the present invention uses three-dimensional convolution to construct pyramid multi-scale features of the channel under the range view to increase the receptive field and reduce model parameters. In order to predict the spatial position in the scene, the present invention introduces a three-dimensional exchange layer to handle the spatial position changes while integrating the effects of object scale changes. In addition, the present invention also designs a novel ConvLSTM layer to capture the grayscale changes of the range view, thereby improving the prediction accuracy.

[0006] In this invention, we evaluate the proposed method on two typical autonomous driving datasets KITTI. The experimental results demonstrate the effectiveness of the proposed method.

[0007] The problems and defects of the existing technology are as follows: the existing point cloud prediction technology relies on more computing resources, or lacks the ability to represent missing details, especially in different objects and scenes, and the generalization effect is weak. Summary of the invention

[0008] In view of the problems existing in the prior art, the present invention provides a point cloud prediction method based on three-dimensional pyramid optical flow.

[0009] The present invention is implemented in this way: a point cloud prediction method based on three-dimensional pyramid optical flow includes:

[0010] Step 1: Randomly sample the current point cloud sequence to reduce the overall computational complexity and thus reduce the use of computing resources by reducing the number of points in the point cloud;

[0011] Step 2: Convert the sampled 3D point cloud into a 2D range view by spherical projection to obtain dense and ordered input data;

[0012] Step 3: Use 3D convolution to map the single-channel range view to multiple channels. Then input the multi-channel range view into the pyramid feature extraction module for feature extraction.

[0013] Step 4: Use a three-dimensional exchange layer to handle spatial position changes and integrate the effects of spatial object scale changes;

[0014] Step 5: Use a convolutional long short-term memory network layer to capture the grayscale changes of the range view to improve the prediction accuracy;

[0015] Step 6: Concatenate the results of step 4 and step 5, and use stacked convolution and transposed convolution to upsample the concatenated results. The obtained aggregated features are used as the input of the next step.

[0016] Step 7: Use a 3D convolutional layer to decode the output of step 6, and use a Sigmoid activation function to map the decoded range between 0 and 1 to form a future range view;

[0017] In step eight, spherical back projection is used to convert the range view in step seven into a predicted point cloud sequence, and the distance between the real point cloud sequence and the predicted point cloud sequence is used as the loss function to train the network framework.

[0018] Furthermore, the step 2 converts the sampled three-dimensional point cloud into a two-dimensional range view by spherical projection to obtain dense and ordered input data, including the following steps: recording the current point cloud sequence The goal of point cloud prediction is to predict future point clouds. Here p is the length of the point cloud sequence. Specifically, we Projection to spherical coordinates:

[0019]

[0020] Here, (H, W) represents the height and width of the range view, and f = f up +f down , where f up represents the vertical field of view of the sensor upward, f down Represents the vertical field of view of the sensor downward. Each pixel (grayscale) in the range view stores the distance r between the sensor and the object:

[0021]

[0022] If there is no 3D point in the corresponding pixel, fill -1, but if a projected point corresponds to multiple 3D points, the point closest to the sensor is selected. Because in practice, the closer the point to the sensor, the greater the contribution. After obtaining the range view of the point cloud, we connect it along the time dimension to obtain a 4D input tensor (C, p, H, W), where C is the number of image channels and the value is 1.

[0023] Further, the single-channel range view is mapped to multiple channels using three-dimensional convolution in step three. The multi-channel range view is input into a pyramid feature extraction module for feature extraction, including the following steps: given a range view, the single-channel range view is first mapped to multiple channels using three-dimensional convolution. Subsequently, the multi-channel range view is input into a pyramid feature extraction module consisting of three-dimensional convolution, normalization, and LeakyReLU activation function.

[0024] Furthermore, the three-dimensional exchange layer in step 4 is used to process the spatial position change and integrate the influence of the scale change of the spatial object, including the following steps: Specifically, for the i-th layer in the pyramid, we use a two-layer stacked convolution structure to estimate the position change P i :

[0025] P i =σ(norm(conv(F i ,w i ))) (3)

[0026] Among them, Fi is the feature generated by the feature extraction layer, w i Represents the convolution parameters of the LeakyReLU activation function. i The dimension is (3,p,h,w), where h i and w i are the width and height of the features in the i-th layer.

[0027] Get the position change P i Finally, we perform a three-dimensional swap operation on the features of the i-th layer to integrate the position change and size change into S i middle:

[0028]

[0029] in represents the initial position of the pixel in the i-th layer, d k Represents the dimension of the i-th layer. Swapping(f,grid) here means filling the pixel values ​​at the corresponding positions in the input to the specified positions.

[0030] Further, the step 5 uses a convolutional long short-term memory network layer to capture the grayscale changes of the range view, thereby improving the prediction accuracy, including the following steps: using a convolution operation to replace the matrix multiplication of each gate in the traditional long short-term memory unit to form a ConvLSTM. The input of the ConvLSTM unit includes the unit state C of the previous frame t-1 , hidden state h t-1 And the grayscale feature G of the current frame t The output is composed of the unit state C of the current frame t and the hidden state h t Then ConvLSTM is used to predict the changes of range views.

[0031] Further, the three-dimensional convolution layer in step seven is used to decode the result of the output of step six, and the decoded range is mapped between 0 and 1 using the Sigmoid activation function to form a future range view, including the following steps: the aggregated features are decoded using a three-dimensional convolution with a convolution kernel size of (1,1,1) and a step size of (1,1,1). This produces a final output tensor of size (2,p,H,W). We apply the Sigmoid function to map the output value to a range between 0 and 1. The first output channel represents the future range view sequence, and the second channel represents the probability of each point in the range view becoming a valid future point.

[0032] Furthermore, the range view of step 7 is converted into a predicted point cloud sequence using spherical back projection in step 8, and the distance between the real point cloud sequence and the predicted point cloud sequence is used as the loss function to train the network framework, including the following steps: the range view of step 7 is converted into a predicted point cloud sequence by spherical back projection, and the CD loss function is used to constrain the gap between the predicted point cloud sequence and the real point cloud sequence. The Adam optimizer is used for training, with a batch size of 4, an initial learning rate of 0.001, and a learning rate decay of 0.99 for each iteration, for a total of 200 epochs.

[0033] Combining all the above technical solutions, the advantages and positive effects of the present invention are as follows: The present invention proposes a point cloud prediction method based on three-dimensional pyramid optical flow, which can generate future point clouds from given past point clouds to make timely and appropriate decisions in combination with downstream tasks of autonomous driving. In the feature extraction module, the present invention reduces the number of parameters in the model by using features with fewer channels. Inspired by optical flow prediction, the present invention decomposes the point cloud prediction problem into predictions of the spatial position and grayscale changes of objects, and proposes variants of the three-dimensional exchange layer and the ConvLSTM layer to solve these problems. The accuracy of point cloud prediction is improved by the mode of decomposition first and then aggregation. Through this research, it is hoped that new perspectives and technical support will be provided for the direction of autonomous driving, and further the development of the field of autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a flow chart of a point cloud prediction method based on three-dimensional pyramid optical flow provided by an embodiment of the present invention.

[0035] Figure 2 It is a flow chart for implementing a point cloud prediction method based on three-dimensional pyramid optical flow provided by an embodiment of the present invention.

[0036] Figure 3 It is a structural schematic diagram of a point cloud prediction method based on three-dimensional pyramid optical flow provided by an embodiment of the present invention.

[0037] Figure 4Schematic diagram of some completion results on the KITTI dataset provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0039] In view of the problems existing in the prior art, the present invention provides a point cloud prediction method, system, computer device and application based on three-dimensional pyramid optical flow. The present invention is described in detail below with reference to the accompanying drawings. The present invention proposes a novel graph attention mechanism, which can generate future point clouds from given past point clouds to make timely and appropriate decisions in combination with downstream tasks of autonomous driving. In the pyramid feature extraction module, the present invention reduces the number of parameters in the model by using features with fewer channels. Inspired by optical flow prediction, the present invention decomposes the point cloud prediction problem into predictions of the spatial position and grayscale changes of objects, and proposes variants of three-dimensional exchange layers and ConvLSTM layers to solve these problems. The accuracy of point cloud prediction is improved by a pattern of decomposition followed by aggregation. The present invention will demonstrate the advantages of our algorithm in prediction through visualization effects.

[0040] like Figure 1 As shown, the point cloud completion method based on graph attention provided by the present invention includes the following steps:

[0041] S101: Randomly sample the current point cloud sequence to reduce the number of points in the point cloud to reduce the overall computational complexity and thus reduce the use of computing resources;

[0042] S102: converting the sampled three-dimensional point cloud into a two-dimensional range view by spherical projection to obtain dense and ordered input data;

[0043] S103: Use 3D convolution to map the single-channel range view to multiple channels. Input the multi-channel range view into the pyramid feature extraction module for feature extraction;

[0044] S104: Use a three-dimensional exchange layer to handle spatial position changes while integrating the effects of spatial object scale changes;

[0045] S105: Use convolutional long short-term memory network layers to capture grayscale changes in range views to improve prediction accuracy;

[0046] S106: Concatenate the results of S104 and S105, and use stacked convolution and transposed convolution to upsample the concatenated results. The obtained aggregated features are used as input for the next step.

[0047] S107: Use a three-dimensional convolutional layer to decode the result outputted in step 6, and use a Sigmoid activation function to map the decoded range between 0 and 1 to form a future range view;

[0048] S108: Use spherical back projection to convert the range view in step seven into a predicted point cloud sequence, use the distance between the real point cloud sequence and the predicted point cloud sequence as the loss function, and train the network framework.

[0049] The point cloud prediction based on three-dimensional pyramid optical flow provided by the present invention can also be implemented by other steps by ordinary technicians in the industry. Figure 1 The point cloud prediction of the three-dimensional pyramid optical flow provided by the present invention is only a specific embodiment.

[0050] like Figure 2 As shown, the point cloud prediction based on three-dimensional pyramid optical flow provided by the present invention includes:

[0051] Point cloud downsampling module 1 randomly samples the current point cloud sequence, reducing the number of points in the point cloud to reduce the overall computational complexity and thus reduce the use of computing resources;

[0052] A spherical projection module 2 converts the sampled three-dimensional point cloud into a two-dimensional range view through spherical projection to obtain dense and ordered input data;

[0053] Feature extraction module 3 uses 3D convolution to map the single-channel range view to multiple channels. The multi-channel range view is input into the pyramid feature extraction module for feature extraction.

[0054] Position exchange module 4, uses a three-dimensional exchange layer to handle spatial position changes while integrating the effects of spatial object scale changes;

[0055] A grayscale prediction module 5 for capturing grayscale changes of range views using a convolutional long short-term memory network layer, thereby improving prediction accuracy;

[0056] Feature aggregation module 6, used to splice the results of the position exchange module and the grayscale prediction module, and upsample the spliced ​​results using stacked convolution and transposed convolution, and the obtained aggregated features are used as input for the next step;

[0057] A decoding module 7 is used to decode the result outputted in step 6 using a three-dimensional convolutional layer, and map the decoded range between 0 and 1 using a Sigmoid activation function to form a future range view;

[0058] The network training module 8 is used to convert the range view of step seven into a predicted point cloud sequence using spherical back projection, use the distance between the real point cloud sequence and the predicted point cloud sequence as the loss function, train the network framework, and obtain the predicted point cloud.

[0059] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0060] The point cloud completion network framework based on graph attention in this invention is shown in the following figure: Figure 3 As shown. Specifically including the following steps:

[0061] Step 1: Randomly sample the current point cloud sequence to reduce the overall computational complexity and thus reduce the use of computing resources by reducing the number of points in the point cloud:

[0062] Step 2: Remember the current point cloud sequence The goal of point cloud prediction is to predict future point clouds. Here p is the length of the point cloud sequence. Specifically, we Projection to spherical coordinates:

[0063]

[0064] Here, (H, W) represents the height and width of the range view, and f = f up +f down , where f up represents the vertical field of view of the sensor upward, f down Represents the vertical field of view of the sensor downward. Each pixel (grayscale) in the range view stores the distance r between the sensor and the object:

[0065]

[0066] If there is no 3D point in the corresponding pixel, fill -1, but if a projected point corresponds to multiple 3D points, the point closest to the sensor is selected. Because in practice, the closer the point to the sensor, the greater the contribution. After obtaining the range view of the point cloud, we connect it along the time dimension to obtain a 4D input tensor (C, p, H, W), where C is the number of image channels and the value is 1.

[0067] Step three, given a range view, first use 3D convolution to map the single-channel range view to multiple channels. Subsequently, the multi-channel range view is input into a pyramid feature extraction module consisting of 3D convolution, normalization, and LeakyReLU activation function. Specifically, the feature extraction module consists of four stacked feature encoding layers. For each feature encoding layer, we use 1×1×1 and 3×3×3 size convolution kernels for feature extraction, and we set the strides to 2 and 4, respectively, to increase the receptive field. In addition, circular padding is used to maintain the aspect ratio consistency of the range view. Starting from the multi-channel range view as the top layer, in each subsequent layer, the range view of the previous layer is scaled by a certain factor and the channel tree is expanded. This process ultimately produces a four-layer feature pyramid. We finally obtain features with 128 channels.

[0068] Step 4, use a three-dimensional exchange layer to handle spatial position changes and integrate the impact of spatial object scale changes, including the following steps: Specifically, for the i-th layer in the pyramid, we use a two-layer stacked convolution structure to estimate the position change Pi:

[0069] P i =σ(norm(conv(F i ,w i ))) (7)

[0070] Among them, F i is the feature generated by the feature extraction layer, w i Represents the convolution parameters of the LeakyReLU activation function. i The dimension is (3,p,h,w), where h i and w i are the width and height of the features in the i-th layer.

[0071] Get the position change P i Finally, we perform a three-dimensional swap operation on the features of the i-th layer to integrate the position change and size change into S i middle:

[0072]

[0073] in represents the initial position of the pixel in the i-th layer, d k Indicates the dimension of the i-th layer. Swapping(f,grid) here means filling the pixel values ​​at the corresponding positions in the input to the specified positions.

[0074] Step 5: Use a convolutional long short-term memory network layer to capture the grayscale changes of the range view, thereby improving the prediction accuracy, including the following steps: Use convolution operations to replace the matrix multiplication of each gate in the traditional LSTM unit to form ConvLSTM. The input of the ConvLSTM unit includes the cell state C of the previous frame t-1 , hidden state h t-1 And the grayscale feature G of the current frame t The output is composed of the unit state C of the current frame t and the hidden state h t Then use ConvLSTM to predict the change of range view. The calculation process of ConvLSTM is as follows:

[0075]

[0076] g t =ψ(conv(G t ;w Fg )+conv(h t-1 ;w hg )+b g ) (12)

[0077]

[0078] in represents the element-by-element product, represents the sigmoid activation function, and ψ represents the tanh activation function.

[0079] In step six, the results of step four and step five are concatenated, and the concatenated results are upsampled using stacked convolution and transposed convolution. The obtained aggregated features are used as the input of the next step.

[0080] Step 7: The aggregated features are decoded using a 3DCNN with a kernel size of (1,1,1) and a decoding layer with a stride of (1,1,1). This produces a final output tensor of size (2,p,H,W). A Sigmoid function is applied to map the output values ​​to a range between 0 and 1. The first output channel represents the sequence of future range views, while the second channel represents the probability of each point in the range view being a valid future point.

[0081] Step 8: The range view in step 7 is converted into a predicted point cloud sequence by spherical back projection, and the CD loss function is used to constrain the gap between the predicted point cloud sequence and the real point cloud sequence. The Adam optimizer is used for training, with a batch size of 4, an initial learning rate of 0.001, and a learning rate decay of 0.99 for each iteration, for a total of 200 epochs.

[0082] The technical effects of the present invention are described in detail below in conjunction with experiments.

[0083] The experiments of the present invention are carried out on the KITTI dataset. For the KITTI dataset, the present invention uses the first 11 sequences for experiments, that is, sequences 00-05 are used for training, sequences 06 and 07 are used for verification, and sequences 08, 09 and 10 are used for testing. Figure 4 The results of the present invention in different scenes in the KITTI dataset are shown, where the red point cloud is achieved by the present invention, and the green and blue represent the current known point cloud and the future ground truth (GT) point cloud. Figure 4 It can be seen that as the time step increases, the prediction scenario implemented by the present invention moves forward accordingly without lag.

[0084] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. It can be understood by a person of ordinary skill in the art that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0085] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A point cloud prediction method based on three-dimensional pyramid optical flow, characterized in that: The point cloud prediction method based on three-dimensional pyramid optical flow includes: Step 1: Randomly sample the current point cloud sequence to reduce the overall computational complexity and thus reduce the use of computing resources by reducing the number of points in the point cloud; Step 2: Convert the sampled 3D point cloud into a 2D range view by spherical projection to obtain dense and ordered input data; Step 3: Use 3D convolution to map the single-channel range view to multiple channels. Then input the multi-channel range view into the pyramid feature extraction module for feature extraction. Step 4: Use a three-dimensional exchange layer to handle spatial position changes and integrate the effects of spatial object scale changes; Step 5: Use a convolutional long short-term memory network layer to capture the grayscale changes of the range view to improve the prediction accuracy; Step 6: Concatenate the results of step 4 and step 5, and use stacked convolution and transposed convolution to upsample the concatenated results. The obtained aggregated features are used as the input of the next step. Step 7: Use a 3D convolutional layer to decode the output of step 6, and use a Sigmoid activation function to map the decoded range between 0 and 1 to form a future range view. In step eight, spherical back projection is used to convert the range view in step seven into a predicted point cloud sequence, and the distance between the real point cloud sequence and the predicted point cloud sequence is used as the loss function to train the network framework.

2. The point cloud prediction method based on three-dimensional pyramid optical flow according to claim 1, characterized in that: The step 2 in which the sampled three-dimensional point cloud is converted into a two-dimensional range view by spherical projection to obtain dense and ordered input data includes the following steps: recording the current point cloud sequence The goal of point cloud prediction is to predict future point clouds. Here p is the length of the point cloud sequence. Specifically, we Project to spherical coordinates to form an extent view: Here, (H, W) represents the height and width of the range view, and f = f up +f down , where f up represents the vertical field of view of the sensor upward, f down Represents the vertical field of view of the sensor downward. Each pixel (grayscale) in the range view stores the distance r between the sensor and the object: If there is no 3D point in the corresponding pixel, fill -1, but if a projected point corresponds to multiple 3D points, the point closest to the sensor is selected. Because in practice, the closer the point to the sensor, the greater the contribution. After obtaining the range view of the point cloud, we connect it along the time dimension to obtain a 4D input tensor (C, p, H, W), where C is the number of image channels and the value is 1.

3. The point cloud prediction method based on three-dimensional pyramid optical flow according to claim 1, characterized in that: The single-channel range view is mapped to multiple channels using three-dimensional convolution in step 3. The multi-channel range view is input into a pyramid feature extraction module for feature extraction, including the following steps: given a range view, the single-channel range view is first mapped to multiple channels using three-dimensional convolution. Subsequently, the multi-channel range view is input into a pyramid feature extraction module consisting of three-dimensional convolution, normalization, and LeakyReLU activation function.

4. The point cloud prediction method based on three-dimensional pyramid optical flow according to claim 1, characterized in that: The three-dimensional exchange layer in step 4 is used to process the spatial position change and integrate the influence of the scale change of the spatial object, including the following steps: Specifically, for the i-th layer in the pyramid, we use a two-layer stacked convolution structure to estimate the position change: P i =σ(norm(conv(F i ,w i ))) (3) Among them, F i is the feature generated by the feature extraction layer, w i Represents the convolution parameters of the LeakyReLU activation function. i The dimension is (3,p,h,w), where h i and w i are the width and height of the features in the i-th layer. Get the position change P i Finally, we perform a three-dimensional swap operation on the features of the i-th layer to integrate the position change and size change into the features: in represents the initial position of the pixel in the i-th layer, d k Represents the dimension of the i-th layer, Swapping(f,grid) means filling the pixel values ​​at the corresponding positions in the input to the specified positions.

5. The point cloud prediction method based on three-dimensional pyramid optical flow according to claim 1, characterized in that: The step 5 uses a convolutional long short-term memory network layer to capture the grayscale changes of the range view, thereby improving the prediction accuracy, including the following steps: using a convolution operation to replace the matrix multiplication of each gate in the traditional long short-term memory network (LSTM) unit to form a ConvLSTM. The input of the ConvLSTM unit includes the unit state C of the previous frame t-1 , hidden state h t-1 And the grayscale feature G of the current frame t The output is composed of the unit state C of the current frame t and the hidden state h t Then ConvLSTM is used to predict the changes of range views.

6. The point cloud prediction method based on three-dimensional pyramid optical flow according to claim 1, characterized in that: The step six uses a three-dimensional convolution layer to decode the result of the output of step six, and uses a Sigmoid activation function to map the decoded range between 0 and 1 to form a future range view, including the following steps: using a three-dimensional convolution layer with a convolution kernel size of (1,1,1) and a step size of (1,1,1) to decode the aggregated features. Finally, an output tensor of size (2,p,H,W) is generated. The output value is mapped to a range between 0 and 1 using a Sigmoid function. The first output channel represents a future range view sequence, and the second channel represents the probability of each point in the range view becoming a valid future point.

7. The point cloud prediction method based on three-dimensional pyramid optical flow according to claim 1, characterized in that: The step 8 converts the range view of step 7 into a predicted point cloud sequence by spherical back-projection, uses the distance between the real point cloud sequence and the predicted point cloud sequence as the loss function, trains the network framework, and obtains the completed point cloud, including the following steps: converts the range view of step 7 into a predicted point cloud sequence by spherical back-projection, and uses the CD loss function to constrain the gap between the predicted point cloud sequence and the real point cloud sequence. The Adam optimizer is used for training, with a batch size of 4, an initial learning rate of 0.001, and a learning rate decay of 0.99 for each iteration, for a total of 200 epochs.

8. A point cloud prediction method based on three-dimensional pyramid optical flow according to any one of claims 1 to 7, characterized in that: The point cloud prediction method based on three-dimensional pyramid optical flow: The point cloud downsampling module randomly samples the current point cloud sequence, reducing the number of points in the point cloud to reduce the overall computational complexity and thus reduce the use of computing resources; The spherical projection module converts the sampled 3D point cloud into a 2D range view through spherical projection to obtain dense and ordered input data; The feature extraction module uses 3D convolution to map the single-channel range view to multiple channels. The multi-channel range view is input into the pyramid feature extraction module for feature extraction. The position exchange module uses a three-dimensional exchange layer to handle spatial position changes while integrating the effects of spatial object scale changes; Grayscale prediction module, which uses convolutional long short-term memory network layers to capture grayscale changes in range views, thereby improving prediction accuracy; The feature aggregation module is used to concatenate the results of the position exchange module and the grayscale prediction module, and upsample the concatenated results using stacked convolution and transposed convolution layers. The obtained aggregated features are used as the input of the next step. A decoding module is used to decode the result of the feature aggregation module using a three-dimensional convolutional layer and map the decoded range between 0 and 1 using a Sigmoid activation function to form a future range view; The network training module is used to convert the range view of the decoding module into a predicted point cloud sequence using spherical back projection, and use the distance between the real point cloud sequence and the predicted point cloud sequence as the loss function to train the network framework and obtain the predicted point cloud.

Images