Iterative pyramid based scene flow estimation method and system

Abstract

This invention discloses a scene flow estimation method based on an iterative pyramid. Based on the observation that high-resolution levels in a pyramid structure have significantly higher estimation accuracy than low-resolution levels, an iterative pyramid structure is designed. This structure transfers high-resolution estimation information from the previous round to the current low-resolution level, providing a more accurate search center and more motion information, effectively continuing the high-precision estimation results and continuously refining the estimation. Furthermore, to address the issue of incomplete correspondence between two point cloud frames due to the sparsity of point cloud data, this invention designs a resampling mechanism. The source point cloud is warped using the scene flow predicted by the previous high-resolution level, and then resampled in the target point cloud, effectively improving the correspondence between the two frames. Scene flow estimation for the current round is then performed on the adjusted point cloud. This invention also discloses a corresponding scene flow estimation system based on an iterative pyramid.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, and more specifically relates to a scene flow estimation method and system based on iterative pyramids. Background Technology

[0002] Scene flow estimation aims to obtain the 3D motion vectors of points in a scene from two consecutive frames of point clouds. This task is fundamental in the field of autonomous driving, providing temporal motion information for advanced tasks such as dynamic object segmentation and multi-object tracking.

[0003] For scene flow estimation tasks, existing research methods widely use pyramid structures to achieve a balance between accuracy and computational cost. A pyramid structure is an "encoder-decoder" network composed of multiple layers. PointPWC-Net designs a pyramid structure to estimate scene flow, capturing large motions without expanding the search range. Furthermore, it proposes a novel block-to-block cost volume to efficiently encode motion information. Based on the pyramid structure, Bi-PointFlowNet designs a bidirectional flow embedding layer that can simultaneously extract forward and backward correlations. Also based on the pyramid structure, 3DFlow proposes a multi-pair cost volume to capture distant points while combining it with backward reliability verification. However, in the pyramid structure, when estimation is performed at low-resolution levels, incorrect search centers will prevent the search range from covering the true corresponding points, introducing accumulated errors.

[0004] Furthermore, due to the sparsity of point cloud data, point clouds in two consecutive time frames do not correspond perfectly, introducing noise into the scene flow estimation task. Therefore, some methods have designed point-to-block and block-to-block cost volumes. By using matching of larger regions (blocks) instead of point matching, the problem of poor point correspondence is alleviated. Secondly, RCP designs a two-stage iterative network, first using point-to-point optimization to obtain rule information, and then performing regularization based on a GRU-based recurrent network. However, these methods have limited effectiveness in addressing the problem of poor point correspondence, which is reflected in the scene flow estimation results, where estimation errors are relatively large in some challenging regions. Summary of the Invention

[0005] This invention aims to address the problems of accumulated errors introduced by low-resolution layers and poor correspondence between two frames of point clouds used as input data in scene flow estimation methods based on pyramid structures. The proposed iterative pyramid structure transfers high-resolution, high-precision estimation information from the previous round to the current round, providing a more accurate search center and supplementing motion information, effectively maintaining and refining the previous accuracy. Simultaneously, the predicted scene flow guides the resampling of the target point cloud, effectively improving the correspondence between two consecutive frames of point clouds. The scene flow estimation accuracy is currently the highest among all evaluation methods on FlyingThings3D and KITTI, demonstrating the effectiveness and generalization of this method.

[0006] To achieve the above objectives, this invention provides a scene flow estimation method based on iterative pyramids, the method comprising the following steps:

[0007] (1) Take two frames of point cloud data at time t and t+1 before and after the time. The first frame is the source point cloud P and the second frame is the target point cloud Q. Input the two frames of point cloud data into the feature extraction network and use the feature extraction network to encode the features of the two frames of point cloud to construct a multi-resolution feature pyramid.

[0008] (2) Based on the multi-resolution feature pyramid, the first round of scene flow estimation is performed. The scene flow estimation starts from the coarsest resolution and is then refined layer by layer to the 1 / 4 resolution level. For each level of scene flow estimation, the cost body is constructed using the source point cloud features and the distorted target point cloud features. The cost body and related information are input into the decoder for processing. The decoder features and the predicted scene flow are output. The decoder features and the predicted scene flow are then upsampled and passed to the next level.

[0009] (3) Use the scene flow predicted at the previous 1 / 4 resolution level to guide the resampling of the target point cloud, extract features from the resampled target point cloud, and use the extracted features for subsequent matching.

[0010] (4) Based on the source point cloud and the resampled target point cloud, scene flow estimation is performed in the second round and subsequent rounds; the scene flow predicted at the 1 / 4 resolution level in the previous round and the decoder output features are downsampled to 1 / 16, 1 / 32 and 1 / 128 resolutions to guide scene flow estimation at the corresponding resolution level in this round, provide search center and supplement motion information; then, based on the above information, scene flow estimation is performed in iterative rounds, and after being processed by the decoder, the scene flow predicted at the original resolution level is finally output.

[0011] In one embodiment of the present invention, in step (1), the step of using a feature extraction network to encode the features of two frames of point clouds to construct a multi-resolution feature pyramid specifically includes:

[0012] For the original resolution level l0, three 1D convolutions with a kernel size of 1 and a ReLU activation function are performed sequentially to encode features. Then, based on the farthest point sampling, the feature is downsampled to a 1 / 4 resolution level l1 to obtain the downsampled point cloud coordinates and corresponding indices. Feature encoding is performed using PointNet++. For each point in the point cloud, it is considered as the center point, and the 16 nearest points are found based on distance. Then, MLP processing is performed, namely 2D convolutions with a kernel size of 1, batch normalization, and a ReLU activation function. Finally, max pooling is performed in the neighborhood S dimension to complete the feature encoding at the 1 / 4 resolution level l1. The above operations are then repeated to generate features at resolutions of 1 / 16, 1 / 32, and 1 / 128. In addition, except for the original resolution level l0, the features obtained at each level are upsampled and then concatenated with the features of the corresponding level to effectively increase the receptive field. The final generated feature pyramid has five levels l0-l4, with resolutions of 1, 1 / 4, 1 / 16, 1 / 32, and 1 / 128, respectively.

[0013] In one embodiment of the present invention, in step (2), the scene flow estimation at each level and the layer-by-layer refinement specifically include:

[0014] The target point cloud is warped using the upsampled scene stream. A cost volume is constructed based on the features of the source point cloud and the warped target point cloud. The cost volume and related information are input into the scene stream decoder and processed using MLP. The decoder features and the predicted scene stream are output. During the layer-by-layer refinement process, information is passed between layers through the upsampling module. The upsampled scene stream provides a center for the search region for the current layer estimation, and the upsampled decoder features provide motion information with a larger receptive field for the current layer estimation.

[0015] In one embodiment of the present invention, in step (2), the upsampling specifically refers to:

[0016] The MLP is used to encode the direction vectors of the center point and neighborhood points and the corresponding decoder output features to generate weights. The number of MLP channels is set to [8,8,16]. The input neighborhood information B*C*S*N is subjected to three 2D convolutions with a kernel size of 1 and a ReLU activation function in sequence. The weights are assigned to the corresponding neighborhood points, and then the sparse point information is converted into dense point information by interpolation.

[0017] In one embodiment of the present invention, in step (2), the input of the cost body and related information into the decoder for processing specifically includes:

[0018] The decoder is constructed using an MLP with the number of MLP channels set to [96, 64]. The decoder input is subjected to two 1D convolutions with a kernel size of 1 followed by a ReLU activation function. A DenseNet-like bypass connection is also added. The decoder input includes the following four types: cost volume (CV), features f(P) from the first frame point cloud, and the upsampled scene flow up(S) estimated from the previous level. l ) and decoder output feature up(f l After MLP processing, the decoder outputs features containing motion information, which are then passed through a fully connected (FC) layer to output the estimated scene stream.

[0019] f l-1 =LP([f(P),V,up(S)) l ),p(f l )])(1)

[0020] S l-1 =C(f) l-1 (2).

[0021] In one embodiment of the present invention, the resampling of the target point cloud in step (3) specifically includes:

[0022] To improve the correspondence between the two point clouds, the coordinates of the target point cloud are adjusted to better correspond with the source point cloud. At 1 / 4 resolution, the source point cloud is distorted using the high-precision scene flow predicted in the previous round. Then, the distorted source point cloud is used to search for the nearest neighbor in the target point cloud at the original resolution based on distance. The found nearest neighbor is used as the new 1 / 4 resolution target point cloud to correct points with large deviations. Afterward, based on the index returned during the downsampling process of the source point cloud when constructing the feature pyramid in the first round, downsampling is performed on the new 1 / 4 resolution target point cloud to obtain target point clouds at 1 / 16, 1 / 32, and 1 / 128 resolutions, thus completing the resampling of the target point cloud. Finally, features are re-extracted from the adjusted target point cloud to encode neighborhood features for subsequent matching.

[0023] In one embodiment of the present invention, in step (4), the scene flow estimation of the iterative rounds specifically includes:

[0024] First, based on the index returned during the source point cloud downsampling process when constructing the feature pyramid in the first round, the decoder features and scene flow predicted at 1 / 4 resolution in the previous round are downsampled to obtain information at 1 / 16, 1 / 32, and 1 / 128 resolutions, which are used to guide the scene flow estimation at the corresponding resolution levels in this round. Among them, the high-precision estimated scene flow provides a more accurate search center for the estimation at the low resolution level in this round, effectively alleviating the problem that the search area cannot cover the true corresponding points. The decoder features estimated by high precision from the previous round further supplement the motion information. Based on the above information, the scene flow estimation of the iterative rounds is guided. After being processed by the decoder, the scene flow predicted at the original resolution level is finally output.

[0025] In one embodiment of the present invention, in step (4), the decoder processing specifically includes:

[0026] The decoder is constructed using an MLP, and a DenseNet-like bypass connection is added. The following five types of information are input into the decoder, and the cost volume is... The source point cloud features f(P), and the decoder output features estimated from the previous level after upsampling. Scene flow passed from the previous round Decoder output features passed from the previous round The superscript of a symbol indicates the round, and the subscript of a symbol indicates the level;

[0027]

[0028]

[0029] In one embodiment of the present invention, the cost body in step (4) specifically includes:

[0030] The target point cloud is warped using the scene stream predicted by the previous round at 1 / 4 resolution, and a cost volume is constructed based on the features of the source point cloud and the warped target point cloud.

[0031] According to another aspect of the present invention, a scene flow estimation system based on iterative pyramids is also provided, comprising a feature extraction module, a decoder module, a resampling module, and a guided optimization module, wherein:

[0032] The feature extraction module inputs two frames of point clouds from consecutive time points into the feature extraction network, sharing parameters. It downsamples using the furthest point sampling method while simultaneously encoding neighborhood features to construct a feature pyramid with resolutions of 1, 1 / 4, 1 / 16, 1 / 32, and 1 / 128. In addition to the original resolution, it upsamples the encoded features at each level and then concatenates them with the corresponding level features, effectively increasing the receptive field. The target point cloud is warped based on the predicted scene flow, and a cost volume is constructed based on the source point cloud features and the warped target point cloud features.

[0033] The decoder module receives input information including: a cost volume encoding flow motion information, features of the source point cloud containing context information, a predicted scene flow providing an initial prediction value, and predicted decoder features containing motion information. Then, an MLP is used to process the input information, outputting a predicted scene flow and decoder output features. In the first round, the predicted scene flow and decoder features are obtained by upsampling the predicted scene flow and decoder features from the previous level. In subsequent rounds, the predicted scene flow comes from the scene flow passed from the previous round, and the predicted decoder output features include the decoder features passed from the previous round and the upsampled decoder features from the previous level.

[0034] The resampling module resamples the target point cloud in the second and subsequent rounds to improve the correspondence between points. First, it warps the source point cloud at 1 / 4 resolution based on the scene flow estimated in the previous round. Then, it takes the warped point as the center and finds the nearest neighbor in the second frame point cloud at the original resolution based on distance. The found nearest neighbor is used as the new target point cloud at 1 / 4 resolution, and it is downsampled according to the downsampling index of the source point cloud obtained in the first round. After that, it extracts features from the new target point cloud for subsequent scene flow estimation.

[0035] The guided optimization module is used to transfer the high-precision estimation information from the previous round to the low-resolution level of the current round, providing a more accurate search center while supplementing motion information, continuing the high-precision scene flow estimation results and continuously refining them. After three iterations, the scene flow corresponding to the first frame point cloud at the original resolution is output as the final output result. The predicted scene flow results are constrained with the true labels. The entire loss function is divided into two parts: first, a multi-scale loss function is added to the estimation of each round for constraint, and then weights are assigned to different rounds for weighted summation. The multi-scale loss function assigns different weights to each level, and the specific calculation method is as follows:

[0036]

[0037] Where, γ l This represents the weights used at different resolution levels, with weight γ. lThe values ​​were set to γ0 = 0.02, γ1 = 0.04, γ2 = 0.08, γ3 = 0.16, and γ4 = 0.16. p i The corresponding predicted scene stream, Represents the true label, and ||*||2 represents the L2 norm;

[0038] The complete loss function is a weighted sum of the loss functions from each round, where the weights assigned to each round grow exponentially. The specific calculation method is as follows:

[0039]

[0040] Where γ = 0.8.

[0041] Compared with the prior art, the above-described technical solution conceived by this invention has the following beneficial effects:

[0042] (1) This invention provides a scene flow estimation system based on iterative pyramids, which effectively alleviates the problem that the search area cannot cover the true corresponding points due to the large deviation of the search center when estimating at low resolution levels. This system adopts layer-by-layer refinement and iterative optimization, thereby significantly improving the scene flow estimation accuracy;

[0043] (2) This invention provides a resampling mechanism that uses the scene flow predicted in the previous round as a guide to resample the target point cloud, which effectively improves the problem of poor correspondence between two frames of point clouds and alleviates the noise problem introduced by the input data.

[0044] (3) The scene flow estimation network IHNet based on iterative pyramids provided in this invention improves both accuracy and generalization ability. On the virtual scene public dataset FlyingThings3D, the method of this invention improves the EPE3D metric by 20.1% compared to the current state-of-the-art method. On the real scene public dataset KITTI, the method of this invention improves the EPE3D metric by 56.0% compared to the current state-of-the-art method. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of the IHNet network structure based on iterative pyramid scene flow estimation network in an embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram of the resampling mechanism in an embodiment of the present invention;

[0047] Figure 3 This is a visualization of the resampling results in an embodiment of the present invention;

[0048] Figure 4This is a visualization result on a public dataset in an embodiment of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0050] To address the cumulative error introduced by misestimation at low resolution in pyramid-based scene flow estimation systems, and the poor correspondence between point clouds in consecutive time frames due to point cloud sparsity, this invention provides an iterative pyramid-based scene flow estimation system. The system includes a feature extraction module, a decoder module, a resampling module, and a guided optimization module. High-precision estimation information from the previous sub-high resolution level is transferred to the current sub-low resolution level, providing a more accurate search center for scene flow estimation and supplementing motion information. Furthermore, the predicted scene flow from the previous sub-high resolution level guides the resampling of the target point cloud. Through distortion and nearest neighbor search based on distance, new target point clouds are obtained, effectively improving the problem of poor point correspondence. The new target point cloud is re-encoded, and then used to complete subsequent scene flow estimation. This system simultaneously employs layer-by-layer refinement and iterative optimization, significantly improving the accuracy of scene flow estimation. Compared with existing methods, this system ranks first in accuracy on the publicly available virtual scene dataset FlyingThings3D and the publicly available real-world scene dataset KITTI, demonstrating strong effectiveness and generalization ability.

[0051] like Figure 1 As shown, this invention provides a scene flow estimation method based on iterative pyramids, including:

[0052] (1) Take two frames of point cloud data at time t and t+1 before and after the time. The first frame is the source point cloud P and the second frame is the target point cloud Q. Input the two frames of point cloud data into the feature extraction network and use the feature extraction network to encode the features of the two frames of point cloud to construct a multi-resolution feature pyramid.

[0053] The feature extraction network specifically includes: for the original resolution level, performing three 1D convolutions with a kernel size of 1 and a ReLU activation function sequentially to encode features. Then, downsampling is performed based on the farthest point sampling to obtain the downsampled point cloud coordinates and corresponding indices. Simultaneously, feature encoding is performed using PointNet++ to encode neighborhood geometric information. For each point in the point cloud set, it is considered the center point, and its 16 nearest neighbors (i.e., neighborhood points S = 16) are found based on distance. Then, MLP processing is performed, namely 2D convolutions with a kernel size of 1, batch normalization, and a ReLU activation function, to obtain B*C*N. l The features of *S are then evaluated. Finally, the maximum value is taken in the neighborhood S dimension to complete the feature encoding at this level, outputting features B*C*N. l The above operations are then repeated to generate features at resolutions of 1 / 16, 1 / 32, and 1 / 128. Additionally, besides the original resolution levels, the features obtained at each level are upsampled and then concatenated with the features at the corresponding level, effectively increasing the receptive field. The final feature pyramid obtained by the feature extraction network has five levels from l0 to l4 (1, 1 / 4, 1 / 16, 1 / 32, and 1 / 128). Note that the above feature extraction process is repeated for the resampled target point cloud.

[0054] (2) Based on the multi-resolution feature pyramid, the first round of scene flow estimation is performed, starting from the coarsest resolution of 1 / 128 (flow initialized to zero). Each level is processed sequentially: warping, constructing the cost volume, inputting the cost volume and related information into the decoder, and the decoder estimating and outputting the predicted scene flow. Then, the process is refined layer by layer to the 1 / 4 resolution level. The information predicted by the previous level is upsampled and passed to the next level.

[0055] For each level of estimation, the target point cloud is warped using an upsampled scene stream. A cost volume is then constructed based on the features of the first frame and the warped second frame. This cost volume encodes flow motion information, providing valuable information for scene flow estimation. The cost volume, along with relevant information, is input into the scene flow decoder, which outputs decoder features and the predicted scene flow. During the layer-by-layer refinement process, information is passed between layers through upsampling. The upsampled scene stream provides a more accurate search center (in the form of a warp) for the current layer, and the upsampled decoder output features provide motion information with a larger receptive field for the current layer's estimation.

[0056] The upsampling process first uses an MLP to encode the direction vectors of the center point and neighborhood points, along with the corresponding decoder output features, to obtain weights. The MLP channel count is set to [8, 8, 16]. The input neighborhood information (B*C*S*N) is sequentially subjected to three 2D convolutions with a kernel size of 1, followed by a ReLU activation function. The obtained weights are then assigned to the corresponding neighborhood points, and the sparse point information (scene flow and decoder output features) is interpolated to obtain dense point information.

[0057] The decoder estimation described above uses an MLP to construct the decoder, where the number of MLP channels is set to [96, 64]. The decoder input is subjected to two 1D convolutions with a kernel size of 1 followed by a ReLU activation function. A DenseNet-like bypass connection is added to improve decoder performance. The decoder input includes the following four types: cost volume (CV), features f(P) from the first frame point cloud, and the upsampled scene stream up(S) estimated from the previous level. i ) and decoder output feature up(f l After MLP processing, the decoder outputs features containing motion information, which are then passed through a fully connected (FC) layer to output the estimated scene stream.

[0058] f l-1 =MLP([f(P),CV,up(S)) l ),up(f l (1)

[0059] S l-1 =FG(f l-1 (2).

[0060] (3) Use the scene flow predicted at the previous 1 / 4 resolution level to guide the resampling of the target point cloud, extract features from the resampled target point cloud, and use the extracted features for subsequent matching.

[0061] like Figure 2 As shown, to improve the poor correspondence between two frame point clouds, the source point cloud at 1 / 4 resolution is distorted using the high-resolution, high-precision prediction scene stream from the previous round. Then, the nearest neighbor is searched in the target point cloud at the original resolution, and the sampled points become the new 1 / 4 resolution target point cloud, thus correcting points with large deviations. Afterwards, based on the index returned during the downsampling process of the source point cloud in the first round of feature pyramid construction, downsampling is performed on the new 1 / 4 resolution target point cloud to obtain target point clouds at 1 / 16, 1 / 32, and 1 / 128 resolutions, thus completing the resampling of the target point cloud. Figure 3 The image shown is a visualization of the resampling results in an embodiment of the present invention. Features are then extracted from the new target point cloud for subsequent matching.

[0062] (4) Guidance-based scene flow estimation process for the second and subsequent rounds. Based on the index returned during the source point cloud downsampling process when constructing the feature pyramid in the first round, the decoder features and scene flow predicted at 1 / 4 resolution in the previous round are downsampled to obtain information at 1 / 16, 1 / 32, and 1 / 128 resolutions. This information is used to guide scene flow estimation at the corresponding resolution level in this round, providing a more accurate search center and supplementing motion information. For scene flow estimation at each level, the target point cloud is warped using the scene flow used for guidance. A cost body is constructed using the source point cloud features and the warped target point cloud features. The cost body and related information containing guidance information are input into the decoder for processing, and the decoder features and predicted scene flow are output. The decoder features and predicted scene flow are then upsampled and passed to the next level. The specific implementation of inputting the scene flow into the decoder for processing is as follows:

[0063] The decoder is constructed using an MLP, and a DenseNet-like bypass connection is added. The following five types of information are input into the decoder, and the cost volume is... The source point cloud features f(P), and the decoder output features estimated from the previous level after upsampling. Scene flow passed from the previous round Decoder output features passed from the previous round The superscript of a symbol indicates the round, and the subscript of a symbol indicates the level;

[0064]

[0065]

[0066] Furthermore, this invention provides a scene flow estimation system based on iterative pyramids, which mainly consists of four components: a feature extraction module, a decoder module, a resampling module, and a guided optimization module, the specific implementation of which is shown below:

[0067] (1) Feature extraction module: The point cloud data from two consecutive time frames is input into the feature extraction network, sharing parameters. Downsampling is performed using the farthest point sampling method, while simultaneously encoding neighborhood features to construct a feature pyramid. The resolutions are 1, 1 / 4, 1 / 16, 1 / 32, and 1 / 128. In addition to the original resolution, the encoded features at each level are upsampled and then concatenated with the corresponding level features to effectively increase the receptive field. The output feature of each level is B*C*N. l .

[0068] (2) Decoder module: The main network structure of the decoder uses MLP for convolutional processing and adds bypass connections from DenseNet. The input information of the decoder includes: cost volume encoding the motion information of the stream, features of the source point cloud containing contextual information, a predicted scene stream providing an initial prediction value, and predicted decoder features containing motion information. Then, the input information is processed by MLP to output the predicted scene stream and decoder output features. In the first round, the predicted scene stream and decoder features are obtained by upsampling the predicted scene stream and decoder features of the previous layer. In subsequent rounds, the predicted scene stream comes from the scene stream passed from the previous round. The predicted decoder output features include the decoder features passed from the previous round and the upsampled decoder features from the previous layer.

[0069] (3) Resampling module: This module resamples the target point cloud for the second and subsequent rounds to improve the correspondence between points. First, the source point cloud at 1 / 4 resolution is warped based on the scene flow estimated in the previous round. Then, the warped point is taken as the center, and the nearest neighbor is found in the target point cloud at the original resolution. The found nearest neighbor is used as the new target point cloud at 1 / 4 resolution. After that, features are re-extracted from the new point cloud for subsequent scene flow estimation.

[0070] (4) A guided optimization module is used to transfer the high-precision estimation information (scene flow and decoder features) from the previous round to the low-resolution level of the current round, providing accurate search center and motion information. The high-precision scene flow estimation results are continued and refined. After three iterations, the scene flow corresponding to the first frame point cloud at the original resolution is output as the final output result.

[0071] The predicted scene flow results are constrained against the true labels. The entire loss function is divided into two parts: first, a multi-scale loss function is added to the estimate of each round to constrain it, and then weights are assigned to different rounds and a weighted sum is performed.

[0072] The multi-scale loss function assigns different weights to each level, and the specific calculation method is as follows:

[0073]

[0074] Where, γ l This represents the weights used at different resolution levels, with weight γ. l The values ​​were set to γ0 = 0.02, γ1 = 0.04, γ2 = 0.08, γ3 = 0.16, and γ4 = 0.16. p i The corresponding predicted scene stream, This represents the real label. ||*||2 represents the L2 norm.

[0075] The complete loss function is a weighted sum of the loss functions from each round. The weights assigned to each round increase exponentially, and the specific calculation method is as follows:

[0076]

[0077] Where γ = 0.8.

[0078] Table 1 presents the quantitative evaluation of the method of the present invention and other methods on the FlyingThings3D and KITTI datasets in the embodiments of the present invention. As can be seen from Table 1, the method of the present invention outperforms other published methods on all metrics on FlyingThings3D and KITTI. Furthermore, as... Figure 4 As shown, visualization results on FlyingThings3D and KITTI demonstrate that the present invention is superior to other published methods.

[0079] Table 1 compares the scene flow estimation system based on iterative pyramid proposed in this invention with the state-of-the-art methods on two public datasets, FlyingThings3D and KITTI.

[0080]

[0081] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A scene flow estimation method based on iterative pyramids, characterized in that, The method includes the following steps: (1) Take two frames of point cloud data at time t and t+1 before and after the time. The first frame is the source point cloud P and the second frame is the target point cloud Q. Input the two frames of point cloud data into the feature extraction network and use the feature extraction network to encode the features of the two frames of point cloud to construct a multi-resolution feature pyramid. (2) Based on the multi-resolution feature pyramid, the first round of scene flow estimation is performed. The scene flow estimation starts from the coarsest resolution and is then refined layer by layer to the 1 / 4 resolution level. For each level of scene flow estimation, the cost body is constructed using the source point cloud features and the distorted target point cloud features. The cost body and related information are input into the decoder for processing. The decoder features and the predicted scene flow are output. The decoder features and the predicted scene flow are then upsampled and passed to the next level. (3) Use the scene flow predicted at the previous 1 / 4 resolution level to guide the resampling of the target point cloud, extract features from the resampled target point cloud, and use the extracted features for subsequent matching. (4) Based on the source point cloud and the resampled target point cloud, scene flow estimation is performed in the second round and subsequent rounds; the scene flow predicted at the 1 / 4 resolution level in the previous round and the decoder output features are downsampled to 1 / 16, 1 / 32 and 1 / 128 resolutions to guide scene flow estimation at the corresponding resolution level in this round, providing a more accurate search center and supplementing motion information; then, based on the above information, scene flow estimation is performed in iterative rounds, and after being processed by the decoder, the scene flow predicted at the original resolution level is finally output.

2. The scene flow estimation method based on iterative pyramids as described in claim 1, characterized in that, In step (1), the process of using a feature extraction network to encode the features of two frames of point clouds to construct a multi-resolution feature pyramid specifically includes: For the original resolution level l0, three 1D convolutions with a kernel size of 1 and a ReLU activation function are performed sequentially to encode features. Then, based on the farthest point sampling, the feature is downsampled to a 1 / 4 resolution level l1 to obtain the downsampled point cloud coordinates and corresponding indices. Feature encoding is performed using PointNet++. For each point in the point cloud, it is considered as the center point, and the 16 nearest points are found based on distance. Then, MLP processing is performed, namely 2D convolutions with a kernel size of 1, batch normalization, and a ReLU activation function. Finally, max pooling is performed in the neighborhood S dimension to complete the feature encoding at the 1 / 4 resolution level l1. The above operations are then repeated to generate features at resolutions of 1 / 16, 1 / 32, and 1 / 128. In addition, except for the original resolution level l0, the features obtained at each level are upsampled and then concatenated with the features of the corresponding level to effectively increase the receptive field. The final generated feature pyramid has five levels l0-l4, with resolutions of 1, 1 / 4, 1 / 16, 1 / 32, and 1 / 128, respectively.

3. The scene flow estimation method based on iterative pyramids as described in claim 1, characterized in that, In step (2), the scene flow estimation at each level and the layer-by-layer refinement are specifically as follows: The target point cloud is warped using the upsampled scene stream. A cost volume is constructed based on the features of the source point cloud and the warped target point cloud. The cost volume and related information are input into the scene stream decoder and processed using MLP. The decoder features and the predicted scene stream are output. During the layer-by-layer refinement process, information is passed between layers through the upsampling module. The upsampled scene stream provides a center for the search region for the current layer estimation, and the upsampled decoder features provide motion information with a larger receptive field for the current layer estimation.

4. The scene flow estimation method based on iterative pyramids as described in claim 1 or 3, characterized in that, In step (2), the upsampling specifically refers to: The MLP is used to encode the direction vectors of the center point and neighborhood points and the corresponding decoder output features to generate weights. The number of MLP channels is set to [8,8,16]. The input neighborhood information B*C*S*N is subjected to three 2D convolutions with a kernel size of 1 and a ReLU activation function in sequence. The weights are assigned to the corresponding neighborhood points, and then the sparse point information is converted into dense point information by interpolation.

5. The scene flow estimation method based on iterative pyramids as described in claim 1, characterized in that, In step (2), the input of the cost body and related information into the decoder for processing specifically includes: The decoder is constructed using an MLP with the number of MLP channels set to [96, 64]. The decoder input is subjected to two 1D convolutions with a kernel size of 1 followed by a ReLU activation function. A DenseNet-like bypass connection is also added. The decoder input includes the following four types: cost volume (CV), features f(P) from the first frame point cloud, and the upsampled scene flow up(S) estimated from the previous level. l ) and decoder output feature up(f l After MLP processing, the decoder outputs features containing motion information, which are then passed through a fully connected (FC) layer to output the estimated scene stream. f l-1 =LP([f(P),V,up(S l ),p(f l )])(1) S l-1 =C(f l-1 )(2)。 6. The scene flow estimation method based on iterative pyramids as described in claim 1, characterized in that, In step (3), the resampling of the target point cloud specifically involves: To improve the correspondence between the two point clouds, the coordinates of the target point cloud are adjusted to better correspond with the source point cloud. At 1 / 4 resolution, the source point cloud is distorted using the high-precision predicted scene stream from the previous round. Then, the distorted source point cloud is used to search for the nearest neighbor in the original resolution target point cloud based on distance. The found nearest neighbor is used as the new 1 / 4 resolution target point cloud to correct points with large deviations. Afterward, based on the index returned during the downsampling process of the source point cloud when constructing the feature pyramid in the first round, downsampling is performed on the new 1 / 4 resolution target point cloud to obtain target point clouds at 1 / 16, 1 / 32, and 1 / 128 resolutions. This completes the resampling of the target point cloud. The adjusted target point cloud is then re-extracted for features, and neighborhood features are encoded for subsequent matching.

7. The scene flow estimation method based on iterative pyramids as described in claim 1, characterized in that, In step (4), the scene flow estimation for the iterative rounds specifically includes: First, based on the index returned during the source point cloud downsampling process when constructing the feature pyramid in the first round, the decoder features and scene flow predicted at 1 / 4 resolution in the previous round are downsampled to obtain information at 1 / 16, 1 / 32, and 1 / 128 resolutions, which are used to guide the scene flow estimation at the corresponding resolution levels in this round. Among them, the high-precision estimated scene flow provides a more accurate search center for the estimation at the low resolution level in this round, effectively alleviating the problem that the search area cannot cover the true corresponding points. The decoder features estimated by high precision from the previous round further supplement the motion information. Based on the above information, the scene flow estimation of the iterative rounds is guided. After being processed by the decoder, the scene flow predicted at the original resolution level is finally output.

8. The scene flow estimation method based on iterative pyramids as described in claim 1 or 7, characterized in that, In step (4), the decoder processing specifically includes: The decoder is constructed using an MLP, and a DenseNet-like bypass connection is added. The following five types of information are input into the decoder, and the cost volume is... The source point cloud features f(P) and the decoder output features up(f) estimated from the previous level after upsampling. l k ), the scene flow passed from the previous round Decoder output features passed from the previous round The superscript of a symbol indicates the round, and the subscript of a symbol indicates the level; 9. The scene flow estimation method based on iterative pyramids as described in claim 8, characterized in that, The cost body in step (4) is specifically: The target point cloud is warped using the scene stream predicted by the previous round at 1 / 4 resolution, and a cost volume is constructed based on the features of the source point cloud and the warped target point cloud.

10. A scene flow estimation system based on iterative pyramids, characterized in that, It includes a feature extraction module, a decoder module, a resampling module, and a guided optimization module, wherein: The feature extraction module inputs two frames of point clouds from consecutive time points into the feature extraction network, shares parameters, downsamples using the farthest point sampling method while encoding neighborhood features, and constructs a feature pyramid with resolutions of 1, 1 / 4, 1 / 16, 1 / 32, and 1 / 128. In addition to the original resolution, the encoded features at each level are upsampled and then concatenated with the corresponding level features to effectively increase the receptive field. The target point cloud is distorted based on the predicted scene flow, and a cost volume is constructed based on the source point cloud features and the distorted target point cloud features. The decoder module receives input information including: a cost volume encoding flow motion information, features of the source point cloud containing context information, a predicted scene flow providing an initial prediction value, and predicted decoder features containing motion information. Then, an MLP is used to process the input information, outputting a predicted scene flow and decoder output features. In the first round, the predicted scene flow and decoder features are obtained by upsampling the predicted scene flow and decoder features from the previous level. In subsequent rounds, the predicted scene flow comes from the scene flow passed from the previous round, and the predicted decoder output features include the decoder features passed from the previous round and the upsampled decoder features from the previous level. The resampling module resamples the target point cloud in the second and subsequent rounds to improve the correspondence between points. First, it warps the source point cloud at 1 / 4 resolution based on the scene flow estimated in the previous round. Then, it takes the warped point as the center and finds the nearest neighbor in the second frame point cloud at the original resolution based on distance. The found nearest neighbor is used as the new target point cloud at 1 / 4 resolution, and it is downsampled according to the downsampling index of the source point cloud obtained in the first round. After that, it extracts features from the new target point cloud for subsequent scene flow estimation. The guided optimization module is used to transfer the high-precision estimation information from the previous round to the low-resolution level of the current round, providing a more accurate search center while supplementing motion information, continuing the high-precision scene flow estimation results and continuously refining them. After three iterations, the scene flow corresponding to the first frame point cloud at the original resolution is output as the final output result. The predicted scene flow results are constrained with the true labels. The entire loss function is divided into two parts: first, a multi-scale loss function is added to the estimation of each round for constraint, and then weights are assigned to different rounds for weighted summation. The multi-scale loss function assigns different weights to each level, and the specific calculation method is as follows: Where, γ l This represents the weights used at different resolution levels, with weight γ. l The values ​​were set to γ0 = 0.02, γ1 = 0.04, γ2 = 0.08, γ3 = 0.16, and γ4 = 0.

16. p i The corresponding predicted scene stream, Represents the true label, and ||*||2 represents the L2 norm; The complete loss function is a weighted sum of the loss functions from each round, where the weights assigned to each round grow exponentially. The specific calculation method is as follows: Where γ = 0.8.

Images