A point cloud scene flow estimation method, system, medium and program product based on a flow matching model

Abstract

The application belongs to the field of computer vision, and constructs an FMFlow3D network, including a matching feature extraction network, a flow embedding layer and an iterative updating network; firstly, matching features and coordinates of two continuous frames of point clouds are extracted, and motion features and initial flow are obtained by calculating cross-frame correlation through the flow embedding layer; an objective velocity field is constructed with the initial flow as a starting distribution and a scene flow true value as a target distribution, and an intermediate state distribution, motion features and matching features are input into the iterative updating network to predict an estimated velocity field; the network is supervised and trained end to end through a joint loss function; in inference, two frames of point clouds are input into the trained network, and the final scene flow is obtained by solving a first-order ordinary differential equation through the estimated velocity field. The application redefines scene flow estimation as a generation task, directly learns a time-varying vector field from source point clouds to target point clouds, avoids the limitations of traditional discrete matching, and significantly improves the accuracy of motion estimation and the adaptability to complex dynamic scenes.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to a point cloud scene flow estimation method, system, medium, and program product based on a flow matching model. Background Technology

[0002] Point cloud scene flow estimation, a long-standing and challenging problem in computer vision, plays an indispensable role in building high-level semantic cognition and directly empowers many downstream tasks such as autonomous driving and robot navigation. In recent years, with the vigorous development and comprehensive penetration of the deep learning paradigm, using neural network models to model the temporal evolution of point clouds end-to-end and overcome the challenges of scene flow estimation has gradually become the mainstream technical approach in academia and industry. Among them, PointPWC-Net and RAFT-3D, as the most representative breakthrough works based on CNN architecture, each demonstrate unique design philosophies. PointPWC-Net pioneered the construction of a multi-level local cost volume that integrates distorted features. Through a coarse-to-fine pyramid structure, it estimates the dense offset field between adjacent point clouds level by level, achieving a balance between efficient computation and representation learning in the hierarchical refinement. RAFT-3D takes a different approach, proposing an iterative inference framework: it first pre-constructs a multi-scale four-dimensional correlation volume across all point pairs, embedding geometric similarity and feature correlation into a unified high-dimensional space; then it deploys a gated recurrent unit (GRU) as the core engine for hidden state propagation, continuously optimizing optical flow prediction through dozens of iterative steps. However, these models cannot explicitly capture the potential probability distribution of dense scene flows, thus making it difficult to handle complex motions.

[0003] Recently, with the rise of diffusion-generative models in visual tasks, their proficiency in learning the integrated joint probability distribution offers a solution to these challenges. Therefore, this invention provides a point cloud scene flow estimation method based on a flow matching model, which combines an iterative inference framework with a flow matching model training paradigm.

[0004] This invention constructs a point cloud scene flow estimation model, 3DFM-Flow, based on a flow matching model. Compared to the denoising probabilistic model, the flow matching training paradigm adopted by this model requires fewer sampling steps and has a faster inference speed. Furthermore, flow matching directly learns a time-varying vector field that maps the source point cloud to the target point cloud, avoiding the coarse assumption of traditional methods that discretize motion into inter-frame matching. This continuous-time modeling can more accurately capture more complex motions. Summary of the Invention

[0005] The purpose of this invention is to provide a point cloud scene flow estimation method, system, medium, and program product based on a flow matching model.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] A point cloud scene flow estimation method based on a flow matching model, the specific steps of which are as follows:

[0008] Step 1: Obtain two consecutive frames of point cloud data and scene flow ground truth.

[0009] Step 2: Construct the point cloud scene flow estimation network FMFlow3D based on the flow matching model, including a matching feature extraction network, a flow embedding layer for calculating cross-frame correlation, and an iterative update network;

[0010] Step 3: Extract matching features and point cloud coordinates of two consecutive frames of point clouds through the feature extraction network, and then calculate cross-frame correlation based on the matching features and point cloud coordinates through the flow embedding layer to obtain motion features and initial flow;

[0011] Step 4: Using the initial flow as the starting distribution and the scene flow ground truth as the target distribution, construct the intermediate state distribution and the target velocity field; input the intermediate state distribution, motion features, and matching features into the iterative update network based on the flow matching model training paradigm to predict and estimate the velocity field;

[0012] Step 5: Construct a joint loss function, including the L2 norm between the ground truth of the scene flow and the estimated scene flow obtained by integrating the estimated velocity field, and the mean square error between the target velocity field and the estimated velocity field; perform end-to-end supervised training of the 3DFM-Flow network based on the joint loss function;

[0013] Step 6: Inference phase. Input two consecutive frames of point cloud images into the trained 3DFM-Flow network, estimate the velocity field and obtain the final estimated scene flow by solving the first-order ordinary differential equation.

[0014] Furthermore, in step 3, the point clouds at two consecutive time points are... and The input is fed into a feature extraction network, and through point convolutional layers, point cloud matching features are obtained. , and point cloud coordinates , .

[0015] Furthermore, in step 3, the matching features are obtained by calculating the cross-frame correlation of the streaming embedding layer. and Calculating cosine similarity captures semantic associations for point cloud coordinates. and Calculate the Euclidean distance to find the nearest point and capture the spatial structure, then construct domain features and obtain motion features through a multilayer perceptron. and initial flow ,

[0016]

[0017]

[0018]

[0019]

[0020]

[0021] in, 1D convolution and As an intermediate feature, and For indexing, These are operations for querying cosine similarity and calculating Euclidean distance, respectively. Geometric features For aggregation operations, This is an operation to splice features along the channel. For 2D convolution, It is a multilayer perceptron.

[0022] Furthermore, the training paradigm for stream matching in step 4 is specifically as follows:

[0023]

[0024]

[0025] in, Given a known initial distribution, For the target distribution, The distribution represents an intermediate state, where T is a constant that is randomly sampled between 0 and 1.

[0026] Will Motion characteristics and matching features Input is fed into an iterative update network to predict and estimate the velocity field. ;

[0027]

[0028] ConvGRU is an iterative update network.

[0029] Furthermore, the joint loss function in step 5 for:

[0030]

[0031]

[0032] in, For scene flow truth value The estimated scene flow obtained by integrating the estimated velocity field The L2 norm between them It is a constant. For the target velocity field and estimated velocity field The mean square error between them.

[0033] Furthermore, the estimated scene flow for:

[0034] .

[0035] Furthermore, the reasoning stage in step 6 The final estimated scene flow is obtained by solving the first-order ordinary differential equation.

[0036]

[0037]

[0038] in, This is the iterative update module after training, where K is the number of sampling steps. For intermediate flow, Let dt be the intermediate velocity field, and dt be the reciprocal of the number of sampling steps.

[0039] A computer system includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of a point cloud scene flow estimation method based on a flow matching model.

[0040] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a point cloud scene flow estimation method based on a flow matching model.

[0041] A computer program product includes a computer program that, when executed by a processor, implements steps of a point cloud scene flow estimation method based on a flow matching model.

[0042] The beneficial effects of this invention are as follows:

[0043] The proposed flow matching training paradigm has fewer sampling steps and faster inference speed compared to the denoising probability model. A point cloud scene flow estimation model, 3DFM-Flow, based on the flow matching model, is proposed. This model rethinks the scene flow task from a regression task to a generation task, directly learning a time-varying vector field that maps source point clouds to target point clouds. This avoids the coarse assumption of traditional methods that discretize motion into inter-frame matching, and achieves higher accuracy in capturing the potential probability distribution of dense scene flows. Attached Figure Description

[0044] Figure 1 This is a flowchart of the present invention;

[0045] Figure 2 This is a structural diagram of a point cloud scene flow estimation model based on the flow matching training paradigm;

[0046] Figure 3 PointNet structure diagram;

[0047] Figure 4 This is a diagram of the flow embedding layer structure. Detailed Implementation

[0048] The present invention will now be further described with reference to the accompanying drawings.

[0049] This invention discloses a point cloud scene flow estimation method, system, medium, and program product based on a flow matching model. The method includes the following steps:

[0050] Step 1: Construct a point cloud scene flow estimation network model 3DFM-Flow based on a flow matching model. The 3DFM-Flow network model includes a matching feature extraction network, a flow embedding layer for calculating cross-frame correlation, and an iterative update network.

[0051] Step 2: Prepare point cloud sequences and ground truth values ​​for point cloud scene flow network training and testing, including point clouds at times t and t+1. , and the truth value of the scene flow at the current moment .

[0052] Step 3: Construct a matching feature extraction network.

[0053] The network can be any point cloud visual representation network, such as PointNet, PointNet++, or similar feature extraction networks. Here, PointNet is used as an example. Figure 3As shown, this feature extraction network consists of three Point ConV layers. The input point cloud size is (N, 3), where N is the number of points, and the output feature size is (N / 128, 3+256). The number of feature channels in each Point ConV layer is (3+64), (3+128), and (3+256), corresponding to N / 4, N / 16, and N / 128 feature points in the point cloud, respectively. The PointNet-based feature extraction network processes two frames of point clouds through point convolutional layers. , Feature extraction is performed to obtain point cloud matching features. , and point cloud coordinates , Its operation is defined as follows:

[0054] =PointNet( )

[0055] =PointNet( )

[0056] PointNet ( This is a feature extraction network based on PointNet. and Given two consecutive point cloud frames as input, for The matching features of the first frame obtained through the feature extraction network. for The coordinates of the first frame obtained through the feature extraction network. for The matching features of the second frame obtained through the feature extraction network. for The coordinates of the second frame obtained through a feature extraction network.

[0057] Step 4: Construct a stream embedding layer to compute cross-frame correlations.

[0058] like Figure 4 As shown, the streaming embedding layer that calculates cross-frame correlation is used to match features. and First, a 1D convolution layer is applied, then cosine similarity is calculated to capture semantic associations for point cloud coordinates. and Calculate Euclidean distance to find relevant points and capture spatial structure, then construct domain features and obtain motion features through a multilayer perceptron. and initial flow Its operation is defined as follows:

[0059]

[0060]

[0061]

[0062]

[0063]

[0064] in, For 1D convolution, These are operations for querying cosine similarity and calculating Euclidean distance, respectively. For aggregation operations, This is an operation to splice features along the channel. For 2D convolution, It is a multilayer perceptron.

[0065] Step 5: Construct an iterative update network based on the flow matching model training paradigm.

[0066] like Figure 2 As shown, let the initial distribution be the initial flow, and the target distribution be the current scene flow truth value. The intermediate state is obtained through a linear transformation. The distribution of intermediate states at time t is defined by the following operation:

[0067]

[0068]

[0069] in Given a known initial distribution, For the target distribution, The distribution represents the intermediate state, and T is a constant that is randomly sampled between 0 and 1.

[0070] During the training phase Motion characteristics and matching features In the ConvGRU iterative update module along the channel combination input, the initial distribution is predicted. To target distribution Estimated velocity field between Its operation is defined as follows:

[0071]

[0072]

[0073] Step Six: Construct the total loss function for the 3DFM-Flow network. This includes the estimated scene flow obtained by integrating the estimated velocity field. The model is constrained by the L2 norm between the target velocity field and the mean square error between the estimated velocity field and the L2 norm. Its operation is defined as follows:

[0074]

[0075]

[0076]

[0077] in, Scene flow truth value The estimated scene flow obtained by integrating the estimated velocity field The L2 norm between them For the final loss function, It is a constant. For the target velocity field and estimated velocity field The mean squared error between them, where MSE is the mean squared error.

[0078] The estimated scene flow for:

[0079] .

[0080] Step Seven: Reasoning Stage The final estimated scene flow is obtained by solving the first-order ordinary differential equation.

[0081]

[0082]

[0083] in, This is the iterative update module after training, where K is the number of sampling steps. For intermediate flow, Let dt be the intermediate velocity field, and dt be the reciprocal of the number of sampling steps.

[0084] To verify the effectiveness of the method, the error metrics obtained through inference evaluation of 3DFM-Flow, Self-Mono-SF, and Multi-Mono-SF on the KITTI2015 dataset were used to evaluate the method's performance.

[0085] Table 1. Comparison of scene flow estimation accuracy of different methods on KITTI 2015

[0086]

[0087] As shown in Table 1, 3DFM-Flow achieved the minimum error across all listed methods, which fully demonstrates the effectiveness of the 3DFM-Flow scheme.

[0088] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A point cloud scene flow estimation method based on a flow matching model, characterized in that: The specific steps are as follows: Step 1: Obtain two consecutive frames of point cloud data and scene flow ground truth. Step 2: Construct the point cloud scene flow estimation network FMFlow3D based on the flow matching model, including a matching feature extraction network, a flow embedding layer for calculating cross-frame correlation, and an iterative update network; Step 3: Extract matching features and point cloud coordinates of two consecutive frames of point clouds through the feature extraction network, and then calculate cross-frame correlation based on the matching features and point cloud coordinates through the flow embedding layer to obtain motion features and initial flow; Step 4: Using the initial flow as the starting distribution and the scene flow ground truth as the target distribution, construct the intermediate state distribution and the target velocity field; input the intermediate state distribution, motion features, and matching features into the iterative update network based on the flow matching model training paradigm to predict and estimate the velocity field; Step 5: Construct a joint loss function, including the L2 norm between the ground truth of the scene flow and the estimated scene flow obtained by integrating the estimated velocity field, and the mean square error between the target velocity field and the estimated velocity field; perform end-to-end supervised training of the FMFlow3D network based on the joint loss function; Step 6: Inference phase. Input two consecutive frames of point cloud images into the trained FMFlow3D network, estimate the velocity field and obtain the final estimated scene flow by solving the first-order ordinary differential equation.

2. The point cloud scene flow estimation method based on the flow matching diffusion model according to claim 1, characterized in that: In step 3, the point clouds at two consecutive time points are... and The input is fed into a feature extraction network, and through point convolutional layers, point cloud matching features are obtained. , and point cloud coordinates , .

3. The point cloud scene flow estimation method based on the flow matching diffusion model according to claim 2, characterized in that: In step 3, the matching features are obtained by calculating the cross-frame correlation of the stream embedding layer. and Calculating cosine similarity captures semantic associations for point cloud coordinates. and Calculate the Euclidean distance to find the nearest point and capture the spatial structure, then construct domain features and obtain motion features through a multilayer perceptron. and initial flow , in, 1D convolution and As an intermediate feature, and For indexing, These are operations for querying cosine similarity and calculating Euclidean distance, respectively. Geometric features For aggregation operations, This is an operation to splice features along the channel. For 2D convolution, It is a multilayer perceptron.

4. The point cloud scene flow estimation method based on the flow matching diffusion model according to claim 3, characterized in that: The training paradigm for stream matching in step 4 is as follows: in, Given a known initial distribution, For the target distribution, The distribution represents an intermediate state, where T is a constant that is randomly sampled between 0 and 1. Will Motion characteristics and matching features Input is fed into an iterative update network to predict and estimate the velocity field. ; ConvGRU is an iterative update network.

5. The point cloud scene flow estimation method based on the flow matching diffusion model according to claim 4, characterized in that: The joint loss function in step 5 for: in, For scene flow truth value The estimated scene flow obtained by integrating the estimated velocity field The L2 norm between them It is a constant. For the target velocity field and estimated velocity field The mean square error between them.

6. The point cloud scene flow estimation method based on the flow matching diffusion model according to claim 5, characterized in that: The estimated scene flow for: 。 7. The point cloud scene flow estimation method based on the flow matching diffusion model according to claim 6, characterized in that: The reasoning stage in step 6 The final estimated scene flow is obtained by solving the first-order ordinary differential equation. in, This is the iterative update module after training, where K is the number of sampling steps. For intermediate flow, Let dt be the intermediate velocity field, and dt be the reciprocal of the number of sampling steps.

8. A computer system comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method of claim 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the steps of the method of claim 7.

10. A computer program product, comprising a computer program, characterized in that: When the computer program is executed by a processor, it implements the steps of the method of claim 7.

Images