Scene flow estimation method based on large displacement decomposition

Abstract

The present invention proposes a scene flow estimation method based on large displacement decomposition. This method combines a pyramid deformation network with scene flow estimation, can decompose large displacement motion into small displacement motion and obtain scene flow through self-supervised learning, thereby improving the robustness of the model to large displacement motion in the scene. The proposed model is named (L3DFlow), which can adapt to large displacement motion in the scene and obtain more accurate scene flow information from the input 3D point cloud data. When estimating the scene flow, the 3D point cloud with a unit time interval is input, and the scene flow can be obtained by forward calculation, which improves the solution speed and can meet the needs of real-time applications.

Description

Technical Field

[0001] The present invention relates to a scene flow estimation method, in particular to a scene flow estimation method based on large displacement decomposition. Background Art

[0002] Scene flow is a three-dimensional extension of optical flow, and this field is currently one of the important research directions in computer vision. Compared with optical flow, scene flow can provide more sufficient motion information. It can provide rich underlying three-dimensional motion information for upper-level visual understanding tasks and vehicle control systems. This information helps the vehicle make subsequent judgments and decisions. However, in actual autonomous driving environments, vehicles often travel at high speeds, resulting in a large number of large displacements. Large displacement motions usually have the characteristics of large spatial spans, making it difficult for the model to accurately learn the correspondence between related points, which in turn affects the effectiveness of motion estimation. Therefore, how to effectively model large displacement motion relationships and fully acquire large displacement motion knowledge when the vehicle is driving at high speed is a major difficulty that needs to be solved in conducting research related to motion clue mining in autonomous driving scenarios.

[0003] To address these issues, this paper proposes a scene flow estimation method that utilizes a pyramidal deformation network to decompose large displacements. This method combines a pyramidal deformation network with scene flow estimation, decomposing large displacements into smaller ones and deriving the scene flow through self-supervised learning. This improves the model's robustness to large displacements within the scene. When estimating the scene flow, a 3D point cloud with unit time intervals is input, and the scene flow can be derived through a simple forward computation, meeting the requirements of real-time applications. Summary of the Invention

[0004] The purpose of this paper is to propose a scene flow estimation method based on large displacement decomposition. This method integrates a pyramid-shaped deformable network with a self-supervised learning algorithm to propose a new network architecture, named (L3DFlow), which performs end-to-end scene flow self-supervised learning. At the same time, a new loss function is proposed to guide the network training process.

[0005] To achieve the above objectives, the present invention provides a method for scene flow estimation based on large displacement decomposition. This method proposes a new pyramid-shaped network, named (L3DFlow), to decompose large displacements in the scene. Furthermore, the pyramid-shaped network is integrated with a self-supervised algorithm to perform end-to-end self-supervised learning of scene flow. The method comprises the following steps:

[0006] S1. First, prepare the 3D point cloud sequence for scene flow network training.

[0007] S2. Use a three-layer convolutional neural network to extract the features of point clouds at two adjacent moments.

[0008] S3. Match point cloud features to obtain point cloud registration information. Use cosine similarity to measure the matching degree between two points on the point cloud features obtained in S2. Use the optimal transportation model combined with the Sinkhorn algorithm to obtain the correlation between point clouds.

[0009] S4. Subtract the correlation between the source point and the point cloud to obtain the initial flow of the scene flow.

[0010] S5. Use the above steps to build a pyramid model. S2 to S4 are the processing of each layer of the pyramid. Sampling the points through sine coding in each k-th layer samples some points X. k Input to the kth level of the pyramid.

[0011] S6. After S1 to S4, the initial flow F of some points is obtained, and F is upsampled to obtain F', so that the number of flows in F is the same as the point X sampled after sine coding in the next layer. k+1 If the number is the same, add F' and the next layer of sampling points, input them to the k+1 layer of the pyramid, and repeat S2 to S6 until the number of sampling points is the same as the number of source points.

[0012] S7. Calculate the loss function of the model, use the confidence-aware distance loss function and the flow smoothness loss function to calculate the loss in each layer, and finally add the losses through the sampling relationship to obtain the total training objective function.

[0013] S8. Experiment and verification of the model. The source point cloud and the target point cloud are input into the model, and the output is the desired estimated scene flow.

[0014] Compared with the prior art, the advantages of the present invention are as follows:

[0015] Existing technologies mainly target two-dimensional image data and solve the problem of large displacement motion on a plane, while ignoring the three-dimensional motion characteristics of objects in the real world. To address the problem of insufficient perception of three-dimensional large displacement motion, the present invention designs a scene flow estimation method based on large displacement decomposition. By constructing a pyramid deformation model, the point cloud is decomposed into multiple parts for solution, thereby decomposing large displacement into small displacement solutions. While ensuring the quality of local motion details, the span range of feature interaction is greatly expanded, thereby improving the ability to extract knowledge of three-dimensional large displacement motion. In addition, the present invention designs a self-supervised scene flow learning method, which effectively alleviates the problem of scarce labeled samples by defining the self-supervised learning objective function of each layer in the pyramid model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a schematic diagram of a pyramid-shaped network;

[0017] Figure 2 Schematic diagram of the self-supervised algorithm flow. DETAILED DESCRIPTION

[0018] In order to deepen the understanding of the present invention, the present embodiment is described in detail below with reference to the accompanying drawings. The present invention is described in more detail below.

[0019] Embodiment: A scene flow estimation method based on large displacement decomposition, the method comprising the following steps:

[0020] S1. First, a lidar sensor is used to capture 3D point clouds that are prepared for scene flow network training.

[0021] S2. Figure 2 As shown, the present invention uses a three-layer neural network to obtain the point cloud features extracted above. First, in the first layer, the present invention uses iterative farthest point sampling to select a subset of points, so that X ij Is the point farthest from the other points in the set. In the second layer, the points are grouped by spherical query. In the third layer, local feature extraction is performed on the local area of ​​each point set. The above point cloud feature extraction operation is performed on both the source point X and the target point Y to obtain the feature Φ of the two point clouds. X ,Φ Y ∈R d .

[0022] S3. Point cloud feature matching to obtain point cloud corresponding registration. For the point features obtained in step S2, first calculate the cosine similarity in the learning feature space:

[0023]

[0024] in Φ corresponding to the i-th and j-th rows respectively X and Φ Y In order to avoid the points between streams being too far apart, which would cause the estimation accuracy to decrease, the cost for points with Euclidean distance less than 10 meters is set to:

[0025] C ij =1-S ij (2)

[0026] The cost of the remaining points is set to ∞, at which point the estimated Euclidean distance between flows is controlled within 10 meters.

[0027] Next, the present invention uses the optimal transmission problem model to obtain the correspondence between the source and target point clouds. The optimal transmission problem describes how to find the most similar relationship between two distributions by minimizing the cost of conversion or mapping. The quality of each source point is set to According to the definition of the model, it is necessary to find a transmission scheme that transfers the mass to the target point cloud with the least overall cost. To this end, the present invention uses the Kantorovich relaxation optimal transmission model:

[0028]

[0029] Among them C ij ≥0 comes from formula (2), T ij ≥0 is the quality of the transfer, and the parameters ε,λ≥0 control the relaxation problem. 1 n ∈R n is a vector with all terms equal to 1, KL is the Kullback-Leibler divergence, which is used to soft-preserve the transmission quality between point clouds. In the second term of formula (3), the Sinkhorn algorithm is used to estimate the optimal transmission matrix from C to T*, thereby obtaining the correlation relationship between point clouds.

[0030] S4. For x i ∈X, first calculate the weight of each point,

[0031]

[0032] in is included with the highest quality transmission of{y j k at point s Neighborhood of indexes. Point to x i The soft correlation can be expressed by the following formula:

[0033]

[0034] and point x i The initial flow is:

[0035]

[0036] The initial flow field is:

[0037]

[0038] in Contains All points.

[0039] S5. Figure 1 As shown in the figure, a pyramid deformation model is constructed to decompose the point cloud into multiple parts for solution, thereby decomposing the large displacement into small displacements for solution. The pyramid deformation model can be expressed as Δ={(Γ k ,Θ k )|k=1,…,m}, where each pyramid level contains a pair of continuous functions (Γk,Θk) and m is the total number of levels.

[0040] In the kth layer of the pyramid, Γk is the position encoding function, which encodes the input point from the previous layer through sinusoidal encoding. Mapping:

[0041]

[0042] Where k0 is a constant that controls the initial sampling frequency of the first layer of the pyramid. The frequency of the sine function is crucial for motion decomposition: the lower the frequency, the fewer the sampling points, and the higher the frequency, the more the sampling points, until the number of sampling points is equal to the number of original points.

[0043] At the k+1th level of the pyramid, Θ k For steps S2 to S4. Input the sinusoidally encoded point set into Θ k In the example above, we get the initial flow F.

[0044] S6. After S1 to S4, the initial flow F of some points is obtained. F is upsampled using bilinear interpolation. F is added to the input point set X after k+1 layer encoding to obtain a point cloud X' that is closer to the target point. X' is input into Θ k+1 Repeat S2 to S6 until the number of points after sampling is equal to the number of original points.

[0045] S7. Calculate the loss function in each layer of the pyramid model. The present invention defines the matching confidence for each point based on correlation:

[0046] The value range of is [-1,1]. In order to obtain a value between 0 and 1, the negative value is removed during the processing and the matching x is set. i The matching confidence is Can be achieved through The distance loss is expressed as:

[0047]

[0048] In order to maintain the geometric structure of the source point cloud, the flow field is expected to be smooth. That is, adjacent source points should have similar flow predictions. Therefore, the present invention uses flow smoothness loss to regulate the learning process:

[0049]

[0050] where N X (x i ) is the radius k in X f And x i The Euclidean neighborhood centered on . Therefore, the overall training objective function of each layer can be expressed as:

[0051]

[0052] For the overall loss function of the pyramid model, the loss function of each layer can be summed:

[0053]

[0054] S8. Validate the test model. Input the source point cloud and target point cloud into the model, and the output is the desired estimated scene flow.

[0055] It should be noted that the above embodiments are not intended to limit the scope of protection of the present invention, and equivalent changes or substitutions made on the basis of the above technical solutions fall within the scope of protection of the claims of the present invention.

Claims

1. A scene flow estimation method based on large displacement decomposition, characterized in that: The method comprises the following steps: S1. First prepare the 3D point cloud sequence for scene flow network training, S2. Use a three-layer convolutional neural network to extract the features of point clouds at two adjacent moments. S3. Point cloud feature matching, obtain the corresponding registration information of the point cloud, use the cosine similarity of the point cloud features obtained in S2 to measure the matching degree between two points, use the optimal transfer model and combine it with the Sinkhorn algorithm to obtain the correlation between the point clouds, S4. Subtract the correlation between the source point and the point cloud to obtain the initial flow, S5. Use the above steps to build a pyramid model. S2 to S4 are the processing of each layer of the pyramid. The points in the space are sampled by sine coding and some points X are sampled in each k layer. k Input to the kth level of the pyramid, S6. After S1 to S4, the initial flow F of some points is obtained, and F is upsampled to obtain F', so that the number of flows in F is the same as the point X sampled after sine coding in the next layer. k+1 The number is the same, add F' and the next layer of sampling points, input to the k+1 layer of the pyramid, repeat S2 to S6 until the number of sampling points is the same as the number of source points, S7. Calculate the loss function of the model, use the confidence-aware distance loss function and the flow smoothness loss function to calculate the loss in each layer, and finally add the losses through the sampling relationship to obtain the total training objective function. S8. Experiment and verification of the model,The source point cloud and target point cloud are input into the model, and the output is the required estimated scene flow.

2. The scene flow estimation method based on large displacement decomposition according to claim 1, characterized in that: In step S2, a three-layer neural network is used to obtain point cloud features. First, in the first layer, iterative farthest point sampling is used to select a subset of points so that X ij It is the point farthest from the other points in the set. In the second layer, the points are grouped by spherical query. In the third layer, local feature extraction is performed in the local area of ​​each point set. The above point cloud feature extraction operation is performed on both the source point X and the target point Y to obtain the feature Φ of the two point clouds. X ,Φ Y ∈R d .

3. The scene flow estimation method based on large displacement decomposition according to claim 2, characterized in that: Step S3 is as follows: In step S2, a three-layer neural network is used to extract point cloud features Φ X ,Φ Y ∈R d , and then calculate the cosine similarity in the feature space, in Φ corresponding to the i-th and j-th rows respectively X and Φ Y In order to avoid the distance between points being too far, which would cause the estimation accuracy to decrease, the cost for points with a Euclidean distance less than 10 meters is set to: C ij =1-S ij (2) The cost of the remaining points is set to ∞. At this time, the Euclidean distance between the points corresponding to the estimated flow is controlled within 10 meters. Use the optimal transmission model to obtain the correspondence between the source and target point clouds, and set the quality of each source point to And use Kantorovich relaxation optimal transmission model: Among them C ij ≥0 comes from formula (2), T ij ≥0 is the quality of the transfer, the parameter ∈,λ≥0 controls the relaxation problem, 1 n is a column vector with all entries equal to 1, KL is the Kullback-Leibler divergence, and the Sinkhorn algorithm is used in the second term of formula (3) to estimate the distance from C to T * The optimal transmission matrix of is obtained, thereby obtaining the correlation between point clouds.

4. The scene flow estimation method based on large displacement decomposition according to claim 3, characterized in that: In step S4, the initial scene flow is calculated and implemented by weight calculation method, and the source point cloud is matched to the domain of the target point cloud to improve the matching speed and accuracy. i ∈X, first calculate the weight of each point, in is included with the highest quality transmission K s index neighborhood, point to x i The soft correlation is expressed by the following formula: and point x i The initial flow is: The initial flow field is: in Contains All points.

5. The scene flow estimation method based on large displacement decomposition according to claim 4, characterized in that: In step S5, a pyramid deformation network is constructed, and the solution steps are divided into multiple layers. Each layer solves a part of the flow, and the large displacement flow is decomposed into small displacement solutions. The pyramid model is expressed as Δ = {(Γ k ,Θ k )|k=1,…,m}, where each pyramid level contains a pair of continuous functions (Γ k ,Θ k ), m is the total number of levels, At the kth level of the pyramid, Γ k It is a position encoding function that encodes the input point from the previous layer through sinusoidal encoding. Mapping: where k0 is a constant that controls the initial sampling frequency of the first level of the pyramid, At the k+1th level of the pyramid, Θ k For steps S2 to S4, the point set after sinusoidal encoding is input into Θ k In the example above, we get the initial flow F.

6. The scene flow estimation method based on large displacement decomposition according to claim 5, characterized in that: In step S6, After S1 to S4, the initial flow F of some points is obtained, and F is upsampled using bilinear interpolation. F is added to the input point set X after k+1 layer encoding to obtain a point cloud X' that is closer to the target point, and X' is input into Θ k+1 Repeat S2 to S6 until the number of points after sampling is equal to the number of source points.

7. The scene flow estimation method based on large displacement decomposition according to claim 6, characterized in that: In step S7, The loss function is calculated at each level of the pyramid model, and the matching confidence is defined for each point based on the correlation: The value range of is [-1,1]. In order to obtain a value between 0 and 1, the negative value is removed during the processing and the matching x is set. i The matching confidence is pass The distance loss is expressed as: Use flow smoothness loss to regularize the learning process: where N X (x i ) is the radius k in X f And x i The Euclidean neighborhood is the center, so the overall training objective function of each layer can be expressed as: For the overall loss function of the pyramid model, sum the loss functions of each layer:

Images