A self-supervised learning scene flow method based on point cloud data

Abstract

A kind of self-supervised learning scene flow method based on point cloud data, utilize the characteristics of point cloud data itself to construct data item constraint and smooth item constraint, use the two constraints to evaluate the similarity of two frames of point cloud, and train a self-supervised network with the error, for obtaining scene flow.The present application utilizes the constraint inside point cloud data to evaluate the similarity of the second frame point cloud after motion and the predicted second frame point cloud, thereby predicting the scene flow, compared with the current method, or using labeled point cloud data for supervised learning, or converting 2D optical flow into 3D by 2.5D or 3D image data with depth information to solve this problem, the present application directly consumes point cloud data and learns scene flow by self-supervised deep learning method.The present application method well considers the characteristics of point cloud data itself, and establishes data item constraint;In addition, the present application also considers the local rigidity of scene level point cloud data, and establishes smooth item constraint.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer software, and relates to self-supervised scene flow learning, in particular to a self-supervised learning scene flow method based on point cloud data. BACKGROUND

[0002] With the rise of direct consumption point cloud PointNet [1, 2] and a series of works, a large number of point cloud-based researches have emerged. Some classic computer vision tasks, such as classification, semantic segmentation, object detection, etc., have found their new ideas in point clouds. Most of them focus on how to understand a single static scene. With the help of the functions of 3D sensors, such as LiDAR, Kinect and various robots, we began to desire to deal with dynamic scenes composed of point clouds. Scene flow aims to calculate point-by-point motion, which can be applied to analysis tasks of point cloud sequences, such as motion segmentation [3], action recognition [4], etc.

[0003] In the past 20 years, some research schemes [5, 6] have solved this problem by converting 2D optical flow into 3D through 2.5D or 3D image data with depth information (such as RGBD, binocular image). FlowNet3D [7] is a pioneering work that directly uses point cloud as input and then learns scene flow based on neural network in an end-to-end manner. It extracts the features of two frames at the same time, then combines them with a flow embedding module, and finally outputs the predicted scene flow. It has been proved that this design is very useful and has achieved encouraging results, which confirms the potential of neural network in learning scene flow. However, it relies heavily on a large amount of labeled data, which makes the training process expensive. Although this method learns scene flow is a difficult problem, it can also estimate reasonable scene flow without labeled data when there are some assumptions, such as small displacement and some local regularization. Unsupervised methods for learning optical flow [8] usually synthesize one frame by deforming another frame to form a photometric consistency constraint. While point clouds are randomly sampled from continuous surfaces. One of the difficulties of using self-supervised methods based on point clouds is that many points in the first frame do not appear in the second frame, so that the synthesized frame will naturally be inconsistent with the second frame. Another difficulty is that, unlike RGB data, the XYZ coordinates in point cloud only represent the occupancy at a specific position, which is defective in recognition.

[0004] REFERENCES

[0005] [1] QI C R, SU H, MO K, et al. Pointnet: Deep learning on point sets for 3d classification and segmentation [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 652-660.

[0006] [2] QI C R, YI L, SU H, et al. Pointnet++: Deep hierarchical feature learning on point sets in a metric space [J]. arXiv preprint arXiv: 1706.02413, 2017.

[0007] [3] RANJAN A, JAMPANI V, BALLES L, et al. Competitive collaboration: Joint unsupervised learning of depth, camera motion, optical flow and motion segmentation [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2019: 12240-12249

[0008] [4] LIANG D, FAN G, LIN G, et al. Three-stream convolutional neural network with multi-task and ensemble learning for 3d action recognition [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition Workshops. 2019: 0-0.

[0009] [5] HERBST E, REN X, FOX D. Rgb-d flow: Dense 3-d motion estimation using color and depth [C] / / 2013 IEEE international Conference on robotics and automation. 2013: 2276-2282.

[0010] [6] WEDEL A, RABE C, VAUDREY T, et al. Efficient dense scene flow from sparse or dense stereo data [C] / / European conference on computer vision. 2008: 739-751.

[0011] [7] LIU X, QI C R, GUIBAS L J. Flownet3d: Learning scene flow in 3d point clouds [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2019: 529-537.

[0012] [8] MEISTER S, HUR J, ROTH S. Unflow: Unsupervised learning of optical flow with a bidirectional census loss [C] / / Proceedings of the AAAI Conference on Artificial Intelligence: Vol 32. 2018. SUMMARY

[0013] The problem to be solved by the present application is: how to learn scene flow based on point cloud data in a self-supervised manner. In the self-supervised manner, the core work of the present application is to evaluate the similarity between the synthesized frame formed by the predicted scene flow and the previous frame point cloud and the real next frame point cloud after motion, and to use the error as the loss function of the self-supervised network.

[0014] The technical scheme of the present application is: a self-supervised learning scene flow method based on point cloud data, which uses the characteristics of point cloud data itself to construct data item constraints and smooth item constraints, uses the two constraints to evaluate the similarity of the second frame point cloud after motion and the predicted second frame point cloud, and uses the error to train a self-supervised network for predicting scene flow according to point cloud.

[0015] Further, the backbone network of the self-supervised network adopts FlowNet3D, first uses the iterative closest point algorithm ICP to generate a rough scene flow as a label for supervised pre-training of the self-supervised network, and then generates the scene flow flow through the pre-trained network, adds the flow to the first frame point cloud P1 after motion to obtain the predicted second frame point cloud P'1 after motion, then evaluates the similarity of P'1 and the real second frame point cloud P2 after motion through data item constraints and smooth item constraints, and uses the evaluation error as the loss function of the network to train until the network converges to obtain the predicted scene flow.

[0016] The object of the present application is to construct a network that directly consumes point cloud data and self-supervised learning scene flow. Unlike the past, which solves this problem by converting 2D optical flow into 3D through 2.5D or 3D image data with depth information, point cloud data only stores the spatial coordinates of objects, so the volume is smaller, and secondly, point cloud data can better reflect the spatial distribution of objects, and with the rise of 3D sensors such as LiDAR, Kinect and various robots, point cloud data will become the mainstream of 3D tasks. The existing technology learns scene flow through a neural network, which relies heavily on a large amount of labeled data, which makes the training process expensive. In order to solve the above difficulties, the present application proposes a self-supervised method for learning scene flow based on unlabeled point cloud data under the premise of small displacement and local rigid body properties.

[0017] Compared with the prior art, the present application has the following advantages.

[0018] The self-supervised method based on point cloud can estimate scene flow, but it is difficult to implement, and the present application proposes an implementable self-supervised learning method based on point cloud. The present application directly uses point cloud data to learn the scene flow task. The past work solves this problem by converting 2D optical flow into 3D through 2.5D or 3D image data with depth information, and point cloud data has more advantages and uses.

[0019] The present application uses a self-supervised method to learn scene flow. Although the recent supervised training using labeled data can obtain good results, the labeling of point cloud is time-consuming and laborious. The self-supervised work of the present application has a wider application field.

[0020] The two constraints designed by the application to evaluate the similarity of two frames of point clouds are data item constraint and smooth item constraint, compared with the traditional Chamfer distance used to evaluate the similarity of two point clouds, the constraints of the application fully consider the characteristics of point cloud data itself, and the comparison experiment also verifies that the constraints have better effects. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 is a schematic diagram of the self-supervised scene flow learning framework of the application.

[0022] Figure 2 is a schematic diagram of the backbone network FlowNet3D of the network structure of the application.

[0023] Figure 3 is a schematic diagram of the two-stage training framework of the application.

[0024] Figure 4 is a schematic diagram of the predicted scene flow result of the application. DETAILED DESCRIPTION

[0025] The application proposes a self-supervised learning scene flow method based on point cloud data, which uses the characteristics of point cloud data itself to construct data item constraint and smooth item constraint, uses the two constraints to evaluate the similarity of the second frame of point cloud after motion and the predicted second frame of point cloud, and uses the error to train a self-supervised network for predicting scene flow according to point cloud. The self-supervised learning network of the application is pre-trained on the synthetic data set FlyingThings3D with proxy labels provided by the iterative closest point algorithm ICP, and the self-supervised method of the application is used for fine-tuning on FlyingThings3D. In the test phase, better results than previous methods are achieved on the test data set of FlyingThings3D. In order to test the generalization ability of the network of the application in the real scene, the method of the application is evaluated on the KITTI road scene data set without fine-tuning, and better results than previous methods are also achieved. Specifically, the Python 3.6 programming language and the Pytorch 1.1 deep learning framework are used.

[0026] Figure 1It is a self-supervised scene flow learning framework diagram used by the present application. The present application uses FlowNet3D as a backbone network. First, a coarse scene flow is generated by ICP as a label for supervised pre-training of the self-supervised network. Then, the pre-trained network is used to generate a scene flow flow. The flow is added to the first frame P1 of the motion to obtain the predicted second frame P'1 after the motion. Then, the similarity of P'1 and the real second frame P2 after the motion is evaluated through data item constraints and smoothing item constraints. The evaluation error is used as the loss function of the network. The network is trained until it converges to obtain the predicted scene flow. The specific implementation of the self-supervised learning network in the method of the present application includes a network configuration stage, a pre-training stage, a self-supervised constraint generation stage, a training stage, and a test stage, which are as follows.

[0027] 1) Network configuration stage. The backbone network used by the present application is the classic network FlowNet3D for generating scene flow under supervision. This network can generate a coarse scene flow under self-supervised mode. In addition, the constraints proposed by the present application can be used to predict an accurate scene flow. The network input data format is x i =B*2048*3, where B represents the batch size Batchsize. In the experiment, the B selected by the present application is 16, and the number of points of each frame of point cloud is 2048. FlowNet3D is as follows. Figure 2, mainly contains three modules: point feature learning module, point fusion module, flow refinement module, and finally adds a fully connected layer. The point feature fusion module is to down-sample the point cloud and generate the features of the points, which contains two set conv layers, each layer is down-sampled first and then connected with an mlp, the first layer is down-sampled by a ratio of 0.5, and the mlp scale is (32, 32, 64), the second layer is down-sampled by a ratio of 0.25, and the mlp scale is (64, 64, 128), the output data format after passing through the module is B*256*128; the point fusion module is to find the corresponding points of the points in one frame of point cloud in another frame and continue to down-sample the features, which contains a flow embedding layer and two set conv layers, the flow embedding layer is composed of an mlp, and the mlp scale is (128, 128, 128), the first set conv layer in this module is down-sampled by a ratio of 0.25, and the mlp scale is (128, 128, 256), the second set conv layer in this module is down-sampled by a ratio of 0.25, and the mlp scale is (256, 256, 512), the output data format after passing through the module is B*16*512. The flow refinement module is to restore the number of points of the point cloud and save the point features through skip connection, which contains 4 set upconv layers, each layer is up-sampled first and then connected with an mlp, the first layer is up-sampled by a ratio of 4, the mlp scale is (128, 128, 256), the second layer is up-sampled by a ratio of 4, the mlp scale is (128, 128, 256), the third layer is up-sampled by a ratio of 4, the mlp scale is (128, 128, 128), the fourth layer is up-sampled by a ratio of 2, the mlp scale is (128, 128, 128), and the output data format after passing through the module is B*2048*128. Finally, a fc layer is used to generate the predicted scene flow, and the data format is B*2048*3.

[0028] 2) In the pre-training stage, in order to deal with the large displacement area in the scene, the present application first uses the proxy label provided by ICP to pre-train the FlowNet3D network. ICP is designed to solve the registration of rigid body, and the point correspondence is taken as the variable to be estimated. This algorithm assumes that there is a global transformation matrix T. The first frame of point cloud P1 is transformed by the transformation matrix T to obtain the second frame of point cloud P2, and the process of estimating the transformation matrix T is to minimize the following constraint:

[0029]

[0030] n is the number of points in P1.

[0031] ICP is strictly good at registration of rigid point clouds and is not suitable for dynamic scenes where different regions have different motion directions and amplitudes. In this task, although each object has its own motion direction, there are still many regions that have roughly the same motion direction, which is why we can use ICP initialization. For simplicity, we calculate a global transformation matrix T for each point cloud pair. In addition, there are some highly dynamic scenes in the FlyingThings3D dataset, so it is inevitable to cause some low signal-to-noise ratio samples. We set a threshold according to the convergence error of ICP to filter some low signal-to-noise ratio samples, and we finally select the top 80% data for pre-training. The supervision signal of the agent is the rough scene flow label is generated by the following equation:

[0032]

[0033] p represents a point in the point cloud P1.

[0034] 3) Self-supervised constraint generation phase, in this phase mainly expounds our backbone network and the constraints we proposed, including data item constraint Surface similarity and smooth item constraint regular term.

[0035] 3.1) Data item constraint. Discrete point clouds are randomly sampled from continuous object surfaces, although many points in the first frame point cloud may not have corresponding relationships in the second frame point cloud, but the potential surfaces of the two frames still have matching relationships. Therefore, the present application considers converting discrete point clouds into continuous space to make the similarity between discrete points into the similarity between two continuous entities, so the conversion should preserve the local spatial characteristics and spatial position. Therefore, the present application sets some parameterized detection points around each point to detect the local density. The detection point combination describes the characteristics of the local environment, and the spatial position is reflected in the center of the detection point. The specific operation will be divided into four stages: local sampling, generation of detection points, projection, and similarity measurement.

[0036] 3.1.1) Local sampling. With the rough scene flow provided by pre-training, the first frame point cloud P1 is transformed to a new position P'1, in order to obtain the local structure of the p' point in P'1, the present application samples some point set S' = {p' i} from the synthesized second frame P'1 as PointNet++, these points satisfy:‖p'-p' i ‖2 < r, in the experiment r takes 1.0. Then use the corresponding same center point and radius, that is, the point corresponding to p' in the second frame point cloud P2, to sample some point set S2 = {pi} The same center ensures the consistency of spatial position. Considering the fact that the density of points in different regions of the scene is not the same, this will result in the number of points sampled being less than the required number. At this point, the strategy adopted by the present invention is to fill in the number of points that are insufficient with those that have already been sampled. In the worst case, if there are no points within the radius, the present invention will reset the radius to t times the distance between the sample center and the nearest point, and the embodiment t takes 1.2. In addition, some regions in the first frame can move out of the field of view in the second frame, and if the data set used at this time provides information about the missing points, the present invention will avoid sampling those missing points. Finally, for each point, the present invention will repeat the above sampling operation.

[0037] 3.1.2) Generation of detection points. The point set S' obtained by the above sampling reflects the spatial local structure of the center point. The present invention sets up a check point in the sphere to measure the distribution of the object surface. Each point in the spherical space with the center point p' and radius r is located by the spherical coordinates , where p e [0, r], θ e [0, 2π), and the first dimension is divided into a parts, the second dimension is divided into b parts, and the third dimension is divided into c parts (in the experiment a, b, c are respectively taken as 3, 6, 3), forming a point similar to (i-th, j-th, k-th), and can be located by the following formula:

[0038]

[0039]

[0040]

[0041] After obtaining all the coordinates and removing the repeated points, a series of detection point sets O = {o m} are obtained.

[0042] 3.1.3) Projection. At this stage, since each point in P'1 has undergone two sampling processes, in the first sampling stage, the sample center is each point in the predicted synthesized second frame P'1 for the first time, and the corresponding neighbor points are also located in the predicted second frame P'1, and the result set is represented as S' = {p i} In the second sampling stage, the sample center is also each point in the predicted synthesized second frame P'1, and the corresponding neighbor points are sought in the real second frame P2, and the result set is S2 = {p i} Assuming that each point is represented by the size of the influence according to the distance, the total influence value received by each detection point from S' is as follows:

[0043]

[0044] Where d(p′) i ,o m ) represents the spatial Euclidean distance between the sampling point and the detection point. After the projection operation, we can obtain Wp′ and Wp, which represent the detection results for the synthesized second frame and the real second frame, respectively.

[0045] 3.1.4) Similarity Measurement. After the above steps, the local structure of the object's potential surface has been represented by values ​​stored in the detection points, the value W. m Higher values ​​indicate a closer proximity to the potential real surface. Meanwhile, a sampling point emits a maximum and minimum value of 1.0 and e. -r This difference is not very significant and is sensitive to sampling density. Therefore, this invention employs the following cosine similarity metric to mitigate the impact of sampling density.

[0046]

[0047] 3.2) Smoothing Constraints. For scene-flow tasks, relying solely on a single data item is unlikely to guarantee an accurate solution. Using only surface similarity as a constraint can lead to ambiguity due to the existence of many equivalent solutions for various scenes, thus failing to adequately address the problem. Given our assumption of local rigidity, this invention proposes two types of smoothing constraints: local motion consistency and local structural consistency.

[0048] 3.2.1) Local Motion Consistency. Under the assumption of small motions, local regions tend to move in the same direction. For each point p in P1, n nearest neighbor points N(p) are selected, and the following motion smoothing loss function is defined:

[0049]

[0050] Where f p It is the scene flow at point p.

[0051] 3.2.1) Local structural consistency. When encountering object rotation, the motion smoothing term easily fails. Considering the local rigid body assumption, the distance between two points located locally within the same object remains constant before and after motion. Therefore, the following structural smoothing loss function is defined:

[0052]

[0053] Where p is a point in P1, p′ is a point in the synthetic point cloud P′1, and N(p) is the set of the n nearest points to p. This is a more flexible smoothing term suitable for all motion scenarios.

[0054] 3.3) Loss function. The final loss function is formulated by the data term and the smooth term proposed before:

[0055] L all = a * L d + b * L s

[0056] where a and b are hyperparameters, which are set to 100.0 and 0.2 in the experiment, respectively, L d is the data term constraint, L s is the smooth term constraint, which can be either motion smooth term or structure smooth term.

[0057] 4) Training stage. The network of the present application is pre-trained on the synthetic dataset FlyingThings3D with proxy labels provided by ICP for 150K iterations, and fine-tuned on FlyingThings3D using the self-supervised method of the present application for 250K iterations. In order to test the generalization ability of the network of the present application in real scenes, the method of the present application is evaluated on KITTI without fine-tuning. All experiments of the present application are performed on 4 Titan Xp GPUs with Adam optimizer (default configuration). During pre-training, the initial learning rate is set to 1e-4, and the decay rate is set to 0.7. In the fine-tuning stage on FlyingThings3D, the initial learning rate is set to 1e-5, and the decay rate is set to 0.7. For the loss function, the configuration of a = 100.0 and b = 0.2 is used. The configuration for generating detection points uses: a = 3, b = 6, c = 3. Of course, in order to further process the large displacement area in the scene, the present application also uses a multi-stage training strategy, such as Figure 3 , first move the first frame to a new position P'1 according to the scene flow flow1 predicted in the first stage, then put P'1 and the real second frame P2 into the network again to get the predicted scene flow flow2, and the final scene flow is the sum of the continuous multi-stage scene flow.

[0058] 5) Test stage, on the test set of FlyingThings3D, the experimental results of the present application using single-stage network are EPE=0.354, ACC1=7.43%, ACC2=27.5%; the experimental results of the present application using two-stage network are EPE=0.314, ACC1=8.58%, ACC2=30.2%, and the results of the supervised method on this data set are EPE=0.1694, ACC1=25.37%, ACC2=57.85%; the smaller the EPE and the larger the ACC represent the better result. On the KITTI road scene data set, the experimental results of the present application using single-stage network are EPE=0.207, Outliers=17.3%; the experimental results of the present application using two-stage network are EPE=0.155, Outliers=8.69%, and the results of the supervised method on this data set are EPE=0.122, Outliers=5.61%, the smaller the EPE and the Outliers represent the better result. The self-supervised method of the present application has achieved very good results on two data sets, among which the result of the self-supervised method on the FlyingThings3D data set is slightly worse than that of the supervised method, but on the KITTI data set, the result of the self-supervised method is very close to that of the supervised method, the reason is that the FlyingThings3D data set is a CAD data set, and the displacement difference between different objects in many scenes is large, which makes the scene very chaotic, and it does not conform to the assumption of the self-supervised method of the present application: small displacement and local rigidity; but the KITTI data set is a road scene data set obtained by laser radar scanning, the motion amplitude of the objects in the data scene is small, and the scene is relatively regular, which conforms to the assumption of the present application, so the result is very close to that of the supervised method. Figure 4 The scene flow predicted by the method of the present application in a scene of the KITTI data set is shown, which contains vehicles, pedestrians, buildings and other objects, Figure 4-1 represent the first frame point cloud, Figure 4-2 represent the real second frame point cloud, Figure 4-3 represent the second frame synthesized by the predicted scene flow, Figure 4-4 represent the overlap of the real second frame point cloud and the predicted second frame point cloud, respectively, wherein the overlapping part of the two frame point clouds is represented by black points, and the remaining part is represented by gray points.

Claims

1. A self-supervised learning scene flow method based on point cloud data, characterized in that The data term constraint and the smooth term constraint are constructed by the characteristics of the point cloud data itself. The two constraints are used to evaluate the similarity between the second frame point cloud after motion and the predicted second frame point cloud, and the error is used to train a self-supervised network for predicting scene flow according to the point cloud; The backbone of the self-supervised network uses FlowNet3D. First, it uses the Iterative Closest Point (ICP) algorithm to generate a coarse scene flow as labels for supervised pre-training of the self-supervised network. Then, it uses the pre-trained network to generate the scene flow and compares the flow with the point cloud of the first frame of motion. The summation yields the second frame point cloud after the predicted motion. Then, the evaluation is performed using data item constraints and smoothing term constraints. The second frame point cloud after actual motion The similarity is used, and the evaluation error is used as the loss function of the network. The network is trained until it converges to obtain the predicted scene flow. The data term constraint is as follows: the discrete point cloud is converted into a continuous space, so that the similarity between discrete points becomes the similarity between two continuous entities, and detection points are set around each point to detect local density. The combination of detection points describes the characteristics of the local environment, and the spatial position is reflected in the center of the detection points, including four stages: local sampling, generation of detection points, projection, and similarity measurement: Step 1.1) Local sampling: Based on the coarse scene flow provided by pre-training, the first frame point cloud is sampled. Change to a new position In order to obtain In The local structure of the points, from the synthesized second frame Medium sampling point set These points satisfy: Then use the same center point And radius r, the set of sampled points from the second frame point cloud P2 The same center ensures consistency in spatial location; Step 1.2) Generation of inspection points, the point set obtained in the last step is sampled Reflecting the spatial local structure of the center point, the inspection points are set to measure the distribution of the object surface, taking the center point Each point in the spherical space with the center of the center point and the radius of r is positioned by the spherical coordinates (p, q, f), where p e [0, r], q e [0, 2p), f e [0, p], and the dimension one is divided into a parts, the dimension two is divided into b parts, and the dimension three is divided into c parts, forming the (i-th, j-th, k-th) point, and positioned by the following formula: After obtaining all the coordinates, and removing the duplicate points, a series of detection point sets are obtained ; Step 1.3) projection, for each point in two sampling processes are performed, in the first sampling stage, the sample center predicts the second frame, the corresponding neighbor points are also in the predicted second frame, the result set is represented as in the second sampling stage, the sample center is also each point in the predicted second frame, but the corresponding neighbor points are found in the real second frame, the result set is assuming that the size of the influence is represented according to the distance, then the total influence value received by each detection point from is as follows: wherein represents the spatial Euclidean distance between the sampling point and the detection point, and after projection, Wp' and Wp represent the detection results for the synthesized second frame and the real second frame, respectively; Step 1.4) Similarity measurement. After the above steps, the local structure of the potential surface of the object is represented by the values stored in the detection points. The points with higher values indicate that they are closer to the potential real surface. At the same time, the maximum and minimum values emitted by a sampling point are 1.0 and 0.0, respectively. This gap is sensitive to the sampling density. The following cosine similarity measure is used to reduce the impact of the sampling density, resulting in the data item constraint : ； The smooth term constraint has two kinds: local motion consistency and local structure consistency, 2.1) Local motion consistency, under the assumption of small motion, local regions tend to move in the same direction, for each point p, select n nearest neighbor points , formulate the following motion smoothing term loss function: ​ wherein is a scene stream of point p; 2.2) Local structure consistency, in the case of object rotation, the following structure smooth term loss function is formulated: where p is a point in is a synthetic point cloud a point in N(p) is the set of n points closest to p. 2.The method of claim 1, wherein the method further comprises The pre-training of the self-supervised network is as follows: the ICP algorithm assumes that there is a global transformation matrix T, and the first frame point cloud P1 is transformed into the second frame point cloud P2 by the transformation matrix T. The process of estimating the transformation matrix T is to minimize the following constraint: n is The number of midpoints; a global transformation matrix T is computed for each point cloud pair, a threshold is set according to the convergence error of ICP to filter low signal-to-noise ratio samples, and the proxy supervisory signal is the coarse scene flow label , generated by the following equation: representing points in a point cloud representing points in a point cloud 3.The method of claim 1, wherein 1.1) for the case where the number of points obtained by sampling is less than the required number of points, directly copying until the number meets the requirement; if the sampling does not have points within the radius, resetting the radius to t times the distance between the center point and the nearest point; Furthermore, if there is no point in the second frame corresponding to the point, then the vanishing point is avoided from being sampled. 4.The method of claim 1, wherein the method further comprises The FlyingThings3D dataset of CAD is used to train the self-supervised network, and the loss function is as follows: wherein and as hyperparameters, is a data item constraint, is a smoothness constraint.

5. The self-supervised learning scene flow method based on point cloud data according to claim 1, characterized in that A multi-stage training strategy is employed, first predicting the scene flow according to the first stage moving the first frame to a new position and the real second frame again into the network to get the predicted scene flow The final scene flow is the sum of the continuous multi-stage scene flow.

Images