Point cloud scene flow estimation method, readable storage medium and intelligent device

Abstract

The application relates to the technical field of automatic driving, in particular to a point cloud scene flow estimation method, a readable storage medium and an intelligent device, and aims to solve the technical problem of how to overcome the low scene flow estimation accuracy for sparse point clouds. To this end, the application obtains an initial source point cloud and an initial target point cloud, performs bidirectional point cloud reconstruction based on the initial source point cloud and the initial target point cloud, obtains a reconstructed source point cloud and a reconstructed target point cloud, and obtains a scene flow prediction result according to the reconstructed source point cloud and the reconstructed target point cloud. The application performs bidirectional point cloud reconstruction based on the initial source point cloud and the initial target point cloud, can overcome the problem of sparseness of actually collected point clouds, and reduce the problem of inconsistent distribution such as noise and misplacement caused by point cloud sparseness. At the same time, the problem of rapid reduction of the proportion of shared point clouds between continuous frames caused by sparseness can also be avoided through bidirectional point cloud reconstruction, and the prediction accuracy of point cloud scene flow prediction can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, specifically to a point cloud scene flow estimation method, a readable storage medium, and an intelligent device. Background Technology

[0002] Point cloud scene flow estimation is a fundamental task in autonomous driving, used for various tasks such as object detection and object tracking. It provides motion estimation of initial frame data by predicting the motion velocity of pixels or point clouds.

[0003] Point cloud data acquired by LiDAR scanning has inherent sparsity. In scene streaming tasks, sparsity can lead to a rapid decrease in the proportion of shared point clouds between consecutive frames and misalignment between given point pairs, resulting in an excessively high proportion of prediction anomalies and a decrease in prediction accuracy.

[0004] Initially, point cloud scene flow was treated as a regression task, directly predicting the 3D spatial velocity of each point based on its coordinates. However, due to the difficulty of regressing space in 3D scenes, this approach requires a large amount of supervised data and struggles to converge. Some methods attempted to avoid this problem through runtime optimization, which can achieve more accurate flow predictions without training data. However, these methods assume a consistent distribution between given source and target points and are susceptible to noisy input data without prior learning. Subsequent methods typically use matching mechanisms to compute point flows. From a matching perspective, point scene flow is considered as the displacement between a given source / target point pair and its matched target, with the matching relationship obtained through a similarity matrix determined by point embeddings. However, this approach heavily relies on the existence of point correspondences.

[0005] Accordingly, there is a need in this field for a new point cloud scene flow estimation scheme to solve the above problems. Summary of the Invention

[0006] In order to overcome the above-mentioned shortcomings, this application is proposed to solve, or at least partially solve, the technical problem of how to overcome the low accuracy of scene flow estimation for sparse point clouds.

[0007] In a first aspect, a point cloud scene flow estimation method is provided, the method comprising:

[0008] Obtain the initial source point cloud and the initial target point cloud; the initial source point cloud and the initial target point cloud are point clouds from two consecutive frames at adjacent time points;

[0009] Based on the initial source point cloud and the initial target point cloud, bidirectional point cloud reconstruction is performed to obtain the reconstructed source point cloud and the reconstructed target point cloud; the bidirectional point cloud reconstruction is to reconstruct the point cloud of the next frame based on the point cloud of the previous frame, and to reconstruct the point cloud of the previous frame based on the point cloud of the next frame.

[0010] Based on the reconstructed source point cloud and the reconstructed target point cloud, obtain the point cloud scene flow prediction result.

[0011] In one technical solution of the above-mentioned point cloud scene flow estimation method, the step of performing bidirectional point cloud reconstruction based on the initial source point cloud and the initial target point cloud to obtain the reconstructed source point cloud and the reconstructed target point cloud includes:

[0012] The initial source point cloud and the initial target point cloud are used as input data for the initial iteration of the bidirectional point cloud reconstruction.

[0013] Based on the input data, bidirectional point cloud reconstruction is performed for a preset number of iterations to obtain the source point cloud and the target point cloud.

[0014] In one technical solution of the above-mentioned point cloud scene flow estimation method, the step of performing bidirectional point cloud reconstruction based on input data for a preset number of iterations to obtain the reconstructed source point cloud and the reconstructed target point cloud includes:

[0015] For each iteration of the bidirectional point cloud reconstruction, the point cloud correspondence between the two consecutive frames of point clouds in the input data of the current iteration is obtained;

[0016] Based on the point cloud correspondence, the previous frame point cloud in the input data is used as the reference point cloud, and the next frame point cloud in the input data is used as the point cloud to be reconstructed. Point cloud reconstruction is performed to obtain the intermediate target point cloud.

[0017] Based on the point cloud correspondence, the point cloud of the next frame in the input data is used as the reference point cloud, and the point cloud of the previous frame in the input data is used as the point cloud to be reconstructed. Point cloud reconstruction is performed to obtain the intermediate source point cloud.

[0018] Determine that the current iteration count has reached the preset iteration count;

[0019] If so, the intermediate source point cloud and the intermediate target point cloud are respectively used as the reconstructed source point cloud and the reconstructed target point cloud;

[0020] If not, the intermediate source point cloud and the intermediate target point cloud are used as the input data for the next iteration, and the next iteration of bidirectional point cloud reconstruction is performed.

[0021] In one technical solution of the above-mentioned point cloud scene flow estimation method, obtaining the point cloud correspondence between two consecutive frames of point clouds in the input data of the current iteration includes:

[0022] Feature extraction is performed on the two consecutive point clouds to obtain the relative features of all points in a single frame point cloud.

[0023] Based on the relative features of a single frame point cloud, the point cloud alignment features of two consecutive frames are obtained through global cross-attention feature extraction.

[0024] Based on the point cloud alignment features, the point cloud correspondence is obtained.

[0025] In one technical solution of the above-mentioned point cloud scene flow estimation method, obtaining the point cloud correspondence based on the point cloud alignment features includes:

[0026] Based on the point cloud alignment features, obtain the point cloud similarity matrix between the two consecutive frames of point clouds;

[0027] The point cloud similarity matrix is ​​used as the point cloud correspondence.

[0028] In one technical solution of the above-mentioned point cloud scene flow estimation method, the point cloud reconstruction includes:

[0029] Based on the correspondence between the reference point cloud and the point cloud, a weighted average is performed on the reference point cloud to obtain the missing point cloud in the point cloud to be reconstructed;

[0030] The obtained point cloud and the point cloud to be reconstructed are distributed and merged to achieve the point cloud reconstruction.

[0031] In one technical solution of the above-mentioned point cloud scene flow estimation method, obtaining the point cloud scene flow prediction result based on the reconstructed source point cloud and the reconstructed target point cloud includes:

[0032] The displacement difference between the reconstructed source point cloud and the reconstructed target point cloud is obtained as the point cloud scene flow prediction result.

[0033] In one technical solution of the above-mentioned point cloud scene flow estimation method, the method further includes:

[0034] Based on a pre-trained scene flow estimation model, bidirectional point cloud reconstruction is performed based on the initial source point cloud and the initial target point cloud, and the point cloud scene flow prediction result is obtained based on the reconstructed source point cloud and the reconstructed target point cloud.

[0035] In one technical solution of the above-mentioned point cloud scene flow estimation method, the method further includes training the scene flow estimation model according to the following steps:

[0036] For each training round, the initial source point cloud and initial target point cloud used for training are input into the scene flow estimation model to obtain the scene flow prediction result of the current training round; the initial source point cloud and initial target point cloud used for training contain the movement speed label information of the point cloud.

[0037] Based on the movement speed label information and the scene flow prediction results, obtain the total loss for the current training round;

[0038] Based on the total loss, update the model parameters of the scene flow estimation model; and determine whether the training cutoff condition has been met.

[0039] If the training cutoff condition is not met, proceed to the next round of training;

[0040] If the training cutoff condition is met, training ends to obtain the trained scene flow estimation model.

[0041] In one technical solution of the above point cloud scene flow estimation method, obtaining the total loss of the current training round based on the movement speed label information and the scene flow prediction result includes:

[0042] Based on the movement speed label information, obtain the point restoration loss of the bidirectional point cloud reconstruction;

[0043] Based on the movement speed label information and the scene flow prediction results of the current round of training, obtain the scene flow loss;

[0044] The total loss is obtained based on the point restoration loss and the scene flow loss.

[0045] In one technical solution of the above point cloud scene flow estimation method, the training cutoff condition is that the number of training rounds reaches a preset round threshold.

[0046] In a second aspect, a smart device is provided, the smart device including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program, which, when executed by the at least one processor, implements the method described in any of the above-described point cloud scene flow estimation methods.

[0047] In a third aspect, a computer-readable storage medium is provided, wherein a plurality of program codes are stored therein, the program codes being adapted to be loaded and run by a processor to perform the method described in any of the above-described point cloud scene flow estimation methods.

[0048] Solution 1. A point cloud scene flow estimation method, characterized in that the method includes:

[0049] Obtain the initial source point cloud and the initial target point cloud; the initial source point cloud and the initial target point cloud are point clouds from two consecutive frames at adjacent time points;

[0050] Based on the initial source point cloud and the initial target point cloud, bidirectional point cloud reconstruction is performed to obtain the reconstructed source point cloud and the reconstructed target point cloud; the bidirectional point cloud reconstruction is to reconstruct the point cloud of the next frame based on the point cloud of the previous frame, and to reconstruct the point cloud of the previous frame based on the point cloud of the next frame.

[0051] Based on the reconstructed source point cloud and the reconstructed target point cloud, obtain the point cloud scene flow prediction result.

[0052] Solution 2. The point cloud scene flow estimation method according to Solution 1, characterized in that,

[0053] The step of performing bidirectional point cloud reconstruction based on the initial source point cloud and the initial target point cloud to obtain the reconstructed source point cloud and the reconstructed target point cloud includes:

[0054] The initial source point cloud and the initial target point cloud are used as input data for the initial iteration of the bidirectional point cloud reconstruction.

[0055] Based on the input data, bidirectional point cloud reconstruction is performed for a preset number of iterations to obtain the source point cloud and the target point cloud.

[0056] Solution 3. The point cloud scene flow estimation method according to Solution 2, characterized in that,

[0057] The step of performing bidirectional point cloud reconstruction based on input data for a preset number of iterations to obtain the source point cloud and the target point cloud includes:

[0058] For each iteration of the bidirectional point cloud reconstruction, the point cloud correspondence between the two consecutive frames of point clouds in the input data of the current iteration is obtained;

[0059] Based on the point cloud correspondence, the previous frame point cloud in the input data is used as the reference point cloud, and the next frame point cloud in the input data is used as the point cloud to be reconstructed. Point cloud reconstruction is performed to obtain the intermediate target point cloud.

[0060] Based on the point cloud correspondence, the point cloud of the next frame in the input data is used as the reference point cloud, and the point cloud of the previous frame in the input data is used as the point cloud to be reconstructed. Point cloud reconstruction is performed to obtain the intermediate source point cloud.

[0061] Determine that the current iteration count has reached the preset iteration count;

[0062] If so, the intermediate source point cloud and the intermediate target point cloud are respectively used as the reconstructed source point cloud and the reconstructed target point cloud;

[0063] If not, the intermediate source point cloud and the intermediate target point cloud are used as the input data for the next iteration, and the next iteration of bidirectional point cloud reconstruction is performed.

[0064] Solution 4. The point cloud scene flow estimation method according to Solution 3, characterized in that,

[0065] The step of obtaining the point cloud correspondence between two consecutive frames of point cloud in the input data of the current iteration includes:

[0066] Feature extraction is performed on the two consecutive point clouds to obtain the relative features of all points in a single frame point cloud.

[0067] Based on the relative features of a single frame point cloud, the point cloud alignment features of two consecutive frames are obtained through global cross-attention feature extraction.

[0068] Based on the point cloud alignment features, the point cloud correspondence is obtained.

[0069] Solution 5. The point cloud scene flow estimation method according to Solution 4, characterized in that,

[0070] The step of obtaining the point cloud correspondence based on the point cloud alignment features includes:

[0071] Based on the point cloud alignment features, obtain the point cloud similarity matrix between the two consecutive frames of point clouds;

[0072] The point cloud similarity matrix is ​​used as the point cloud correspondence.

[0073] Solution 6. The point cloud scene flow estimation method according to Solution 3, characterized in that,

[0074] The point cloud reconstruction includes:

[0075] Based on the correspondence between the reference point cloud and the point cloud, a weighted average is performed on the reference point cloud to obtain the missing point cloud in the point cloud to be reconstructed;

[0076] The obtained point cloud and the point cloud to be reconstructed are distributed and merged to achieve the point cloud reconstruction.

[0077] Scheme 7. The point cloud scene flow estimation method according to any one of Schemes 1 to 6, characterized in that,

[0078] The step of obtaining point cloud scene flow prediction results based on the reconstructed source point cloud and the reconstructed target point cloud includes:

[0079] The displacement difference between the reconstructed source point cloud and the reconstructed target point cloud is obtained as the point cloud scene flow prediction result.

[0080] Solution 8. The point cloud scene flow estimation method according to Solution 1, characterized in that the method further includes:

[0081] Based on a pre-trained scene flow estimation model, bidirectional point cloud reconstruction is performed based on the initial source point cloud and the initial target point cloud, and the point cloud scene flow prediction result is obtained based on the reconstructed source point cloud and the reconstructed target point cloud.

[0082] Solution 9. The point cloud scene flow estimation method according to Solution 8, characterized in that,

[0083] The method further includes training the scene flow estimation model according to the following steps:

[0084] For each training round, the initial source point cloud and initial target point cloud used for training are input into the scene flow estimation model to obtain the scene flow prediction result of the current training round; the initial source point cloud and initial target point cloud used for training contain the movement speed label information of the point cloud.

[0085] Based on the movement speed label information and the scene flow prediction results, obtain the total loss for the current training round;

[0086] Based on the total loss, update the model parameters of the scene flow estimation model; and determine whether the training cutoff condition has been met.

[0087] If the training cutoff condition is not met, proceed to the next round of training;

[0088] If the training cutoff condition is met, training ends to obtain the trained scene flow estimation model.

[0089] Solution 10. The point cloud scene flow estimation method according to Solution 9, characterized in that,

[0090] The step of obtaining the total loss for the current training round based on the movement speed label information and the scene flow prediction result includes:

[0091] Based on the movement speed label information, obtain the point restoration loss of the bidirectional point cloud reconstruction;

[0092] Based on the movement speed label information and the scene flow prediction results of the current round of training, obtain the scene flow loss;

[0093] The total loss is obtained based on the point restoration loss and the scene flow loss.

[0094] Solution 11. The point cloud scene flow estimation method according to Solution 9 or 10, characterized in that,

[0095] The training cutoff condition is when the number of training rounds reaches a preset round threshold.

[0096] Option 12. A smart device, characterized in that it comprises:

[0097] At least one processor;

[0098] And, a memory communicatively connected to the at least one processor;

[0099] The memory stores a computer program, which, when executed by the at least one processor, implements the point cloud scene flow estimation method as described in any one of schemes 1 to 11.

[0100] Scheme 13. A computer-readable storage medium storing a plurality of program codes, characterized in that the program codes are adapted to be loaded and run by a processor to perform the point cloud scene flow estimation method as described in any one of Schemes 1 to 11.

[0101] The above-described technical solutions of this application have at least one or more of the following beneficial effects:

[0102] In implementing the point cloud scene flow estimation method provided in this application, this application acquires an initial source point cloud and an initial target point cloud, performs bidirectional point cloud reconstruction based on the initial source point cloud and the initial target point cloud to obtain a reconstructed source point cloud and a reconstructed target point cloud, and obtains the scene flow prediction result based on the reconstructed source point cloud and the reconstructed target point cloud. Through the above configuration, this application's bidirectional point cloud reconstruction based on the initial source point cloud and the initial target point cloud can overcome the sparsity problem of the actual collected point cloud and reduce the problems of noise and misalignment caused by point cloud sparsity. At the same time, bidirectional point cloud reconstruction can also avoid the problem of a rapid decrease in the proportion of shared point clouds between consecutive frames caused by sparsity, effectively improving the prediction accuracy of point cloud scene flow prediction. Attached Figure Description

[0103] The disclosure of this application will become more readily understood with reference to the accompanying drawings. It will be readily understood by those skilled in the art that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this application. Wherein:

[0104] Figure 1 This is a schematic diagram of the main steps of a point cloud scene flow estimation method according to an embodiment of this application;

[0105] Figure 2 This is a block diagram of the main implementation system for bidirectional point cloud reconstruction according to one embodiment of the present application.

[0106] Figure 3 This is a schematic diagram of scene flow prediction results according to an example of an embodiment of this application;

[0107] Figure 4 This is a schematic flowchart of the main steps of a point cloud scene flow estimation method according to one embodiment of the present application.

[0108] Figure 5 A schematic diagram of the main training process of a scene flow estimation model according to one embodiment of this application;

[0109] Figure 6 This is a schematic diagram of the main structure of a smart device according to an embodiment of this application.

[0110] Figure label:

[0111] 11: Memory; 12: Processor; 21: First frame point cloud; 22: Point cloud after actual movement of the first frame point cloud at the next moment; 23: Point cloud at the next moment predicted based on the first frame point cloud. Detailed Implementation

[0112] Some embodiments of this application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of this application and are not intended to limit the scope of protection of this application.

[0113] In the description of this application, "module" and "processor" can include hardware, software, or a combination of both. A module can include hardware circuitry, various suitable sensors, communication ports, memory, and can also include software components, such as program code, or a combination of software and hardware. A processor can be a central processing unit, microprocessor, image processor, digital signal processor, or any other suitable processor. The processor has data and / or signal processing capabilities. The processor can be implemented in software, in hardware, or a combination of both. Computer-readable storage media includes any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The terms "at least one A or B" or "at least one of A and B" have a similar meaning to "A and / or B" and can include only A, only B, or A and B. The singular terms "a" or "this" can also include plural forms.

[0114] The relevant user personal information that may be involved in the various embodiments of this application is processed in strict accordance with the requirements of laws and regulations, following the principles of legality, legitimacy, and necessity, based on the reasonable purpose of the business scenario, and includes personal information that users actively provide or that is generated as a result of using the product / service, as well as personal information obtained with user authorization.

[0115] The personal information processed in this application will vary depending on the specific product / service scenario and will be based on the specific scenario in which the user uses the product / service. This may involve the user's account information, device information, driving information, vehicle information, or other related information. This application will treat the user's personal information and its processing with the utmost diligence.

[0116] This application attaches great importance to the security of users' personal information and has taken reasonable and feasible security protection measures that comply with industry standards to protect users' information and prevent unauthorized access, disclosure, use, modification, damage or loss of personal information.

[0117] Here we will first explain some of the terms used in this application.

[0118] Scene flow is a three-dimensional version of optical flow, describing the changes of each point in an image or point cloud between two consecutive frames.

[0119] See appendix Figure 1 , Figure 1 This is a schematic diagram illustrating the main steps of a point cloud scene flow estimation method according to an embodiment of this application. Figure 1 As shown, the point cloud scene flow estimation method in this application embodiment mainly includes the following steps S101 to S103.

[0120] Step S101: Obtain the initial source point cloud and the initial target point cloud; the initial source point cloud and the initial target point cloud are point clouds from two consecutive frames at adjacent time points.

[0121] In this embodiment, initial source point clouds and initial target point clouds can be obtained for scene flow prediction.

[0122] In one implementation, the initial source point cloud and the initial target point cloud can be acquired based on lidar acquisition.

[0123] Step S102: Based on the initial source point cloud and the initial target point cloud, perform bidirectional point cloud reconstruction to obtain the reconstructed source point cloud and the reconstructed target point cloud; bidirectional point cloud reconstruction is to reconstruct the point cloud of the next frame based on the previous frame point cloud, and to reconstruct the point cloud of the previous frame based on the next frame point cloud.

[0124] In this embodiment, due to the sparsity between the initial source point cloud and the initial target point cloud, bidirectional point cloud reconstruction can be performed. That is, point cloud reconstruction of the initial target point cloud is performed based on the initial source point cloud, and point cloud reconstruction of the initial source point cloud is performed based on the initial target point cloud, thereby obtaining the reconstructed source point cloud and the reconstructed target point cloud. This bidirectional point cloud reconstruction process effectively avoids the problems of isolated and inconsistent point cloud distribution between consecutive frames caused by unidirectional point cloud reconstruction.

[0125] Step S103: Obtain the point cloud scene flow prediction result based on the reconstructed source point cloud and the reconstructed target point cloud.

[0126] In this embodiment, the point cloud scene flow prediction result can be obtained based on the reconstructed source point cloud and the reconstructed target point cloud.

[0127] In one implementation, the displacement difference between the reconstructed source point cloud and the reconstructed target point cloud can be obtained as the point cloud scene flow prediction result.

[0128] Based on the methods described in steps S101 to S103 above, this embodiment of the application obtains an initial source point cloud and an initial target point cloud, performs bidirectional point cloud reconstruction based on the initial source point cloud and the initial target point cloud to obtain a reconstructed source point cloud and a reconstructed target point cloud, and obtains a scene flow prediction result based on the reconstructed source point cloud and the reconstructed target point cloud. Through the above configuration, this embodiment of the application performs bidirectional point cloud reconstruction based on the initial source point cloud and the initial target point cloud, which can overcome the sparsity problem of the actual collected point cloud and reduce the problems of noise and misalignment caused by point cloud sparsity. At the same time, bidirectional point cloud reconstruction can also avoid the problem of a rapid decrease in the proportion of shared point clouds between consecutive frames caused by sparsity, effectively improving the prediction accuracy of point cloud scene flow prediction.

[0129] The following is a further explanation of step S102.

[0130] In one embodiment of this application, step S102 may further include steps S1021 and S1022:

[0131] Step S1021: Use the initial source point cloud and the initial target point cloud as input data for the initial iteration of bidirectional point cloud reconstruction.

[0132] In this embodiment, the initial source point cloud and the initial target point cloud can be used as input data for the initial iteration of bidirectional point cloud reconstruction.

[0133] Step S1022: Based on the input data, perform bidirectional point cloud reconstruction for a preset number of iterations to obtain the source point cloud and the target point cloud for reconstruction.

[0134] In this embodiment, step S1022 may further include steps S10221 to S10226:

[0135] Step S10221: For each iteration of bidirectional point cloud reconstruction, obtain the point cloud correspondence between the point clouds of the previous and next frames in the input data of the current iteration.

[0136] In this embodiment, step S10221 may further include steps S102211 to S102213:

[0137] Step S102211: Extract features from the point clouds of the two frames before and after the current frame to obtain the relative features of all points in the single frame point cloud.

[0138] In this embodiment, a graph neural network can be used to extract neighborhood features from the point clouds of two consecutive frames. A local point cloud attention layer is then applied to obtain the relative information, or relative features, of all points within a single frame based on the neighborhood features.

[0139] Step S102212: Based on the relative features of a single frame point cloud, obtain the point cloud alignment features of two consecutive frames of point clouds through global cross-attention feature extraction.

[0140] In this embodiment, based on the relative features of a single frame point cloud, global cross-attention extraction can be performed through a global cross-attention layer to obtain the point cloud alignment features of two consecutive frames.

[0141] Step S102213: Obtain the point cloud correspondence based on the point cloud alignment features.

[0142] In this embodiment, the point cloud similarity matrix between two consecutive frames can be obtained based on the point cloud alignment features, and the point cloud similarity matrix can be used as the point cloud correspondence.

[0143] Step S10222: Based on the point cloud correspondence, take the previous frame point cloud in the input data as the reference point cloud and the next frame point cloud in the input data as the point cloud to be reconstructed, and perform point cloud reconstruction to obtain the intermediate target point cloud.

[0144] Step S10223: Based on the point cloud correspondence, take the next frame point cloud in the input data as the reference point cloud and the previous frame point cloud in the input data as the point cloud to be reconstructed, and perform point cloud reconstruction to obtain the intermediate source point cloud.

[0145] In this embodiment, point cloud reconstruction can be performed on the point cloud of the next frame based on the point cloud of the previous frame, and point cloud reconstruction can be performed on the point cloud of the previous frame based on the point cloud of the next frame. Specifically, point cloud reconstruction can be performed based on the following steps S201 to S202:

[0146] Step S201: Based on the correspondence between the reference point cloud and the point cloud, perform a weighted average on the reference point cloud to obtain the missing point cloud in the point cloud to be reconstructed.

[0147] In this embodiment, the reference point cloud can be weighted and averaged based on the reference point cloud and the point cloud similarity matrix between the two consecutive frames, thereby obtaining the missing point cloud in the point cloud to be reconstructed.

[0148] Step S202: Distribute and merge the obtained point cloud and the point cloud to be reconstructed to achieve point cloud reconstruction.

[0149] In this embodiment, the obtained point cloud and the point cloud to be reconstructed can be distributed and merged to achieve point cloud reconstruction.

[0150] Step S10224: Determine if the current iteration count has reached the preset iteration count.

[0151] In this embodiment, it can be determined whether the current iteration number has reached the preset iteration number. If yes, then step S10225 is executed; if no, then step S10226 is executed.

[0152] Step S10225: Use the intermediate source point cloud and the intermediate target point cloud as the source point cloud and the target point cloud for reconstruction, respectively.

[0153] In this embodiment, if the bidirectional point cloud reconstruction reaches a preset number of iterations, the intermediate source point cloud obtained in the current iteration can be used as the reconstruction source point cloud, and the intermediate target point cloud obtained in the current iteration can be used as the reconstruction target point cloud.

[0154] Step S10226: Use the intermediate source point cloud and the intermediate target point cloud as input data for the next iteration, and perform the next iteration of bidirectional point cloud reconstruction.

[0155] In this embodiment, if the bidirectional point cloud iteration does not reach the preset number of iterations, the intermediate source point cloud and intermediate target point cloud obtained in the current iteration can be used as the input data for the next iteration, and the next iteration of bidirectional point cloud reconstruction can be performed.

[0156] In one implementation, see Appendix Figure 2 , Figure 2 This is a block diagram of the main implementation system for bidirectional point cloud reconstruction according to one embodiment of this application. Figure 2 As shown, each iteration in bidirectional point cloud reconstruction can be achieved through... Figure 2 The basic loop module shown in the upper middle figure is implemented. Specifically, the input data includes the source point cloud (p s,j ) and target point cloud (p t,j The process involves inputting the point cloud into a similarity alignment module to obtain a point cloud similarity matrix; then, based on the correspondence reconstruction module, reconstructing the point cloud using the similarity matrix, the source point cloud, and the target point cloud to obtain candidate points for both the source and target point clouds (i.e., obtaining the missing points in the point cloud to be reconstructed); finally, a distribution optimization module merges the point cloud distributions to obtain the reconstructed source point cloud (p). s,j+1 ) and target point cloud (p t,j+1 Bidirectional point cloud reconstruction can be performed on a preset number of iterations, i.e. Figure 2 As shown in the lower middle figure, the initial source point cloud (p s,0 ) and initial target point cloud (p t,0The input is fed into the basic loop module and iterated multiple times. Each iteration yields an intermediate source point cloud (p). s,j ) and intermediate target point cloud (p t,j When the preset number of iterations is reached, the reconstructed source point cloud (p) can be output. s,k ) and reconstruct the target point cloud (p t,k ).

[0157] In one example, see Appendix Figure 3 , Figure 3 This is a schematic diagram of scene flow prediction results according to an example of an embodiment of this application. For example... Figure 3 As shown, label 21 corresponds to the source point cloud of the first frame, label 22 is the target point cloud after the source point cloud has actually moved in the next moment, and label 22 is the position of the target point cloud predicted based on the scene flow prediction results obtained in this application. Figure 3 As can be seen, the scene flow prediction results obtained through this application can achieve good prediction results in sparse point cloud conditions.

[0158] In one implementation, see Appendix Figure 4 , Figure 4 This is a schematic flowchart illustrating the main steps of a point cloud scene flow estimation method according to one embodiment of this application. Figure 4 As shown, the point cloud scene flow estimation method may include the following steps S301 to S307:

[0159] Step S301: Input the initial source point cloud and the initial target point cloud.

[0160] In this embodiment, step S301 is similar to the method described in step S101 above, and will not be repeated here for the sake of simplicity.

[0161] Step S302: Determine whether the bidirectional point cloud reconstruction has reached the preset number of iterations; if yes, proceed to step S307; if no, proceed to step 303.

[0162] In this embodiment, step S302 is similar to the method described in step S10224 above, and will not be repeated here for the sake of simplicity.

[0163] Step S303: Extract point cloud features.

[0164] In this embodiment, step S303 is similar to the method described in steps S102211 and S10212 above, and will not be repeated here for the sake of simplicity.

[0165] Step S304: Calculate the point cloud similarity matrix.

[0166] In this embodiment, step S304 is similar to the method described in step S102213 above, and will not be repeated here for the sake of simplicity.

[0167] Step S305: Reconstruct the bidirectional point cloud.

[0168] In this embodiment, step S305 is similar to the method described in steps S10222 and S10223 above, and will not be repeated here for the sake of simplicity.

[0169] Step S306: After the original point cloud (point cloud to be reconstructed) and the reconstructed point cloud distribution are restored, proceed to step S302.

[0170] In this embodiment, step S306 is similar to the method described in step S202 above, and will not be repeated here for the sake of simplicity.

[0171] Step S307: Calculate the point cloud scene flow prediction results.

[0172] In this embodiment, step S307 is similar to the method described in step S103 above, and will not be repeated here for the sake of simplicity.

[0173] In one embodiment of this application, a bidirectional point cloud reconstruction can be performed based on a preset trained scene flow estimation model, based on an initial source point cloud and an initial target point cloud, and the point cloud scene flow prediction result can be obtained based on the reconstructed source point cloud and the reconstructed target point cloud.

[0174] In one implementation, the scene flow estimation model can be trained according to the following steps S401 to S405:

[0175] Step S401: For each training round, input the initial source point cloud and initial target point cloud used for training into the scene flow estimation model to obtain the scene flow prediction result of the current training round; the initial source point cloud and initial target point cloud used for training contain the movement speed label information of the point cloud.

[0176] In this embodiment, the scene flow estimation model can be trained based on the initial source point cloud and the initial target point cloud, which contain point cloud movement speed label information.

[0177] Step S402: Based on the movement speed label information and scene flow prediction results, obtain the total loss for the current round of training.

[0178] In this embodiment, step S402 may further include steps S4021 to S4023:

[0179] Step S4021: Obtain the point restoration loss of bidirectional point cloud reconstruction based on the movement speed label information.

[0180] In this embodiment, bidirectional point cloud reconstruction needs to be performed for a preset number of iterations. Since the initial source and target point clouds used for training are not sparse point clouds, they can be downsampled before bidirectional point cloud reconstruction. The point restoration loss for each iteration of bidirectional point cloud reconstruction can be obtained based on the movement speed label information and the unsampled initial source and target point clouds.

[0181] Step S4022: Obtain the scene flow loss based on the movement speed label information and the scene flow prediction results of the current round of training.

[0182] In this embodiment, the scene flow prediction ground truth between the initial source point cloud and the initial target point cloud can be obtained based on the movement speed label information, and the scene flow loss can be obtained based on the scene flow prediction ground truth and the scene flow prediction result.

[0183] Step S4023: Obtain the total loss based on the point restoration loss and the scene flow loss.

[0184] In this embodiment, the point restoration loss and scene flow loss can be calculated by weighted average and arithmetic average to obtain the total loss.

[0185] Step S403: Update the model parameters of the scene flow estimation model based on the total loss.

[0186] In this embodiment, the model parameters of the scene flow estimation model can be updated by gradient descent based on the total loss.

[0187] Step S404: Determine whether the training cutoff condition has been met. If not, proceed to step S401 for the next round of training; if yes, proceed to step S405.

[0188] Step S405: End training and obtain the trained scene flow estimation model.

[0189] In this implementation, training can end if the training cutoff condition is met; otherwise, the next round of training can begin.

[0190] Among them, the training cutoff condition is that the trained argument reaches a preset threshold number of rounds.

[0191] In one implementation, see Appendix Figure 5 , Figure 5 A schematic diagram illustrating the main training process of a scene flow estimation model according to one embodiment of this application. (See diagram below.) Figure 5 As shown, the scene flow estimation model may include the following steps S501 to S511:

[0192] Step S501: Input the initial source point cloud and the initial target point cloud for training.

[0193] Step S502: Determine whether the preset number of rounds threshold has been reached; if yes, end; if no, proceed to step S503.

[0194] Step S503: Determine whether the preset number of iterations for bidirectional point cloud reconstruction has been reached; if yes, proceed to step S509; if no, proceed to step S504.

[0195] Step S504: Extract point cloud features.

[0196] Step S505: Calculate the point cloud similarity matrix.

[0197] Step S506: Reconstruct the bidirectional point cloud.

[0198] Step S507: Calculate the point restoration loss.

[0199] Step S508: After the original point cloud and the reconstructed point cloud distribution are restored, proceed to step S503.

[0200] Step S509: Calculate the point cloud scene flow prediction results.

[0201] Step S510: Calculate the scene flow loss.

[0202] Step S511: Merge losses (obtain the total loss based on scene flow loss and point restoration loss) and optimize parameters before proceeding to step S502.

[0203] It should be noted that although the steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effect of this application, different steps do not necessarily have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders. These adjusted solutions are equivalent to the technical solutions described in this application and therefore will also fall within the protection scope of this application.

[0204] Those skilled in the art will understand that all or part of the processes in the method of the above-described embodiment can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0205] Another aspect of this application provides a computer-readable storage medium.

[0206] In one embodiment of a computer-readable storage medium according to this application, the computer-readable storage medium may be configured to store a program that performs the point cloud scene flow estimation method of the above-described method embodiments. This program may be loaded and run by a processor to implement the above-described point cloud scene flow estimation method. For ease of explanation, only the parts related to the embodiments of this application are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of this application. The computer-readable storage medium may be a storage device comprising various electronic devices. Optionally, in the embodiments of this application, the computer-readable storage medium is a non-transitory computer-readable storage medium.

[0207] Another aspect of this application provides a smart device.

[0208] In one embodiment of a smart device according to this application, the smart device may include at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program, which, when executed by the at least one processor, implements the method described in any of the above embodiments. The smart device described in this application may include driving equipment, smart vehicles, robots, and other devices. See appendix. Figure 2 , Figure 2 The image exemplarily illustrates a communication connection between memory 11 and processor 12 via a bus.

[0209] In some embodiments of this application, the smart device may further include at least one sensor for sensing information. The sensor is communicatively connected to any type of processor mentioned in this application. Optionally, the smart device may further include an autonomous driving system for guiding the smart device to drive autonomously or assisting in driving. The processor communicates with the sensor and / or the autonomous driving system to perform the methods described in any of the above embodiments. (The electronic devices described in this application may be, but are not limited to, mobile phones, tablets, desktop computers, laptops, handheld computers, notebook computers, in-vehicle devices, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), augmented reality (AR) / virtual reality (VR) devices, etc., and this application does not limit them.)

[0210] The technical solution of this application has been described above with reference to one embodiment shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. Without departing from the principles of this application, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of this application.

Claims

1. A point cloud scene flow estimation method, characterized in that, The method includes: Obtain the initial source point cloud and the initial target point cloud; the initial source point cloud and the initial target point cloud are point clouds from two consecutive frames at adjacent time points; The initial source point cloud and the initial target point cloud are used as input data for the initial iteration of the bidirectional point cloud reconstruction. Based on the input data, bidirectional point cloud reconstruction is performed for a preset number of iterations to obtain the source point cloud and the target point cloud for reconstruction. Based on the reconstructed source point cloud and the reconstructed target point cloud, obtain the point cloud scene flow prediction result; The step of performing bidirectional point cloud reconstruction based on input data for a preset number of iterations to obtain the source point cloud and the target point cloud includes: For each iteration of the bidirectional point cloud reconstruction, the point cloud correspondence between the two consecutive frames of point clouds in the input data of the current iteration is obtained; Based on the point cloud correspondence, the previous frame point cloud in the input data is used as the reference point cloud, and the next frame point cloud in the input data is used as the point cloud to be reconstructed. Point cloud reconstruction is performed to obtain the intermediate target point cloud. Based on the point cloud correspondence, the point cloud of the next frame in the input data is used as the reference point cloud, and the point cloud of the previous frame in the input data is used as the point cloud to be reconstructed. Point cloud reconstruction is performed to obtain the intermediate source point cloud. Determine that the current iteration count has reached the preset iteration count; If so, the intermediate source point cloud and the intermediate target point cloud are respectively used as the reconstructed source point cloud and the reconstructed target point cloud; If not, the intermediate source point cloud and the intermediate target point cloud are used as the input data for the next iteration, and the next iteration of bidirectional point cloud reconstruction is performed.

2. The point cloud scene flow estimation method according to claim 1, characterized in that, The step of obtaining the point cloud correspondence between two consecutive frames of point cloud in the input data of the current iteration includes: Feature extraction is performed on the two consecutive point clouds to obtain the relative features of all points in a single frame point cloud. Based on the relative features of a single frame point cloud, the point cloud alignment features of two consecutive frames are obtained through global cross-attention feature extraction. Based on the point cloud alignment features, the point cloud correspondence is obtained.

3. The point cloud scene flow estimation method according to claim 2, characterized in that, The step of obtaining the point cloud correspondence based on the point cloud alignment features includes: Based on the point cloud alignment features, obtain the point cloud similarity matrix between the two consecutive frames of point clouds; The point cloud similarity matrix is ​​used as the point cloud correspondence.

4. The point cloud scene flow estimation method according to claim 1, characterized in that, The point cloud reconstruction includes: Based on the correspondence between the reference point cloud and the point cloud, a weighted average is performed on the reference point cloud to obtain the missing point cloud in the point cloud to be reconstructed; The obtained point cloud and the point cloud to be reconstructed are distributed and merged to achieve the point cloud reconstruction.

5. The point cloud scene flow estimation method according to any one of claims 1 to 4, characterized in that, The step of obtaining point cloud scene flow prediction results based on the reconstructed source point cloud and the reconstructed target point cloud includes: The displacement difference between the reconstructed source point cloud and the reconstructed target point cloud is obtained as the point cloud scene flow prediction result.

6. The point cloud scene flow estimation method according to claim 1, characterized in that, The method further includes: Based on a pre-trained scene flow estimation model, bidirectional point cloud reconstruction is performed based on the initial source point cloud and the initial target point cloud, and the point cloud scene flow prediction result is obtained based on the reconstructed source point cloud and the reconstructed target point cloud.

7. The point cloud scene flow estimation method according to claim 6, characterized in that, The method further includes training the scene flow estimation model according to the following steps: For each training round, the initial source point cloud and initial target point cloud used for training are input into the scene flow estimation model to obtain the scene flow prediction result of the current training round; the initial source point cloud and initial target point cloud used for training contain the movement speed label information of the point cloud. Based on the movement speed label information and the scene flow prediction results, obtain the total loss for the current training round; Based on the total loss, update the model parameters of the scene flow estimation model; and determine whether the training cutoff condition has been met. If the training cutoff condition is not met, proceed to the next round of training; If the training cutoff condition is met, training ends to obtain the trained scene flow estimation model.

8. The point cloud scene flow estimation method according to claim 7, characterized in that, The step of obtaining the total loss for the current training round based on the movement speed label information and the scene flow prediction result includes: Based on the movement speed label information, obtain the point restoration loss of the bidirectional point cloud reconstruction; Based on the movement speed label information and the scene flow prediction results of the current round of training, obtain the scene flow loss; The total loss is obtained based on the point restoration loss and the scene flow loss.

9. The point cloud scene flow estimation method according to claim 7 or 8, characterized in that, The training cutoff condition is when the number of training rounds reaches a preset round threshold.

10. A smart device, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores a computer program, which, when executed by the at least one processor, implements the point cloud scene flow estimation method according to any one of claims 1 to 9.

11. A computer-readable storage medium storing a plurality of program codes, characterized in that, The program code is adapted to be loaded and run by a processor to perform the point cloud scene flow estimation method according to any one of claims 1 to 9.

Images