Point cloud scene flow estimation method, readable storage medium and intelligent equipment

Abstract

The invention relates to the technical field of automatic driving, in particular to a point cloud scene flow estimation method, a readable storage medium and intelligent equipment, and aims to solve the technical problem of low scene flow estimation accuracy for sparse point clouds. In order to achieve the purpose, the method comprises the steps of obtaining an initial source point cloud and an initial target point cloud, performing bidirectional point cloud reconstruction based on the initial source point cloud and the initial target point cloud, obtaining a reconstructed source point cloud and a reconstructed target point cloud, and obtaining a scene flow prediction result according to the reconstructed source point cloud and the reconstructed target point cloud. Bidirectional point cloud reconstruction is carried out based on the initial source point cloud and the initial target point cloud, the problem of sparsity of actually acquired point clouds can be solved, and the problem of inconsistent distribution such as noise and dislocation caused by the sparsity of the point clouds is reduced. Meanwhile, the problem that the shared point cloud proportion between continuous frames is rapidly reduced due to sparsity can be avoided through bidirectional point cloud reconstruction, and the prediction accuracy of point cloud scene flow prediction can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a point cloud scene flow estimation method, a readable storage medium, and an intelligent device. Background Art

[0002] Point cloud scene flow estimation is a fundamental task in autonomous driving and is used for various tasks such as object detection and object tracking. It provides motion estimation of initial frame data by predicting the motion speed of pixels or point clouds.

[0003] The point cloud data obtained by lidar scanning is inherently sparse. In scene flow tasks, sparsity can lead to a rapid decrease in the proportion of point clouds shared between consecutive frames and misalignment between given point pairs, resulting in an excessively large proportion of predicted anomalies and a decrease in prediction accuracy.

[0004] Initially, point cloud scene flow was treated as a regression task, directly predicting the 3D spatial velocity of each point based on its coordinates. However, due to the difficulty of regressing space in 3D scenes, this approach requires a large amount of supervised data and is difficult to converge. Some methods attempt to avoid this problem through runtime optimization, which can obtain more accurate flow predictions without training data. However, these methods assume that there is distribution consistency between given source and target points and are susceptible to noisy input data without prior learning. Subsequent methods typically use matching mechanisms to calculate point flow. From a matching perspective, point scene flow is regarded as the displacement between the self point and its matching target in a given source / target point pair, and the matching relationship is obtained through the similarity matrix determined by the point embedding. However, this method heavily relies on the existence of point correspondences.

[0005] Accordingly, the field needs a new point cloud scene flow estimation solution to solve the above problems. Summary of the Invention

[0006] In order to overcome the above-mentioned defects, the present application is proposed to solve or at least partially solve the technical problem of how to overcome the low accuracy of scene flow estimation for sparse point clouds.

[0007] In a first aspect, a point cloud scene flow estimation method is provided, the method comprising:

[0008] Acquire an initial source point cloud and an initial target point cloud; the initial source point cloud and the initial target point cloud are point clouds of two frames before and after adjacent moments;

[0009] Based on the initial source point cloud and the initial target point cloud, bidirectional point cloud reconstruction is performed to obtain a reconstructed source point cloud and a reconstructed target point cloud; the bidirectional point cloud reconstruction is to reconstruct the point cloud of the next frame of point cloud based on the previous frame of point cloud, and to reconstruct the point cloud of the previous frame of point cloud based on the next frame of point cloud;

[0010] A point cloud scene flow prediction result is obtained based on the reconstructed source point cloud and the reconstructed target point cloud.

[0011] In one technical solution of the above-mentioned point cloud scene flow estimation method, performing bidirectional point cloud reconstruction based on the initial source point cloud and the initial target point cloud to obtain a reconstructed source point cloud and a reconstructed target point cloud includes:

[0012] Using the initial source point cloud and the initial target point cloud as input data for an initial iteration of the bidirectional point cloud reconstruction;

[0013] Based on the input data, bidirectional point cloud reconstruction is performed for a preset number of iterations to obtain the reconstructed source point cloud and the reconstructed target point cloud.

[0014] In one technical solution of the above-mentioned point cloud scene flow estimation method, performing a preset number of iterations of bidirectional point cloud reconstruction based on the input data to obtain the reconstructed source point cloud and the reconstructed target point cloud includes:

[0015] For each iteration of the bidirectional point cloud reconstruction, obtaining a point cloud correspondence relationship between two frames of point clouds in the input data of the current iteration;

[0016] Based on the point cloud correspondence, the previous frame point cloud in the input data is used as the reference point cloud, and the next frame point cloud in the input data is used as the point cloud to be reconstructed, and point cloud reconstruction is performed to obtain an intermediate target point cloud;

[0017] Based on the point cloud correspondence, the point cloud of the next frame in the input data is used as the reference point cloud, and the point cloud of the previous frame in the input data is used as the point cloud to be reconstructed, and point cloud reconstruction is performed to obtain an intermediate source point cloud;

[0018] Determining that the current number of iterations reaches the preset number of iterations;

[0019] If yes, the intermediate source point cloud and the intermediate target point cloud are used as the reconstructed source point cloud and the reconstructed target point cloud respectively;

[0020] If not, the intermediate source point cloud and the intermediate target point cloud are respectively used as the input data for the next iteration, and the next iteration of the bidirectional point cloud reconstruction is performed.

[0021] In one technical solution of the above-mentioned point cloud scene flow estimation method, obtaining the point cloud correspondence relationship between the previous and next frames of point cloud in the input data in the current iteration includes:

[0022] Performing feature extraction on the two frames of point cloud respectively to obtain relative features of all points in a single frame of point cloud;

[0023] According to the relative features of the single-frame point cloud, the point cloud alignment features of the previous and next two frames of point cloud are obtained by global cross-attention feature extraction;

[0024] The point cloud correspondence is obtained according to the point cloud alignment feature.

[0025] In one technical solution of the above-mentioned point cloud scene flow estimation method, obtaining the point cloud correspondence relationship according to the point cloud alignment feature includes:

[0026] Obtaining a point cloud similarity matrix between the two frames of point cloud according to the point cloud alignment feature;

[0027] The point cloud similarity matrix is ​​used as the point cloud correspondence.

[0028] In one technical solution of the above-mentioned point cloud scene flow estimation method, the point cloud reconstruction includes:

[0029] performing weighted averaging on the reference point cloud according to the correspondence between the reference point cloud and the point cloud to obtain a point cloud missing from the point cloud to be reconstructed;

[0030] The obtained point cloud and the point cloud to be reconstructed are distributed and merged to achieve the point cloud reconstruction.

[0031] In one technical solution of the above-mentioned point cloud scene flow estimation method, obtaining a point cloud scene flow prediction result based on the reconstructed source point cloud and the reconstructed target point cloud includes:

[0032] A displacement difference between the reconstructed source point cloud and the reconstructed target point cloud is obtained as the point cloud scene flow prediction result.

[0033] In one technical solution of the above-mentioned point cloud scene flow estimation method, the method further includes:

[0034] Based on the preset trained scene flow estimation model, bidirectional point cloud reconstruction is performed based on the initial source point cloud and the initial target point cloud, and the point cloud scene flow prediction result is obtained according to the obtained reconstructed source point cloud and reconstructed target point cloud.

[0035] In one technical solution of the above-mentioned point cloud scene flow estimation method, the method further includes training the scene flow estimation model according to the following steps:

[0036] For each round of training, the initial source point cloud and the initial target point cloud used for training are input into the scene flow estimation model to obtain the scene flow prediction results of the current round of training; the initial source point cloud and the initial target point cloud used for training contain the movement speed label information of the point cloud;

[0037] Obtaining a total loss for a current round of training based on the movement speed label information and the scene flow prediction result;

[0038] updating the model parameters of the scene flow estimation model according to the total loss; and determining whether a training cutoff condition is met;

[0039] If the training end condition is not met, proceed to the next round of training;

[0040] If the training cutoff condition is reached, the training is terminated to obtain the trained scene flow estimation model.

[0041] In one technical solution of the above-mentioned point cloud scene flow estimation method, obtaining the total loss of the current round of training based on the movement speed label information and the scene flow prediction result includes:

[0042] Obtaining a point restoration loss of the bidirectional point cloud reconstruction according to the movement speed label information;

[0043] Obtaining a scene flow loss according to the movement speed label information and the scene flow prediction result of the current round of training;

[0044] The total loss is obtained according to the point restoration loss and the scene flow loss.

[0045] In one technical solution of the above-mentioned point cloud scene flow estimation method, the training cutoff condition is that the number of training rounds reaches a preset round threshold.

[0046] In a second aspect, an intelligent device is provided, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program, and when the computer program is executed by the at least one processor, the method described in any one of the technical solutions of the above-mentioned point cloud scene flow estimation method is implemented.

[0047] In a third aspect, a computer-readable storage medium is provided, which stores a plurality of program codes, wherein the program codes are suitable for being loaded and run by a processor to execute the method described in any one of the technical solutions of the above-mentioned point cloud scene flow estimation method.

[0048] Solution 1. A point cloud scene flow estimation method, characterized in that the method comprises:

[0049] Acquire an initial source point cloud and an initial target point cloud; the initial source point cloud and the initial target point cloud are point clouds of two frames before and after adjacent moments;

[0050] Based on the initial source point cloud and the initial target point cloud, bidirectional point cloud reconstruction is performed to obtain a reconstructed source point cloud and a reconstructed target point cloud; the bidirectional point cloud reconstruction is to reconstruct the point cloud of the next frame of point cloud based on the previous frame of point cloud, and to reconstruct the point cloud of the previous frame of point cloud based on the next frame of point cloud;

[0051] A point cloud scene flow prediction result is obtained based on the reconstructed source point cloud and the reconstructed target point cloud.

[0052] Solution 2. The point cloud scene flow estimation method according to Solution 1 is characterized in that:

[0053] The performing bidirectional point cloud reconstruction based on the initial source point cloud and the initial target point cloud to obtain a reconstructed source point cloud and a reconstructed target point cloud includes:

[0054] Using the initial source point cloud and the initial target point cloud as input data for an initial iteration of the bidirectional point cloud reconstruction;

[0055] Based on the input data, bidirectional point cloud reconstruction is performed for a preset number of iterations to obtain the reconstructed source point cloud and the reconstructed target point cloud.

[0056] Solution 3. The point cloud scene flow estimation method according to Solution 2 is characterized in that:

[0057] The step of performing bidirectional point cloud reconstruction for a preset number of iterations based on the input data to obtain the reconstructed source point cloud and the reconstructed target point cloud includes:

[0058] For each iteration of the bidirectional point cloud reconstruction, obtaining a point cloud correspondence relationship between two frames of point clouds in the input data of the current iteration;

[0059] Based on the point cloud correspondence, the previous frame point cloud in the input data is used as the reference point cloud, and the next frame point cloud in the input data is used as the point cloud to be reconstructed, and point cloud reconstruction is performed to obtain an intermediate target point cloud;

[0060] Based on the point cloud correspondence, the point cloud of the next frame in the input data is used as the reference point cloud, and the point cloud of the previous frame in the input data is used as the point cloud to be reconstructed, and point cloud reconstruction is performed to obtain an intermediate source point cloud;

[0061] Determining that the current number of iterations reaches the preset number of iterations;

[0062] If yes, the intermediate source point cloud and the intermediate target point cloud are used as the reconstructed source point cloud and the reconstructed target point cloud respectively;

[0063] If not, the intermediate source point cloud and the intermediate target point cloud are respectively used as the input data for the next iteration, and the next iteration of the bidirectional point cloud reconstruction is performed.

[0064] Solution 4. The point cloud scene flow estimation method according to Solution 3 is characterized in that:

[0065] The step of obtaining the point cloud correspondence between two frames of point clouds in the input data of the current iteration includes:

[0066] Performing feature extraction on the two frames of point cloud respectively to obtain relative features of all points in a single frame of point cloud;

[0067] According to the relative features of the single-frame point cloud, the point cloud alignment features of the previous and next two frames of point cloud are obtained by global cross-attention feature extraction;

[0068] The point cloud correspondence is obtained according to the point cloud alignment feature.

[0069] Solution 5. The point cloud scene flow estimation method according to Solution 4 is characterized in that:

[0070] The acquiring the point cloud correspondence relationship according to the point cloud alignment feature includes:

[0071] Obtaining a point cloud similarity matrix between the two frames of point cloud according to the point cloud alignment feature;

[0072] The point cloud similarity matrix is ​​used as the point cloud correspondence.

[0073] Solution 6. The point cloud scene flow estimation method according to Solution 3 is characterized in that:

[0074] The point cloud reconstruction comprises:

[0075] performing weighted averaging on the reference point cloud according to the correspondence between the reference point cloud and the point cloud to obtain a point cloud missing from the point cloud to be reconstructed;

[0076] The obtained point cloud and the point cloud to be reconstructed are distributed and merged to achieve the point cloud reconstruction.

[0077] Solution 7. The point cloud scene flow estimation method according to any one of Solutions 1 to 6, characterized in that:

[0078] The obtaining of a point cloud scene flow prediction result according to the reconstructed source point cloud and the reconstructed target point cloud includes:

[0079] A displacement difference between the reconstructed source point cloud and the reconstructed target point cloud is obtained as the point cloud scene flow prediction result.

[0080] Solution 8. The point cloud scene flow estimation method according to Solution 1, further comprising:

[0081] Based on the preset trained scene flow estimation model, bidirectional point cloud reconstruction is performed based on the initial source point cloud and the initial target point cloud, and the point cloud scene flow prediction result is obtained according to the obtained reconstructed source point cloud and reconstructed target point cloud.

[0082] Solution 9. The point cloud scene flow estimation method according to Solution 8 is characterized in that:

[0083] The method further comprises training the scene flow estimation model according to the following steps:

[0084] For each round of training, the initial source point cloud and the initial target point cloud used for training are input into the scene flow estimation model to obtain the scene flow prediction results of the current round of training; the initial source point cloud and the initial target point cloud used for training contain the movement speed label information of the point cloud;

[0085] Obtaining a total loss for a current round of training based on the movement speed label information and the scene flow prediction result;

[0086] updating the model parameters of the scene flow estimation model according to the total loss; and determining whether a training cutoff condition is met;

[0087] If the training end condition is not met, proceed to the next round of training;

[0088] If the training cutoff condition is reached, the training is terminated to obtain the trained scene flow estimation model.

[0089] Solution 10. The point cloud scene flow estimation method according to Solution 9 is characterized in that:

[0090] The obtaining of the total loss of the current round of training according to the movement speed label information and the scene flow prediction result includes:

[0091] Obtaining a point restoration loss of the bidirectional point cloud reconstruction according to the movement speed label information;

[0092] Obtaining a scene flow loss according to the movement speed label information and the scene flow prediction result of the current round of training;

[0093] The total loss is obtained according to the point restoration loss and the scene flow loss.

[0094] Solution 11. The point cloud scene flow estimation method according to Solution 9 or 10 is characterized in that:

[0095] The training cutoff condition is that the number of training rounds reaches a preset round number threshold.

[0096] Solution 12. A smart device, comprising:

[0097] at least one processor;

[0098] and, a memory communicatively coupled to the at least one processor;

[0099] Wherein, a computer program is stored in the memory, and when the computer program is executed by the at least one processor, the point cloud scene flow estimation method described in any one of solutions 1 to 11 is implemented.

[0100] Solution 13. A computer-readable storage medium storing a plurality of program codes, characterized in that the program codes are suitable for being loaded and run by a processor to execute the point cloud scene flow estimation method described in any one of Solutions 1 to 11.

[0101] The above one or more technical solutions of this application have at least one or more of the following beneficial effects:

[0102] In implementing the technical solution of the point cloud scene flow estimation method provided by this application, this application obtains an initial source point cloud and an initial target point cloud, performs bidirectional point cloud reconstruction based on the initial source point cloud and the initial target point cloud, obtains a reconstructed source point cloud and a reconstructed target point cloud, and obtains a scene flow prediction result based on the reconstructed source point cloud and the reconstructed target point cloud. Through the above configuration, this application performs bidirectional point cloud reconstruction based on the initial source point cloud and the initial target point cloud, which can overcome the sparsity problem of the actual collected point cloud and reduce the problem of inconsistent distribution such as noise and misalignment caused by the sparsity of the point cloud. At the same time, the bidirectional point cloud reconstruction can also avoid the problem of a rapid decrease in the proportion of shared point clouds between consecutive frames due to sparsity, which can effectively improve the prediction accuracy of the point cloud scene flow prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0103] The disclosure of this application will become more easily understood with reference to the accompanying drawings. Those skilled in the art will readily appreciate that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this application. Among them:

[0104] Figure 1 This is a flow chart of the main steps of a point cloud scene flow estimation method according to an embodiment of the present application;

[0105] Figure 2 This is a main implementation system block diagram of bidirectional point cloud reconstruction according to an implementation method of the present application;

[0106] Figure 3 is a schematic diagram of a scene flow prediction result according to an example of an embodiment of the present application;

[0107] Figure 4 1 is a flow chart of the main steps of a point cloud scene flow estimation method according to an implementation of an embodiment of the present application;

[0108] Figure 5 A schematic diagram of the main training process of a scene flow estimation model according to one implementation of an embodiment of the present application;

[0109] Figure 6 It is a schematic diagram of the main structure of a smart device according to an embodiment of the present application.

[0110] Reference numerals:

[0111] 11: Memory; 12: Processor; 21: First frame point cloud; 22: The actual point cloud after movement at the next moment of the first frame point cloud; 23: The point cloud at the next moment predicted based on the first frame point cloud. DETAILED DESCRIPTION

[0112] Some embodiments of the present application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present application and are not intended to limit the scope of protection of the present application.

[0113] In the description of this application, "module" and "processor" may include hardware, software, or a combination of both. A module may include hardware circuitry, various suitable sensors, communication ports, and memory. It may also include software components, such as program code, or a combination of software and hardware. A processor may be a central processing unit (CPU), a microprocessor, an image processor, a digital signal processor, or any other suitable processor. A processor has data and / or signal processing capabilities. A processor may be implemented in software, hardware, or a combination of both. Computer-readable storage media include any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc. The term "A and / or B" refers to all possible combinations of A and B, such as only A, only B, or both A and B. The terms "at least one of A or B" or "at least one of A and B" have similar meanings to "A and / or B" and may include only A, only B, or both A and B. The singular forms "a" and "the" may also include the plural forms.

[0114] The relevant user personal information that may be involved in the various embodiments of this application is strictly in accordance with the requirements of laws and regulations, following the principles of legality, legitimacy and necessity, and based on the reasonable purposes of business scenarios, to process the personal information that users actively provide during the use of products / services or generated due to the use of products / services, as well as the personal information obtained with the user's authorization.

[0115] The user personal information processed by this application will vary depending on the specific product / service scenario and must be based on the specific scenario in which the user uses the product / service. This may involve the user's account information, device information, driving information, vehicle information, or other related information. This application will treat the user's personal information and its processing with a high degree of diligence.

[0116] This application attaches great importance to the security of user personal information and has taken reasonable and feasible security protection measures that comply with industry standards to protect user information and prevent personal information from being accessed, disclosed, used, modified, damaged or lost without authorization.

[0117] Here we first explain some terms involved in this application.

[0118] Scene flow is a three-dimensional version of optical flow, which describes the changes of each point in an image or point cloud between the previous and next frames.

[0119] See attached Figure 1 , Figure 1 FIG. 1 is a flow chart showing the main steps of a point cloud scene flow estimation method according to an embodiment of the present application. Figure 1 As shown, the point cloud scene flow estimation method in the embodiment of the present application mainly includes the following steps S101 to S103.

[0120] Step S101: Acquire an initial source point cloud and an initial target point cloud; the initial source point cloud and the initial target point cloud are point clouds of two frames before and after adjacent moments.

[0121] In this embodiment, an initial source point cloud and an initial target point cloud for scene flow prediction may be obtained.

[0122] In one embodiment, the initial source point cloud and the initial target point cloud may be acquired based on lidar acquisition.

[0123] Step S102: Based on the initial source point cloud and the initial target point cloud, bidirectional point cloud reconstruction is performed to obtain a reconstructed source point cloud and a reconstructed target point cloud; bidirectional point cloud reconstruction is to reconstruct the point cloud of the next frame of point cloud based on the previous frame of point cloud, and to reconstruct the point cloud of the previous frame of point cloud based on the next frame of point cloud.

[0124] In this embodiment, due to the sparsity between the initial source point cloud and the initial target point cloud, bidirectional point cloud reconstruction can be performed on the initial source point cloud and the initial target point cloud. This involves reconstructing the initial target point cloud based on the initial source point cloud, and then reconstructing the initial source point cloud based on the initial target point cloud, thereby obtaining a reconstructed source point cloud and a reconstructed target point cloud. This bidirectional point cloud reconstruction process effectively avoids the issues of isolated and inconsistent point cloud distribution between frames caused by unidirectional point cloud reconstruction.

[0125] Step S103: Obtaining a point cloud scene flow prediction result based on the reconstructed source point cloud and the reconstructed target point cloud.

[0126] In this embodiment, a point cloud scene flow prediction result may be obtained based on the reconstructed source point cloud and the reconstructed target point cloud.

[0127] In one embodiment, the displacement difference between the reconstructed source point cloud and the reconstructed target point cloud can be obtained as the point cloud scene flow prediction result.

[0128] Based on the method described in steps S101 to S103 above, the embodiment of the present application obtains the initial source point cloud and the initial target point cloud, performs bidirectional point cloud reconstruction based on the initial source point cloud and the initial target point cloud, obtains the reconstructed source point cloud and the reconstructed target point cloud, and obtains the scene flow prediction result based on the reconstructed source point cloud and the reconstructed target point cloud. Through the above configuration, the embodiment of the present application performs bidirectional point cloud reconstruction based on the initial source point cloud and the initial target point cloud, which can overcome the sparsity problem of the actual collected point cloud and reduce the problem of inconsistent distribution such as noise and misalignment caused by the sparsity of the point cloud. At the same time, the bidirectional point cloud reconstruction can also avoid the problem of rapid reduction in the proportion of shared point clouds between consecutive frames due to sparsity, which can effectively improve the prediction accuracy of the point cloud scene flow prediction.

[0129] The following further describes step S102.

[0130] In one implementation of the embodiment of the present application, step S102 may further include the following steps S1021 and S1022:

[0131] Step S1021: using the initial source point cloud and the initial target point cloud as input data for an initial iteration of bidirectional point cloud reconstruction.

[0132] In this embodiment, the initial source point cloud and the initial target point cloud may be used as input data for an initial iteration of bidirectional point cloud reconstruction.

[0133] Step S1022: Based on the input data, bidirectional point cloud reconstruction is performed for a preset number of iterations to obtain a reconstructed source point cloud and a reconstructed target point cloud.

[0134] In this embodiment, step S1022 may further include the following steps S10221 to S10226:

[0135] Step S10221: For each iteration of bidirectional point cloud reconstruction, obtain the point cloud correspondence between the previous and next two frames of point cloud in the input data in the current iteration.

[0136] In this embodiment, step S10221 may further include the following steps S102211 to S102213:

[0137] Step S102211: Perform feature extraction on the two frames of point cloud respectively to obtain the relative features of all points in the single frame point cloud respectively.

[0138] In this embodiment, a graph neural network can be used to extract point domain features from the previous and next frame point clouds. A local point cloud attention layer is then applied to obtain relative information, i.e., relative features, of all points within a single frame based on the point domain features.

[0139] Step S102212: Based on the relative features of the single-frame point cloud, the point cloud alignment features of the previous and next two frames of point cloud are obtained through global cross-attention feature extraction.

[0140] In this embodiment, global cross attention extraction can be performed based on the relative features of a single-frame point cloud through a global cross attention layer to obtain point cloud alignment features of the previous and next two frames of point clouds.

[0141] Step S102213: Obtain the point cloud correspondence based on the point cloud alignment features.

[0142] In this embodiment, a point cloud similarity matrix between two frames of point cloud can be obtained based on the point cloud alignment feature, and the point cloud similarity matrix can be used as the point cloud correspondence relationship.

[0143] Step S10222: Based on the point cloud correspondence, the previous frame point cloud in the input data is used as the reference point cloud, and the next frame point cloud in the input data is used as the point cloud to be reconstructed, and point cloud reconstruction is performed to obtain an intermediate target point cloud.

[0144] Step S10223: Based on the point cloud correspondence, the point cloud of the next frame in the input data is used as the reference point cloud, and the point cloud of the previous frame in the input data is used as the point cloud to be reconstructed, and point cloud reconstruction is performed to obtain an intermediate source point cloud.

[0145] In this embodiment, the point cloud of the next frame can be reconstructed based on the point cloud of the previous frame, and the point cloud of the previous frame can be reconstructed based on the point cloud of the next frame. Specifically, the point cloud reconstruction can be performed based on the following steps S201 to S202:

[0146] Step S201: performing weighted averaging on the reference point cloud according to the corresponding relationship between the reference point cloud and the point cloud to obtain the point cloud missing from the point cloud to be reconstructed.

[0147] In this embodiment, a weighted average may be performed on the reference point cloud based on the reference point cloud and the point cloud similarity matrix between the two preceding and following frames of point cloud, thereby obtaining the point cloud missing from the point cloud to be reconstructed.

[0148] Step S202: Distribute and merge the obtained point cloud and the point cloud to be reconstructed to achieve point cloud reconstruction.

[0149] In this embodiment, the obtained point cloud and the point cloud to be reconstructed may be distributed and merged to achieve point cloud reconstruction.

[0150] Step S10224: Determine whether the current number of iterations reaches the preset number of iterations.

[0151] In this embodiment, it is possible to determine whether the current number of iterations reaches a preset number of iterations. If so, step S10225 is executed; if not, step S10226 is executed.

[0152] Step S10225: The intermediate source point cloud and the intermediate target point cloud are used as the reconstructed source point cloud and the reconstructed target point cloud respectively.

[0153] In this embodiment, if the bidirectional point cloud reconstruction reaches a preset number of iterations, the intermediate source point cloud obtained in the current iteration can be used as the reconstruction source point cloud, and the intermediate target point cloud obtained in the current iteration can be used as the reconstruction target point cloud.

[0154] Step S10226: The intermediate source point cloud and the intermediate target point cloud are used as input data for the next iteration respectively, and the next iteration of bidirectional point cloud reconstruction is performed.

[0155] In this embodiment, if the bidirectional point cloud iteration does not reach the preset number of iterations, the intermediate source point cloud and the intermediate target point cloud obtained in the current iteration can be used as input data for the next iteration, and the next iteration of bidirectional point cloud reconstruction can be performed.

[0156] In one embodiment, please refer to the attached Figure 2 , Figure 2 This is a main system block diagram for implementing bidirectional point cloud reconstruction according to an embodiment of the present application. Figure 2 As shown, each iteration in the bidirectional point cloud reconstruction can be achieved by Figure 2 The basic module of the loop shown in the upper figure is implemented. Specifically, the input data includes the source point cloud (p s,j ) and target point cloud (p t,j ), input the input point cloud to the similarity alignment module to obtain the point cloud similarity matrix; based on the correspondence reconstruction module, the point cloud similarity matrix, the source point cloud and the target point cloud are used to reconstruct the point cloud, and the candidate points of the source point cloud and the target point cloud are obtained (that is, the point cloud missing in the point cloud to be reconstructed is obtained); based on the distribution optimization module, the distribution merging of the point cloud is realized to obtain the reconstructed source point cloud (p s,j+1 ) and target point cloud (p t,j+1 ). The bidirectional point cloud reconstruction can be performed for a preset number of iterations, i.e. Figure 2 As shown in the figure below, the initial source point cloud (p s,0 ) and the initial target point cloud (p t,0) is input into the loop basic module and multiple iterations are performed. Each iteration can obtain the intermediate source point cloud (p s,j ) and intermediate target point cloud (p t,j ). When the preset number of iterations is reached, the reconstructed source point cloud (p s,k ) and reconstructed target point cloud (p t,k ).

[0157] For an example, see the attached Figure 3 , Figure 3 FIG. 1 is a schematic diagram of a scene flow prediction result according to an example of an embodiment of the present application. Figure 3 As shown, reference numeral 21 corresponds to the source point cloud of the first frame, reference numeral 22 is the target point cloud after the source point cloud actually moves at the next moment, and reference numeral 22 is the position of the target point cloud predicted according to the scene flow prediction result obtained in this application. Figure 3 It can be seen from the figure that the scene flow prediction results obtained by this application can achieve better prediction results in the case of sparse point clouds.

[0158] In one embodiment, please refer to the attached Figure 4 , Figure 4 FIG. 1 is a flow chart of the main steps of a point cloud scene flow estimation method according to an embodiment of the present application. Figure 4 As shown, the point cloud scene flow estimation method may include the following steps S301 to S307:

[0159] Step S301: Initial source point cloud and initial target point cloud are input.

[0160] In this embodiment, step S301 is similar to the method described in the aforementioned step S101, and for the sake of simplicity, it will not be repeated here.

[0161] Step S302: Determine whether the bidirectional point cloud reconstruction reaches a preset number of iterations; if so, execute step S307; if not, execute step 303.

[0162] In this embodiment, step S302 is similar to the method described in the aforementioned step S10224, and for the sake of simplicity, it is not repeated here.

[0163] Step S303: extracting point cloud features.

[0164] In this embodiment, step S303 is similar to the method described in the aforementioned steps S102211 and S10212, and will not be repeated here for the sake of simplicity.

[0165] Step S304: Calculate the point cloud similarity matrix.

[0166] In this embodiment, step S304 is similar to the method described in the aforementioned steps S102 and S13, and for the sake of simplicity, it will not be repeated here.

[0167] Step S305: Reconstructing a bidirectional point cloud.

[0168] In this embodiment, step S305 is similar to the method described in the aforementioned steps S10222 and S10223, and will not be repeated here for the sake of simplicity.

[0169] Step S306: After the distribution of the original point cloud (point cloud to be reconstructed) and the reconstructed point cloud is restored, jump to step S302.

[0170] In this embodiment, step S306 is similar to the method described in the aforementioned step S202, and for the sake of simplicity, it is not repeated here.

[0171] Step S307: Calculate the point cloud scene flow prediction result.

[0172] In this embodiment, step S307 is similar to the method described in the aforementioned step S103, and for the sake of simplicity, it will not be repeated here.

[0173] In one embodiment of the present application, bidirectional point cloud reconstruction can be performed based on a preset trained scene flow estimation model, based on the initial source point cloud and the initial target point cloud, and the point cloud scene flow prediction result can be obtained based on the obtained reconstructed source point cloud and the reconstructed target point cloud.

[0174] In one embodiment, the scene flow estimation model may be trained according to the following steps S401 to S405:

[0175] Step S401: For each round of training, the initial source point cloud and the initial target point cloud used for training are input into the scene flow estimation model to obtain the scene flow prediction results of the current round of training; the initial source point cloud and the initial target point cloud used for training contain the movement speed label information of the point cloud.

[0176] In this embodiment, the scene flow estimation model may be trained based on the initial source point cloud and the initial target point cloud containing point cloud movement speed label information.

[0177] Step S402: Obtain the total loss of the current round of training based on the movement speed label information and the scene flow prediction result.

[0178] In this embodiment, step S402 may further include the following steps S4021 to S4023:

[0179] Step S4021: Obtaining point restoration loss of bidirectional point cloud reconstruction according to the movement speed tag information.

[0180] In this embodiment, bidirectional point cloud reconstruction is performed for a preset number of iterations. The initial source and target point clouds used for training are not sparse point clouds. Therefore, the initial source and target point clouds can be downsampled before bidirectional point cloud reconstruction. The point restoration loss for each iteration of bidirectional point cloud reconstruction can be obtained based on the movement speed label information and the un-downsampled initial source and target point clouds.

[0181] Step S4022: Obtain scene flow loss based on the movement speed label information and the scene flow prediction result of the current round of training.

[0182] In this embodiment, the scene flow prediction true value between the initial source point cloud and the initial target point cloud can be obtained according to the moving speed label information, and the scene flow loss can be obtained according to the scene flow prediction true value and the scene flow prediction result.

[0183] Step S4023: Obtain the total loss based on the point restoration loss and the scene flow loss.

[0184] In this embodiment, the point restoration loss and the scene flow loss may be weighted averaged and arithmetic averaged to obtain the total loss.

[0185] Step S403: Update the model parameters of the scene flow estimation model according to the total loss.

[0186] In this embodiment, gradient descent can be performed to update the model parameters of the scene flow estimation model based on the total loss.

[0187] Step S404: Determine whether the training end condition is met. If not, jump to step S401 for the next round of training; if so, jump to step S405.

[0188] Step S405: End the training and obtain the trained scene flow estimation model.

[0189] In this embodiment, if the training meets the training end condition, the training may be terminated; if the training end condition is not met, the next round of training may be performed.

[0190] The training cutoff condition may be that the training discussion reaches a preset round number threshold.

[0191] In one embodiment, please refer to the attached Figure 5 , Figure 5 Schematic diagram of the main training process of the scene flow estimation model according to an embodiment of the present application. Figure 5 As shown, the scene flow estimation model may include the following steps S501 to S511:

[0192] Step S501: Input the initial source point cloud and the initial target point cloud for training.

[0193] Step S502: Determine whether the preset round number threshold is reached; if so, end; if not, execute step S503.

[0194] Step S503: Determine whether the preset number of iterations for bidirectional point cloud reconstruction has been reached; if so, execute step S509; if not, execute step S504.

[0195] Step S504: extracting point cloud features.

[0196] Step S505: Calculate the point cloud similarity matrix.

[0197] Step S506: Reconstructing a bidirectional point cloud.

[0198] Step S507: Calculate point restoration loss.

[0199] Step S508: After the distribution of the original point cloud and the reconstructed point cloud is restored, jump to step S503.

[0200] Step S509: Calculate the point cloud scene flow prediction result.

[0201] Step S510: Calculate scene flow loss.

[0202] Step S511: Merge the loss (according to the scene flow loss and the point restoration loss, the total loss is obtained) to optimize the parameters and then jump to step S502.

[0203] It should be pointed out that although the various steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effect of the present application, different steps do not have to be performed in such an order. They can be performed simultaneously (in parallel) or in other orders. These adjusted solutions are equivalent to the technical solutions described in this application, and therefore will also fall within the scope of protection of this application.

[0204] It will be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment of the present application can also be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the above-mentioned various method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device, medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory, random access memory, electric carrier signal, telecommunication signal and software distribution medium, etc. that can carry the computer program code.

[0205] Another aspect of the present application provides a computer-readable storage medium.

[0206] In an embodiment of a computer-readable storage medium according to the present application, the computer-readable storage medium can be configured to store a program for executing the point cloud scene flow estimation method of the above-mentioned method embodiment, and the program can be loaded and run by the processor to implement the above-mentioned point cloud scene flow estimation method. For ease of explanation, only the parts related to the embodiment of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present application. The computer-readable storage medium can be a storage device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiment of the present application is a non-transitory computer-readable storage medium.

[0207] Another aspect of the present application provides a smart device.

[0208] In an embodiment of an intelligent device according to the present application, the intelligent device may include at least one processor; and a memory in communication with the at least one processor; wherein the memory stores a computer program, and when the computer program is executed by the at least one processor, the method described in any of the above embodiments is implemented. The intelligent device described in the present application may include a driving device, a smart car, a robot, and the like. Figure 2 , Figure 2 exemplarily shows that the memory 11 and the processor 12 are communicatively connected via a bus.

[0209] In some embodiments of the present application, the smart device may further include at least one sensor, which is used to sense information. The sensor is communicatively connected to any type of processor mentioned in this application. Optionally, the smart device may further include an autonomous driving system, which is used to guide the smart device to drive independently or assist in driving. The processor communicates with the sensor and / or the autonomous driving system to complete the method described in any of the above embodiments. (The electronic device described in this application may be, but is not limited to, a mobile phone, a tablet computer, a desktop, a laptop, a handheld computer, a notebook computer, a vehicle-mounted device, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), an augmented reality (AR)\virtual reality (VR) device, etc., and the embodiments of this application are not limited to this.)

[0210] Thus far, the technical solution of the present application has been described in conjunction with an embodiment shown in the accompanying drawings. However, it is readily understood by those skilled in the art that the scope of protection of the present application is obviously not limited to these specific embodiments. Without departing from the principles of the present application, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present application.

Claims

1. A point cloud scene flow estimation method, It is characterized in that The method comprises: Acquire an initial source point cloud and an initial target point cloud; the initial source point cloud and the initial target point cloud are two frames of point clouds at adjacent moments; Based on the initial source point cloud and the initial target point cloud, a bidirectional point cloud reconstruction is performed to obtain a reconstructed source point cloud and a reconstructed target point cloud; the bidirectional point cloud reconstruction is to perform a point cloud reconstruction of a subsequent frame of point cloud based on a previous frame of point cloud, and to perform a point cloud reconstruction of a previous frame of point cloud based on the subsequent frame of point cloud; A point cloud scene flow prediction result is obtained according to the reconstructed source point cloud and the reconstructed target point cloud.

2. The point cloud scene flow estimation method according to claim 1, It is characterized in that The performing bidirectional point cloud reconstruction based on the initial source point cloud and the initial target point cloud to obtain a reconstructed source point cloud and a reconstructed target point cloud includes: Using the initial source point cloud and the initial target point cloud as input data for an initial iteration of the bidirectional point cloud reconstruction; Based on the input data, bidirectional point cloud reconstruction is performed for a preset number of iterations to obtain the reconstructed source point cloud and the reconstructed target point cloud.

3. The point cloud scene flow estimation method according to claim 2, It is characterized in that The step of performing bidirectional point cloud reconstruction for a preset number of iterations based on the input data to obtain the reconstructed source point cloud and the reconstructed target point cloud includes: For each iteration of the bidirectional point cloud reconstruction, obtaining a point cloud correspondence relationship between two previous and next frames of point cloud in the input data in the current iteration; Based on the point cloud correspondence, the previous frame point cloud in the input data is used as the reference point cloud, and the next frame point cloud in the input data is used as the point cloud to be reconstructed, and point cloud reconstruction is performed to obtain an intermediate target point cloud; Based on the point cloud correspondence, the point cloud of the next frame in the input data is used as the reference point cloud, and the point cloud of the previous frame in the input data is used as the point cloud to be reconstructed, and point cloud reconstruction is performed to obtain an intermediate source point cloud; Determining that the current number of iterations reaches the preset number of iterations; If yes, the intermediate source point cloud and the intermediate target point cloud are used as the reconstructed source point cloud and the reconstructed target point cloud respectively; If not, the intermediate source point cloud and the intermediate target point cloud are used as the input data for the next iteration respectively, and the next iteration of the bidirectional point cloud reconstruction is performed.

4. The point cloud scene flow estimation method according to claim 3, It is characterized in that The step of obtaining the point cloud correspondence between two frames of point clouds in the input data in the current iteration includes: Performing feature extraction on the two frames of point cloud respectively, so as to obtain relative features of all points in the single frame point cloud respectively; According to the relative features of the single-frame point cloud, the point cloud alignment features of the previous and next two frames of point cloud are obtained by global cross-attention feature extraction; The point cloud correspondence is acquired according to the point cloud alignment feature.

5. The point cloud scene flow estimation method according to claim 4, It is characterized in that The step of obtaining the point cloud correspondence relationship according to the point cloud alignment feature includes: According to the point cloud alignment feature, a point cloud similarity matrix between the two frames of point cloud is obtained; The point cloud similarity matrix is ​​used as the point cloud correspondence relationship.

6. The point cloud scene flow estimation method according to claim 3, It is characterized in that The point cloud reconstruction comprises: According to the correspondence between the reference point cloud and the point cloud, weighted averaging is performed on the reference point cloud to obtain the point cloud missing from the point cloud to be reconstructed; The obtained point cloud and the point cloud to be reconstructed are distributed and merged to achieve the point cloud reconstruction.

7. The point cloud scene flow estimation method according to any one of claims 1 to 6, It is characterized in that The obtaining a point cloud scene flow prediction result according to the reconstructed source point cloud and the reconstructed target point cloud includes: The displacement difference between the reconstructed source point cloud and the reconstructed target point cloud is obtained as the point cloud scene flow prediction result.

8. The point cloud scene flow estimation method according to claim 1, It is characterized in that The method further comprises: Based on the preset trained scene flow estimation model, bidirectional point cloud reconstruction is performed based on the initial source point cloud and the initial target point cloud, and the point cloud scene flow prediction result is obtained according to the obtained reconstructed source point cloud and the reconstructed target point cloud.

9. The point cloud scene flow estimation method according to claim 8, It is characterized in that The method further comprises training the scene flow estimation model according to the following steps: For each round of training, the initial source point cloud and the initial target point cloud used for training are input into the scene flow estimation model to obtain the scene flow prediction result of the current round of training; the initial source point cloud and the initial target point cloud used for training contain the moving speed label information of the point cloud; Obtaining a total loss of a current round of training according to the moving speed label information and the scene flow prediction result; According to the total loss, updating the model parameters of the scene flow estimation model; and determining whether a training cutoff condition is met; If the training cut-off condition is not met, the next round of training is carried out; If the training cutoff condition is reached, the training is terminated to obtain the trained scene flow estimation model.

10. The point cloud scene flow estimation method according to claim 9, It is characterized in that The obtaining the total loss of the current round of training according to the moving speed label information and the scene flow prediction result includes: According to the moving speed label information, obtaining the point restoration loss of the bidirectional point cloud reconstruction; Obtaining a scene flow loss according to the moving speed label information and the scene flow prediction result of the current round of training; The total loss is obtained according to the point restoration loss and the scene flow loss.

Images