Point cloud data completion method and device, computer device, readable storage medium and program product

Abstract

This application relates to a point cloud data completion method, apparatus, computer device, readable storage medium, and program product. The method includes: extracting features from the point cloud data to be completed at the current moment to obtain point cloud features of the point cloud data to be completed; temporally fusing the historical fused point cloud features from the previous moment with the point cloud features to obtain fused point cloud features at the current moment; identifying key points based on the fused point cloud features at the current moment to obtain key point identification results; and generating target point cloud data based on the key point identification results and the point cloud data to be completed. This method can improve the quality of point cloud completion.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a point cloud data completion method, apparatus, computer equipment, readable storage medium, and program product. Background Technology

[0002] In current industrial scenarios, robot target grasping has become a common requirement. Before grasping, point cloud data containing the target needs to be input, and six degrees of freedom estimation of the grasping points is required. The quality of the input point cloud directly determines the accuracy of the grasping pose estimation. However, due to the influence of target occlusion and sensor characteristics, the collected point cloud often contains a large number of holes. Therefore, before performing grasping pose estimation, point cloud completion processing must be performed to ensure point cloud quality and guarantee grasping accuracy.

[0003] In related technologies, point cloud completion often adopts a static inference mode, which achieves spatial missing completion through feature extraction and upsampling to meet the basic requirements of grasping pose estimation.

[0004] However, since the point cloud completion scheme is based on static reasoning, it can only simply fill in the gaps in the point cloud space and cannot achieve dynamic understanding of the point cloud in dynamic grasping scenarios. This results in insufficient quality of the point cloud after completion, which directly affects the accuracy of grasping pose estimation. Summary of the Invention

[0005] Based on this, this application addresses the aforementioned technical problems by providing a point cloud data completion method, apparatus, computer equipment, readable storage medium, and program product that can improve the quality of point cloud completion.

[0006] Firstly, this application provides a point cloud data completion method, applied to a point cloud data completion model, including:

[0007] Feature extraction is performed on the point cloud data to be completed at the current moment to obtain the point cloud features of the point cloud data to be completed.

[0008] The historical fused point cloud features from the previous moment are temporally fused with the point cloud features to obtain the fused point cloud features at the current moment.

[0009] Key point identification is performed based on the fused point cloud features at the current moment to obtain key point identification results;

[0010] Based on the key point identification results and the point cloud data to be completed, target point cloud data is generated.

[0011] The point cloud data completion method provided in this application integrates historical fused point cloud features with current point cloud features through temporal fusion, upgrading static point cloud completion to dynamic understanding. The generated target point cloud data has both spatial integrity and temporal continuity, effectively improving the quality of the completed point cloud data. This solves the problem of inaccurate grasping pose estimation caused by insufficient point cloud quality in dynamic grasping scenarios, and effectively improves the accuracy of grasping pose estimation.

[0012] In an optional embodiment of the first aspect, the step of temporally fusing the historical fused point cloud features from the previous moment with the point cloud features to obtain the fused point cloud features at the current moment includes:

[0013] The historical fused point cloud features from the previous moment are transformed into the coordinate system of the current moment to obtain the target fused point cloud features;

[0014] The target fused point cloud feature is fused with the point cloud feature in a time sequence to obtain the fused point cloud feature at the current moment.

[0015] The point cloud data completion method provided in this application transforms the historical fused point cloud features from the previous moment to the coordinate system of the current moment, and then performs temporal fusion with the point cloud features of the current moment. This integrates historical temporal information with current spatial information, breaking the limitations of static features at a single moment. The generated fused point cloud features at the current moment have both temporal continuity and spatial integrity, effectively solving the problem of insufficient feature fusion accuracy caused by missing temporal information and coordinate system deviation in dynamic capture scenarios. This improves the quality of the fused point cloud features. Performing subsequent point cloud completion processing based on these fused point cloud features can effectively improve the quality of point cloud completion.

[0016] In an optional embodiment of the first aspect, transforming the historical fused point cloud features from the previous moment to the coordinate system of the current moment to obtain the target fused point cloud features includes:

[0017] The motion data from the previous moment to the current moment is obtained from the motion sensing module, and the motion transformation matrix from the previous moment to the current moment is determined based on the motion data;

[0018] The motion transformation matrix is ​​used to transform the historical fused point cloud features of the previous moment to obtain the target fused point cloud features.

[0019] The point cloud data completion method provided in this application determines the motion transformation matrix by acquiring motion data, and performs coordinate system transformation on the historical fused point cloud features of the previous moment to accurately align the historical time-series features with the coordinate system of the current moment. This effectively solves the problem of feature misalignment and fusion distortion caused by coordinate system deviation in dynamic grasping scenarios, thereby improving the accuracy of the fused point cloud features at the current moment and further ensuring the accuracy of grasping pose estimation.

[0020] In an optional embodiment of the first aspect, the step of temporally fusing the target fused point cloud feature with the point cloud feature to obtain the fused point cloud feature at the current moment includes:

[0021] The target fused point cloud feature is concatenated with the point cloud feature to obtain the concatenated feature;

[0022] Attention feature fusion is performed on the stitched features and the target fused point cloud features to obtain the fused point cloud features at the current time.

[0023] The point cloud data completion method provided in this application deeply integrates historical time-series information and current spatial information, and highlights key features through an attention mechanism. It can accurately reflect the motion change pattern and spatial structure features of the target object in a dynamic grasping scenario, effectively solving the problems of insufficient point cloud completion quality and inaccurate grasping pose estimation caused by redundant feature information and lack of prominence of key features in a dynamic grasping scenario, and further improving the accuracy and reliability of robot grasping.

[0024] In an optional embodiment of the first aspect, the step of performing attention feature fusion on the stitched features and the target fused point cloud features to obtain the fused point cloud features at the current moment includes:

[0025] Using the spliced ​​features as the query vector and the target fused point cloud features as the value vector and key vector, intermediate features are generated;

[0026] Using the intermediate features as the query vector and the concatenated features as the value vector and key vector, the fused point cloud features at the current moment are generated.

[0027] The point cloud data completion method provided in this application uses a two-round progressive attention feature fusion approach. First, it guides the selection of historical temporal features with spliced ​​features, and then guides the strengthening of current spatial features with optimized temporal features (intermediate features). This achieves deep interaction and adaptive weighting between historical temporal information and current spatial information, further improving the expression accuracy and robustness of the fused features. It effectively improves the feature fusion accuracy in dynamic capture scenarios, thereby improving the quality of fused point cloud features. Point cloud completion based on these fused point cloud features can effectively improve the quality of point cloud completion.

[0028] In an optional embodiment of the first aspect, the training process of the point cloud data completion model includes:

[0029] Input the sample point cloud data into the initial completion model, and output the predicted point cloud data.

[0030] For a target point, the loss weight of the target point is determined based on the distance between the target point and each ground truth key point and the three-dimensional Gaussian distribution corresponding to each ground truth key point. The target point is any point in the ground truth point cloud data corresponding to the predicted point cloud data or the sample point cloud data.

[0031] Based on the predicted point cloud data, the loss weights corresponding to each point in the predicted point cloud data, the ground truth point cloud data, and the loss weights corresponding to each point in the ground truth point cloud data, the weighted chamfer loss corresponding to the initial completion model is determined.

[0032] The parameters of the initial completion model are adjusted based on the weighted chamfer loss.

[0033] When the adjusted initial completion model meets the training stopping condition, the point cloud data completion model is obtained.

[0034] The point cloud data completion method provided in this application constructs a weighted chamfer loss based on ground truth key points and a 3D Gaussian distribution, and applies differentiated constraints to the predicted point cloud data and ground truth point cloud data. This enhances the completion accuracy of key point regions, enables efficient training of the initial completion model, and results in a more accurate and robust point cloud data completion model. This effectively solves the problems of insufficient constraints on key regions and low completion quality in traditional point cloud completion models, thereby improving the reliability of point cloud completion in dynamic grasping scenarios and effectively enhancing the accuracy of grasping pose estimation.

[0035] In an optional embodiment of the first aspect, when the distance between the target point and the truth key point is greater than or equal to the upper limit of the distance, the loss weight of the target point remains unchanged with a first loss weight, the first loss weight being a loss weight determined based on the upper limit of the distance.

[0036] The point cloud data completion method provided in this application can accurately reflect the difference in the influence of ground truth key points on target points, making loss calculation more targeted, thereby providing a more reliable loss basis for model training and improving the accuracy of the model.

[0037] Secondly, this application also provides a point cloud data completion device, applied to a point cloud data completion model, comprising:

[0038] The extraction module is used to extract features from the point cloud data to be completed at the current time to obtain the point cloud features of the point cloud data to be completed.

[0039] The fusion module is used to perform temporal fusion of the historical fused point cloud features from the previous moment with the point cloud features to obtain the fused point cloud features at the current moment.

[0040] The identification module is used to identify key points based on the fused point cloud features at the current moment, and obtain key point identification results;

[0041] The completion module is used to generate target point cloud data based on the key point identification results and the point cloud data to be completed.

[0042] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0043] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0044] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any of the above aspects.

[0045] Regarding the beneficial effects of any of the technical solutions in the second to fifth aspects mentioned above, refer to the beneficial effects of the corresponding technical solutions in the first aspect; repeated examples will not be listed here. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a schematic diagram of an optional process for a point cloud data completion method in one embodiment;

[0048] Figure 2 This is a schematic diagram of an optional process for step 102 in one embodiment;

[0049] Figure 3 This is a schematic diagram of an optional process for step 201 in one embodiment;

[0050] Figure 4 This is a schematic diagram of an optional process for step 202 in one embodiment;

[0051] Figure 5 This is a schematic diagram of an optional process for step 402 in one embodiment;

[0052] Figure 6 This is a schematic diagram of an optional training process for a point cloud data completion model in one embodiment;

[0053] Figure 7 This is a schematic diagram of an optional structure of a point cloud data completion device in one embodiment;

[0054] Figure 8 This is a schematic diagram of an optional internal structure of a computer device in one embodiment. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application.

[0056] The terms "first," "second," etc., used in this application may be used to describe various elements, but these elements are not limited by these terms. These terms are used only to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0057] In one embodiment, such as Figure 1 As shown, a point cloud data completion method is provided. This embodiment uses the application of this method to a point cloud data completion module deployed on a terminal as an example for illustration. It can be understood that this method can also be applied to a point cloud data completion module deployed on a server. In this embodiment, the method includes the following steps 101 to 104, wherein:

[0058] Step 101: Extract features from the point cloud data to be completed at the current time to obtain the point cloud features of the point cloud data to be completed.

[0059] In this embodiment, the point cloud data to be completed can be the point cloud data of the target object obtained by reconstructing image data acquired at the current moment through image acquisition devices (such as RGB-D (Red, Green, Blue, Depth) cameras, multi-view cameras, etc.), or it can be the point cloud data of the target object acquired through devices such as LiDAR. Due to factors such as target object occlusion and sensor acquisition characteristics, the point cloud data to be completed has problems such as spatial voids and data sparsity, and cannot directly meet the accuracy requirements of robot grasping pose estimation. Therefore, it belongs to the original point cloud data that needs to be completed. Here, "current moment" refers to the time node when the robot acquires the point cloud data to be completed.

[0060] For example, a transformer encoder or a deep learning-based point cloud feature extraction network (such as PointNet, PointCNN, DGCNN, etc.) can be used to extract features from the point cloud data to be completed at the current time, obtaining the point cloud features of the point cloud data to be completed. The point cloud features are information used to characterize the geometric attributes and spatial distribution patterns of the point cloud data to be completed. This application does not limit the specific type of point cloud features. For example, point cloud features may include the normal vector, curvature, neighborhood point density, three-dimensional coordinate distribution characteristics, and local feature descriptors of the local point cloud, or the centroid, size, and shape features of the point cloud at the global level.

[0061] Step 102: Perform temporal fusion of the historical fused point cloud features from the previous moment with the point cloud features to obtain the fused point cloud features at the current moment.

[0062] In this embodiment, the historical fused point cloud features from the previous moment are the fused features obtained by the robot after performing feature fusion on the point cloud data to be completed at the previous time node of the current moment. These historical fused point cloud features are stored in the robot's local storage module, readily available, and have already incorporated the temporal information of multiple frames of point cloud data prior to the current moment. The point cloud features at the current moment only represent the instantaneous features of the point cloud data to be completed at the current moment. Using them alone is susceptible to the influence of missing points in the current frame and noise, resulting in incomplete feature representation. Therefore, the historical fused point cloud features from the previous moment can be temporally fused with the point cloud features to obtain the fused point cloud features at the current moment.

[0063] In other words, the fused point cloud features obtained through temporal fusion operations at the current moment retain the effective information of the point cloud data acquired in previous moments, while also including the latest state of the point cloud data at the current moment, effectively improving the completeness and robustness of the features. This application does not limit the specific method of temporal fusion in its embodiments. For example, temporal fusion can employ weighted fusion, temporal attention fusion, recurrent neural network fusion, etc.

[0064] Step 103: Based on the fused point cloud features at the current moment, perform key point identification to obtain the key point identification results.

[0065] In this embodiment, key points refer to the feature points of the target object that play a key role in estimating the robot's grasping pose, such as the grasping point, corner points, and geometric center of the target object. In this embodiment, the type and number of key points are not specifically limited, but can be set according to the shape of the target object and the grasping task requirements.

[0066] Keypoint recognition is the process of reconstructing the coordinates of specific keypoints from the fused point cloud features that incorporate temporal information. For example, the fused point cloud features at the current moment can be input into a Multilayer Perceptron (MLP), which consists of an input layer, multiple hidden layers, and an output layer. Through layer-by-layer linear transformations and nonlinear activations, the MLP regresses and outputs the three-dimensional spatial coordinates (x, y, z) of the keypoints. The keypoint recognition result is a set containing the three-dimensional coordinate information of all identified keypoints.

[0067] Step 104: Generate target point cloud data based on the key point identification results and the point cloud data to be completed.

[0068] In this embodiment, based on the spatial location of key points, the empty areas in the point cloud data to be completed can be accurately filled to obtain the completed target point cloud data. For example, the key point recognition results can be used as anchor points, and a multi-layer upsampling Transformer can be used to upsample the point cloud data to be completed. The upsampling Transformer can gradually fill the spatial holes in the point cloud data to be completed, using key points as the core, restoring the complete three-dimensional structure of the target object, and finally outputting the completed target point cloud data.

[0069] It is understood that the above-described upsampling process of the point cloud data to be completed using a multi-layer upsampling Transformer is only an example in this application embodiment. In fact, this application embodiment does not specifically limit the algorithm for point cloud completion, and any method of point cloud completion based on key point guidance is applicable to this application embodiment.

[0070] The point cloud data completion method provided in this application integrates historical fused point cloud features with current point cloud features through temporal fusion, upgrading static point cloud completion to dynamic understanding. The generated target point cloud data has both spatial integrity and temporal continuity, effectively improving the quality of the completed point cloud data. This solves the problem of inaccurate grasping pose estimation caused by insufficient point cloud quality in dynamic grasping scenarios, and effectively improves the accuracy of grasping pose estimation.

[0071] In one exemplary embodiment, reference is made to Figure 2 As shown, in step 102, the historical fused point cloud features from the previous moment are temporally fused with the current point cloud features to obtain the fused point cloud features at the current moment. This may include the following steps 201 to 202, wherein:

[0072] Step 201: Transform the historical fused point cloud features from the previous moment to the coordinate system of the current moment to obtain the target fused point cloud features;

[0073] Step 202: Perform temporal fusion of the target fused point cloud features and the point cloud features to obtain the fused point cloud features at the current moment.

[0074] In this embodiment, the historical fused point cloud features from the previous moment are the fused point cloud features obtained by the robot after completing temporal fusion at the previous time node. These historical fused point cloud features can be stored in the robot's local storage module and retrieved as needed. Their coordinate system corresponds to the relative spatial position of the robot and the target object at the previous moment. Since the robot may move or the target object may change position in a dynamic grasping scenario, there will be a deviation between the coordinate system of the previous moment and the coordinate system of the current moment. If the historical fused point cloud features are directly fused with the point cloud features of the current moment, feature misalignment and fusion distortion will occur due to the inconsistency of coordinate systems, thus affecting the accuracy of subsequent key point recognition and point cloud completion. Therefore, it is necessary to unify the coordinate systems of the two before performing feature fusion, that is, to transform the historical fused point cloud features from the previous moment's coordinate system to the current moment's coordinate system to obtain the target fused point cloud features.

[0075] In one exemplary embodiment, reference is made to Figure 3 As shown, in step 201, the historical fused point cloud features from the previous moment are transformed to the coordinate system of the current moment to obtain the target fused point cloud features. This may include the following steps 301 to 302, wherein:

[0076] Step 301: Obtain motion data from the previous moment to the current moment from the motion sensing module, and determine the motion transformation matrix from the previous moment to the current moment based on the motion data;

[0077] Step 302: The historical fused point cloud features of the previous moment are transformed using a motion transformation matrix to obtain the target fused point cloud features.

[0078] In this embodiment, the motion sensing module is a hardware module mounted on the robot for collecting motion information of itself and the target object. It can integrate sensors such as an IMU (Inertial Measurement Unit) and an odometer, and can collect and store the robot's motion state data at different times in real time. The motion data of the robot from the previous moment to the current moment can be obtained from the motion sensing module, including data representing changes in robot posture and position offset between two time points, including but not limited to parameters such as robot translation, rotation angle, angular velocity, and linear velocity.

[0079] Based on the acquired motion data, the motion transformation matrix can be determined. For example, kinematic algorithms can be used to convert the collected translational, rotational, and other motion parameters into a matrix form capable of coordinate system transformation, thus obtaining the motion transformation moments from the previous moment to the current moment. For example, the motion transformation matrix can be a homogeneous transformation matrix, comprising a rotation matrix and a translation vector. The rotation matrix represents the attitude change of the coordinate system, and the translation vector represents the positional shift of the coordinate system.

[0080] In one example, the motion data collected by the motion sensing module can be denoised using a Kalman filter algorithm to filter out errors caused by data noise. Then, based on the rigid body kinematics formula, the denoised translation and rotation angles are substituted into the calculation to obtain the motion transformation matrix from the previous moment to the current moment. This application does not limit the specific type of motion data or the calculation algorithm of the motion transformation matrix; any method that can accurately reflect the coordinate system offset and realize matrix calculation is applicable to this application.

[0081] Since the motion transformation matrix can represent the coordinate system deviation between the previous moment and the current moment, the historical fused point cloud features can be transformed to the coordinate system of the current moment by transforming the historical fused point cloud features through the motion transformation matrix.

[0082] For example, the historical fused point cloud features (existing in the form of feature vectors) from the previous moment can be multiplied by the motion transformation matrix. The attitude deviation of the historical fused point cloud features can be corrected by the rotation matrix, and the position deviation of the historical fused point cloud features can be corrected by the translation vector. The result of the operation is the target fused point cloud feature. The target fused point cloud feature retains all the temporal information and target object features of the historical fused point cloud features, and is fully adapted to the coordinate system at the current moment, eliminating the feature misalignment and fusion distortion problems caused by coordinate system deviation.

[0083] The point cloud data completion method provided in this application determines the motion transformation matrix by acquiring motion data, and performs coordinate system transformation on the historical fused point cloud features of the previous moment to accurately align the historical time-series features with the coordinate system of the current moment. This effectively solves the problem of feature misalignment and fusion distortion caused by coordinate system deviation in dynamic grasping scenarios, thereby improving the accuracy of the fused point cloud features at the current moment and further ensuring the accuracy of grasping pose estimation.

[0084] After obtaining the target fused point cloud features, a temporal fusion can be performed between the target fused point cloud features and the point cloud features at the current moment to obtain the fused point cloud features at the current moment, which are then stored. The temporal fusion operation integrates the effective information from the target fused point cloud features and the point cloud features at the current moment, filters redundant noise, and generates fused point cloud features that can completely represent the current state of the target object. This application does not limit the specific method of temporal fusion in its embodiments; any method that can effectively integrate the features of both is applicable to this application.

[0085] For example, temporal fusion can adopt a weighted fusion method: based on the confidence levels of the historical fused point cloud features from the previous moment and the current moment's point cloud features, different fusion weights are assigned to the two (e.g., if the current frame's point cloud quality is high, a higher weight is assigned; if the historical frame features are more stable, the weight of the historical features is appropriately increased), and the fused point cloud features at the current moment are obtained by weighted summation. Alternatively, a temporal attention fusion method can be used, which automatically filters out the effective information in the target fused point cloud features and the current point cloud features through an attention mechanism, focusing on fusing the feature parts that are more critical to the 3D structural representation of the target object, and suppressing the influence of noise and invalid information.

[0086] The point cloud data completion method provided in this application transforms the historical fused point cloud features from the previous moment to the coordinate system of the current moment, and then performs temporal fusion with the point cloud features of the current moment. This integrates historical temporal information with current spatial information, breaking the limitations of static features at a single moment. The generated fused point cloud features at the current moment have both temporal continuity and spatial integrity, effectively solving the problem of insufficient feature fusion accuracy caused by missing temporal information and coordinate system deviation in dynamic capture scenarios. This improves the quality of the fused point cloud features. Performing subsequent point cloud completion processing based on these fused point cloud features can effectively improve the quality of point cloud completion.

[0087] In one exemplary embodiment, reference is made to Figure 4 As shown, in step 202, the target fused point cloud features and the point cloud features are temporally fused to obtain the fused point cloud features at the current time. This may include the following steps 401 to 402, wherein:

[0088] Step 401: The target fused point cloud features are stitched together with the point cloud features to obtain the stitched features;

[0089] Step 402: Attention feature fusion is performed on the spliced ​​features and the target fused point cloud features to obtain the fused point cloud features at the current time.

[0090] In this embodiment, after obtaining the target fused point cloud features, the target fused point cloud features can be concatenated with the point cloud features to obtain concatenated features. For example, a feature dimension concatenation method can be used, concatenating the target fused point cloud features (assuming a dimension of C1×N) with the point cloud features at the current time (assuming a dimension of C2×N) to obtain a concatenated feature with a dimension of (C1+C2)×N, where C represents the number of feature channels and N represents the number of feature points. The concatenated feature simultaneously contains historical temporal information and current spatial information, preserving both the motion change features of the target object at the previous time and covering the spatial geometric details of the target object at the current time.

[0091] This application does not limit the specific method of feature stitching. Any method that can effectively stitch together the target fusion point cloud features with the current point cloud features without losing feature information is applicable to this application.

[0092] Although the spliced ​​features initially integrate historical time-series information and current spatial information, they suffer from feature redundancy and unclear weights for key information. If directly used for subsequent keypoint recognition, information interference can lead to a decrease in keypoint recognition accuracy. Therefore, in this embodiment, attention feature fusion is used to further optimize and filter the spliced ​​features, thereby improving the representational capability of the fused features.

[0093] For example, an attention mechanism can be used to achieve attention feature fusion. This mechanism can adaptively assign different weights to features in different regions of the stitched features, focusing on core features related to key points and motion changes of the target object, while weakening the influence of redundant background features and invalid information. The resulting fused point cloud features at the current moment deeply integrate historical temporal information and current spatial information, and highlight key features through the attention mechanism. This accurately reflects the motion change patterns and spatial structure features of the target object in dynamic grasping scenarios, effectively solving the problems of insufficient point cloud completion quality and inaccurate grasping pose estimation caused by redundant feature information and lack of emphasis on key features in dynamic grasping scenarios, further improving the accuracy and reliability of robot grasping.

[0094] In one exemplary embodiment, reference is made to Figure 5 As shown, in step 402, attention feature fusion is performed on the spliced ​​features and the target fused point cloud features to obtain the fused point cloud features at the current time. This may include the following steps 501 to 502, wherein:

[0095] Step 501: Using the spliced ​​features as the query vector and the target fused point cloud features as the value vector and key vector, generate intermediate features;

[0096] Step 502: Using the intermediate features as the query vector and the concatenated features as the value vector and key vector, generate the fused point cloud features for the current time step.

[0097] In dynamic grasping scenarios, if the target object itself is moving (e.g., grasping a moving object), simple coordinate system alignment will still result in motion blur. This application incorporates a deformable attention mechanism, which adaptively focuses on useful feature regions instead of rigidly aligning all regions. The deformable attention mechanism includes two attention processes: in the first attention fusion process, the concatenated features are used as the query vector, and the target fused point cloud features are used as the value vector and key vector to calculate intermediate features; in the second attention fusion process, the intermediate features are used as the query vector, and the concatenated features are used as the value vector and key vector to calculate the final temporally fused point cloud features.

[0098] In the attention processing, the query vector, key vector, and value vector are key inputs for feature matching and weighted fusion in the attention mechanism. In this embodiment, the concatenated feature is obtained by concatenating the target fused point cloud feature with the current point cloud feature, containing both historical temporal information and current spatial structure information. Using this as the query vector, it can be guided by the current comprehensive feature to achieve adaptive retrieval and matching of historical features. The target fused point cloud feature is a historical temporal feature aligned to the coordinate system; its information is stable and spatially aligned with the current moment. Using it as both the key and value vectors provides reliable historical feature support for the attention mechanism. By using the concatenated feature as the query vector and the target fused point cloud feature as both the key and value vectors for attention calculation, the initial weighted fusion of the current and historical features is completed, resulting in an intermediate feature. This intermediate feature filters and strengthens the historical temporal information, effectively improving the effectiveness and relevance of the temporal features.

[0099] In the second round of attention fusion, since the intermediate features are features resulting from the first round of attention fusion and already possess precise temporal guidance capabilities, using them as query vectors can further focus on the key structural and motion-related features of the target object. The concatenated features contain complete spatial information of the point cloud at the current moment; using them simultaneously as both key and value vectors provides comprehensive and detailed current spatial feature information for the second round of attention fusion. By using the intermediate features as query vectors and the concatenated features as key and value vectors for the second round of attention calculation, deep interaction and weighted fusion of temporal guidance features and current complete features are achieved, generating the fused point cloud features for the current moment. This fused point cloud feature, after two rounds of progressive attention fusion processing, exhibits a tighter integration of temporal and spatial information, more prominent key features, and stronger representational capabilities.

[0100] The point cloud data completion method provided in this application uses a two-round progressive attention feature fusion approach. First, it guides the selection of historical temporal features with spliced ​​features, and then guides the strengthening of current spatial features with optimized temporal features (intermediate features). This achieves deep interaction and adaptive weighting between historical temporal information and current spatial information, further improving the expression accuracy and robustness of the fused features. It effectively improves the feature fusion accuracy in dynamic capture scenarios, thereby improving the quality of fused point cloud features. Point cloud completion based on these fused point cloud features can effectively improve the quality of point cloud completion.

[0101] In one exemplary embodiment, reference is made to Figure 6 As shown, the training process of the point cloud data completion model may include the following steps 601 to 605, wherein:

[0102] Step 601: Input the sample point cloud data into the initial completion model and output the predicted point cloud data.

[0103] Step 602: For the target point, determine the loss weight of the target point based on the distance between the target point and each ground truth key point and the three-dimensional Gaussian distribution corresponding to each ground truth key point. The target point is any point in the ground truth point cloud data corresponding to the predicted point cloud data or the sample point cloud data.

[0104] Step 603: Based on the predicted point cloud data, the loss weights corresponding to each point in the predicted point cloud data, the ground truth point cloud data, and the loss weights corresponding to each point in the ground truth point cloud data, determine the weighted chamfer loss corresponding to the initial completion model.

[0105] Step 604: Adjust the parameters of the initial completion model based on the weighted chamfer loss;

[0106] Step 605: When the adjusted initial completion model meets the training stopping condition, the point cloud data completion model is obtained.

[0107] In this embodiment, the sample point cloud data is training data with real-world annotations. This sample point cloud data can contain defects such as missing data, occlusion, or sparsity, and is used to simulate point cloud data to be completed in real-world application scenarios. The initial completion model is an untrained neural network model with its parameters initialized, possessing the basic network structure for point cloud feature extraction, temporal fusion, key point recognition, and point cloud completion. The sample point cloud data is input into the initial completion model, and through a series of processes including feature extraction, temporal fusion, and key point-guided completion, the model outputs predicted point cloud data corresponding to the sample point cloud data. This predicted point cloud data is the result obtained by the initial completion model after completing the sample point cloud data. This embodiment does not limit the specific network structure of the initial completion model.

[0108] The ground truth point cloud data is the complete and unmissing standard point cloud data corresponding to the sample point cloud data. The ground truth keypoints are the pre-annotated 3D points in the ground truth point cloud data that play a key role in estimating the pose of the target object. Each ground truth keypoint corresponds to an independent 3D Gaussian distribution, and the coordinates of the ground truth keypoint are the mean vector of this 3D Gaussian distribution. The 3D Gaussian distribution uses a spherically symmetric covariance matrix.

[0109] The target point can be any point in the predicted point cloud data or any point in the ground truth point cloud data. For each target point, the distance from the target point to each ground truth keypoint is calculated, and the loss weight corresponding to the target point is determined by combining the distance from the target point to each ground truth keypoint with the corresponding 3D Gaussian distribution. The closer the target point is to the ground truth keypoint, the greater the loss weight; the farther the target point is from the ground truth keypoint, the smaller the loss weight. After determining the loss weight of the target point relative to each ground truth keypoint, the final loss weight corresponding to the target point can be determined based on these loss weights. For example, the maximum value among the loss weights can be used as the loss weight of the target point, or the minimum value among the loss weights can be used as the loss weight of the target point, or the mean of the loss weights can be used as the loss weight of the target point. In this way, the loss weights of each point in the predicted point cloud data and the loss weights of each point in the ground truth point cloud data can be calculated.

[0110] The loss weights can be used to calculate the weighted chamfer loss of the initial completion model. The weighted chamfer loss is a loss function used to measure the degree of difference between the predicted point cloud data and the ground truth point cloud data. It introduces keypoint-guided loss weights on top of the traditional chamfer distance, achieving multi-scale weighted constraints. For example, the minimum distance from each point in the predicted point cloud data to the ground truth point cloud data can be calculated separately and multiplied by the corresponding point's loss weight to obtain the first weighted distance sum; the minimum distance from each point in the ground truth point cloud data to the predicted point cloud data can be calculated and multiplied by the corresponding point's loss weight to obtain the second weighted distance sum; the first weighted distance sum and the second weighted distance sum are then added to obtain the weighted chamfer loss corresponding to the initial completion model. This weighted chamfer loss can focus on constraining the completion accuracy near keypoints while also considering the overall point cloud matching degree, improving the reliability and practicality of the model completion.

[0111] In an exemplary embodiment, when the distance between the target point and the truth key point is greater than or equal to the upper limit of the distance, the loss weight of the target point remains unchanged with the first loss weight, which is a loss weight determined based on the upper limit of the distance.

[0112] In this embodiment, for each target point, the corresponding loss weight is calculated by combining all ground truth keypoints. That is, a target point will correspond to multiple loss weights that are one-to-one with ground truth keypoints. The calculation of each loss weight is constrained by the distance between the target point and the corresponding ground truth keypoint. A preset upper limit for distance is used to define the effective influence range of the ground truth keypoint. When the distance between the target point and a ground truth keypoint is less than this upper limit, the loss weight of the target point corresponding to this ground truth keypoint will be dynamically adjusted according to the actual distance between them. That is, the closer the distance, the greater the weight, thereby strengthening the constraint accuracy of the area near the keypoint. When the distance between the target point and the ground truth keypoint is greater than or equal to the upper limit, it indicates that the target point has exceeded the effective influence range of this ground truth keypoint. At this time, the loss weight of the target point corresponding to this ground truth keypoint will remain unchanged at the first loss weight and will no longer change with the increase of distance.

[0113] The first loss weight is a fixed weight calculated based on a preset distance upper limit. Its value is strongly correlated with the distance upper limit, ensuring the scientific and reasonable nature of the weight setting while minimizing the problem of abnormal weight fluctuations due to excessive distance. This weight setting method can accurately reflect the difference in influence between ground truth key points and target points, making loss calculation more targeted, thus providing a more reliable loss basis for model training and improving model accuracy.

[0114] In an exemplary embodiment, the initial completion model employs a multi-layer upsampling structure to adapt to the needs of multi-scale point cloud completion. For ground truth point cloud data, downsampling processing can be performed according to the network scale of each layer of the initial completion model to obtain ground truth point cloud data at each scale consistent with the output scale of each layer of the model, ensuring that the prediction results of each layer have corresponding standard references. During the process of the initial completion model predicting and outputting predicted point cloud data, the predicted point cloud data at each scale is output simultaneously, realizing multi-scale synchronous prediction and supervision. The calculation process of the weighted chamfer loss at each scale can be performed with reference to the calculation logic of the weighted chamfer loss in the aforementioned embodiment, and will not be repeated here in this embodiment. Subsequently, the weighted chamfer losses calculated at each scale are weighted and fused, and the supervision information at each scale is integrated to finally obtain the weighted chamfer loss corresponding to the initial completion model.

[0115] After obtaining the weighted chamfer loss corresponding to the initial completion model, the parameters of the initial completion model can be updated and adjusted using the backpropagation algorithm. By calculating the gradient of the weighted chamfer loss with respect to each network parameter in the initial completion model, and according to the preset optimizer and learning rate, the difference between the predicted point cloud data and the ground truth point cloud data is gradually reduced, so that the predicted point cloud data output by the initial completion model continuously approaches the ground truth point cloud data, thereby improving the model's point cloud completion capability. In this embodiment, no specific type of optimizer or value of the learning rate is limited.

[0116] In this embodiment, the training stopping condition can be that the weighted chamfer loss converges to a preset range, the number of model training iterations reaches a preset number, or the model's accuracy on the validation set meets a preset requirement. This embodiment does not limit the specific form of the training stopping condition. When the adjusted initial completion model meets the training stopping condition, it indicates that the model has stable and reliable point cloud completion performance. At this point, training is stopped, and the initial completion model that meets the training stopping condition is determined as the point cloud data completion model.

[0117] The point cloud data completion method provided in this application constructs a weighted chamfer loss based on ground truth key points and a 3D Gaussian distribution, and applies differentiated constraints to the predicted point cloud data and ground truth point cloud data. This enhances the completion accuracy of key point regions, enables efficient training of the initial completion model, and results in a more accurate and robust point cloud data completion model. This effectively solves the problems of insufficient constraints on key regions and low completion quality in traditional point cloud completion models, thereby improving the reliability of point cloud completion in dynamic grasping scenarios and effectively enhancing the accuracy of grasping pose estimation.

[0118] To enable those skilled in the art to better understand the embodiments of this application, the embodiments of this application are described below through specific examples. In the embodiments of this application, the point cloud data completion model includes a transformer encoder, a timing processing module, a multilayer perceptron, and a multilayer upsampling module.

[0119] First, the point cloud data to be completed is processed by a transformer encoder to extract global features, obtaining the encoded features ft (i.e., point cloud features) at the current time t. Then, a temporal processing module fuses the features from before time t with ft, resulting in the temporally fused feature Ft_update. Next, Ft_update is processed by a multilayer perceptron to regress keypoints. Finally, based on the regressed keypoints, a multilayer upsampling transformer is used to upsample and complete the point cloud data, yielding the completed target point cloud data.

[0120] The temporal processing module fuses the feature ft at time t with the temporal fusion feature Ft-1_update before time t to obtain the new fused feature Ft_update at time t. For example, firstly, based on the robot's IMU and odometry sensors, the motion transformation matrix Tmatrix from time t-1 to time t is calculated. Ft-1_update is then motion-aligned using Tmatrix to obtain the aligned feature Ft-1_update~. Then, Ft-1_update~ and the current ft are concatenated to obtain ft-1,t. To address the motion ambiguity that motion alignment might cause for moving targets, this embodiment adds a deformable attention module to the temporal processing module. In this module, ft-1,t is used as the query, and Ft-1_update~ is used as the value and key. The output is Ft. Then, Ft is used as the query, and ft-1,t as the value and key to obtain the fused feature Ft_update at the current time t.

[0121] During training, both the regressed keypoints and the imputed point cloud are supervised. L2 loss is used for keypoint supervision. For calculating the loss of the imputed point cloud, this embodiment proposes a multi-scale weighted L2 chamfer distance loss calculation. First, a three-dimensional Gaussian distribution is calculated based on the keypoints. The closer to the keypoint, the larger the weight; the farther away from the keypoint, the smaller the weight. When the distance from the keypoint exceeds a certain threshold, the weight remains constant. The mean vector μ of the three-dimensional Gaussian distribution is [μx, μy, μz]. μx, μy, and μz are the x, y, and z coordinates of the keypoint, respectively. The covariance matrix Σ of the three-dimensional Gaussian distribution adopts a spherically symmetric distribution and is expressed as:

[0122]

[0123] in, These are empirical values. When calculating the L2 chamfer distance, each point can be weighted. The closer a point is to a critical area, the greater its weight, thus the model focuses more on the completion effect of critical areas. When the distance to a critical point exceeds a certain threshold, the weights no longer change to ensure that distant point clouds also receive sufficient attention.

[0124] To ensure the accuracy of point cloud completion during upsampling, multi-scale supervision is employed. For example, there are five upsampling scales; the outputs of the four scales other than the first can be supervised. When calculating the loss at each scale, the ground truth point cloud data is downsampled based on the current scale. During inference, the input is the hollow point cloud, which yields the completed point cloud and the predicted keypoints.

[0125] The temporal fusion strategy proposed in the point cloud data completion method provided in this application can complement information from multiple frames, ensuring physical continuity under spatiotemporal processing. The utilization of temporal information elevates point cloud completion from static reasoning to dynamic understanding, not only completing spatial gaps but also temporal continuity, providing a more powerful tool for 3D perception in dynamic scenes. Simultaneously, Ft-1_update can be stored locally for on-demand access. This maximizes the utilization of temporal information without significantly increasing computational load. Through keypoint-guided 3D Gaussian weighted loss, the model precisely focuses its attention on key regions of interest (such as handles and contact points) during training, ensuring the highest completion accuracy in these regions, thereby directly improving the reliability and success rate of downstream tasks (such as robot grasping). Multi-scale supervision, by applying constraints at multiple upsampling levels, ensures progressive and accurate generation from the overall contour to local details, minimizing the possibility of blurred distant regions or structural errors caused by only supervising the final output. The threshold saturation mechanism in the weighted loss (weights remain constant after the distance exceeds a threshold) prevents the model from completely ignoring non-critical regions, ensuring the global integrity and rationality of point cloud completion. The predicted keypoints jointly output by the network are themselves a strong semantic understanding of object structure (such as axes of symmetry and centers of functional components), which can be directly used for pose calculation, object alignment, etc., enhancing the interpretability and practicality of the system. The Transformer-based encoder-decoder architecture excels at handling irregular point clouds and modeling long-range dependencies. Combined with a coarse-to-fine generation strategy, the solution has good generalization ability for different degrees of occlusion and different object categories.

[0126] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0127] Based on the same inventive concept, this application also provides a point cloud data completion device for implementing the point cloud data completion method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more point cloud data completion device embodiments provided below can be found in the limitations of the point cloud data completion method described above, and will not be repeated here.

[0128] In one exemplary embodiment, such as Figure 7 As shown, a point cloud data completion device 700 is provided, applied to a point cloud data completion model. The device includes: an extraction module 701, a fusion module 702, a recognition module 703, and a completion module 704, wherein:

[0129] The extraction module 701 is used to extract features from the point cloud data to be completed at the current time to obtain the point cloud features of the point cloud data to be completed.

[0130] The fusion module 702 is used to perform temporal fusion of the historical fused point cloud features from the previous moment with the point cloud features to obtain the fused point cloud features at the current moment.

[0131] The recognition module 703 is used to identify key points based on the fused point cloud features at the current moment and obtain the key point recognition results.

[0132] The completion module 704 is used to generate target point cloud data based on the key point identification results and the point cloud data to be completed.

[0133] The point cloud data completion device provided in this application integrates historical fused point cloud features with current point cloud features through temporal fusion, upgrading static point cloud completion to dynamic understanding. The generated target point cloud data has both spatial integrity and temporal continuity, effectively improving the quality of the completed point cloud data. This solves the problem of inaccurate grasping pose estimation caused by insufficient point cloud quality in dynamic grasping scenarios, and effectively improves the accuracy of grasping pose estimation.

[0134] In an exemplary embodiment, the step of temporally fusing the historical fused point cloud features from the previous moment with the current point cloud features to obtain the fused point cloud features at the current moment includes:

[0135] The historical fused point cloud features from the previous moment are transformed into the coordinate system of the current moment to obtain the target fused point cloud features;

[0136] The target fused point cloud feature is fused with the point cloud feature in a time sequence to obtain the fused point cloud feature at the current moment.

[0137] In an exemplary embodiment, transforming the historical fused point cloud features from the previous moment to the coordinate system of the current moment to obtain the target fused point cloud features includes:

[0138] The motion data from the previous moment to the current moment is obtained from the motion sensing module, and the motion transformation matrix from the previous moment to the current moment is determined based on the motion data;

[0139] The motion transformation matrix is ​​used to transform the historical fused point cloud features of the previous moment to obtain the target fused point cloud features.

[0140] In an exemplary embodiment, the step of temporally fusing the target fused point cloud feature with the point cloud feature to obtain the fused point cloud feature at the current moment includes:

[0141] The target fused point cloud feature is concatenated with the point cloud feature to obtain the concatenated feature;

[0142] Attention feature fusion is performed on the stitched features and the target fused point cloud features to obtain the fused point cloud features at the current time.

[0143] In an exemplary embodiment, the step of performing attention feature fusion on the stitched features and the target fused point cloud features to obtain the fused point cloud features at the current moment includes:

[0144] Using the spliced ​​features as the query vector and the target fused point cloud features as the value vector and key vector, intermediate features are generated;

[0145] Using the intermediate features as the query vector and the concatenated features as the value vector and key vector, the fused point cloud features at the current moment are generated.

[0146] In an exemplary embodiment, the training process of the point cloud data completion model includes:

[0147] Input the sample point cloud data into the initial completion model, and output the predicted point cloud data.

[0148] For a target point, the loss weight of the target point is determined based on the distance between the target point and each ground truth key point and the three-dimensional Gaussian distribution corresponding to each ground truth key point. The target point is any point in the ground truth point cloud data corresponding to the predicted point cloud data or the sample point cloud data.

[0149] Based on the predicted point cloud data, the loss weights corresponding to each point in the predicted point cloud data, the ground truth point cloud data, and the loss weights corresponding to each point in the ground truth point cloud data, the weighted chamfer loss corresponding to the initial completion model is determined.

[0150] The parameters of the initial completion model are adjusted based on the weighted chamfer loss.

[0151] When the adjusted initial completion model meets the training stopping condition, the point cloud data completion model is obtained.

[0152] In an exemplary embodiment, when the distance between the target point and the truth key point is greater than or equal to the upper limit of the distance, the loss weight of the target point remains unchanged with a first loss weight, which is a loss weight determined based on the upper limit of the distance.

[0153] Each module in the aforementioned point cloud data completion device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0154] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 8As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements a point cloud data completion method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0155] Those skilled in the art will understand that Figure 8 The structure shown is a block diagram of a partial structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0156] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0157] In one exemplary embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described method embodiments.

[0158] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0159] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0160] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program mentioned can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0161] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0162] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A point cloud data completion method, characterized in that, The method, applied to point cloud data completion models, includes: Feature extraction is performed on the point cloud data to be completed at the current moment to obtain the point cloud features of the point cloud data to be completed. The historical fused point cloud features from the previous moment are temporally fused with the point cloud features to obtain the fused point cloud features at the current moment. Key point identification is performed based on the fused point cloud features at the current moment to obtain key point identification results; Based on the key point identification results and the point cloud data to be completed, target point cloud data is generated.

2. The method according to claim 1, characterized in that, The step of temporally fusing the historical fused point cloud features from the previous moment with the current point cloud features to obtain the fused point cloud features at the current moment includes: The historical fused point cloud features from the previous moment are transformed into the coordinate system of the current moment to obtain the target fused point cloud features; The target fused point cloud feature is fused with the point cloud feature in a time sequence to obtain the fused point cloud feature at the current moment.

3. The method according to claim 2, characterized in that, The step of transforming the historical fused point cloud features from the previous moment to the coordinate system of the current moment to obtain the target fused point cloud features includes: The motion data from the previous moment to the current moment is obtained from the motion sensing module, and the motion transformation matrix from the previous moment to the current moment is determined based on the motion data; The motion transformation matrix is ​​used to transform the historical fused point cloud features of the previous moment to obtain the target fused point cloud features.

4. The method according to claim 2 or 3, characterized in that, The step of temporally fusing the target fused point cloud features with the point cloud features to obtain the fused point cloud features at the current moment includes: The target fused point cloud feature is concatenated with the point cloud feature to obtain the concatenated feature; Attention feature fusion is performed on the stitched features and the target fused point cloud features to obtain the fused point cloud features at the current time.

5. The method according to claim 4, characterized in that, The step of performing attention feature fusion on the stitched features and the target fused point cloud features to obtain the fused point cloud features at the current time includes: Using the spliced ​​features as the query vector and the target fused point cloud features as the value vector and key vector, intermediate features are generated; Using the intermediate features as the query vector and the concatenated features as the value vector and key vector, the fused point cloud features at the current moment are generated.

6. The method according to claim 1, characterized in that, The training process of the point cloud data completion model includes: Input the sample point cloud data into the initial completion model, and output the predicted point cloud data. For a target point, the loss weight of the target point is determined based on the distance between the target point and each ground truth key point and the three-dimensional Gaussian distribution corresponding to each ground truth key point. The target point is any point in the ground truth point cloud data corresponding to the predicted point cloud data or the sample point cloud data. Based on the predicted point cloud data, the loss weights corresponding to each point in the predicted point cloud data, the ground truth point cloud data, and the loss weights corresponding to each point in the ground truth point cloud data, the weighted chamfer loss corresponding to the initial completion model is determined. The parameters of the initial completion model are adjusted based on the weighted chamfer loss. When the adjusted initial completion model meets the training stopping condition, the point cloud data completion model is obtained.

7. The method according to claim 6, characterized in that, When the distance between the target point and the truth key point is greater than or equal to the upper limit of the distance, the loss weight of the target point remains unchanged with the first loss weight, which is a loss weight determined based on the upper limit of the distance.

8. A point cloud data completion device, characterized in that, The device, applied to point cloud data completion models, includes: The extraction module is used to extract features from the point cloud data to be completed at the current time to obtain the point cloud features of the point cloud data to be completed. The fusion module is used to perform temporal fusion of the historical fused point cloud features from the previous moment with the point cloud features to obtain the fused point cloud features at the current moment. The identification module is used to identify key points based on the fused point cloud features at the current moment, and obtain key point identification results; The completion module is used to generate target point cloud data based on the key point identification results and the point cloud data to be completed.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images