A method, device, robot and medium for completing point cloud data

Abstract

The present application relates to the field of data processing technology, and in particular to a method, device, robot and medium for completing point cloud data, which is used to solve the problem of low task execution success rate and accuracy in the point cloud completion method in the related technology. The method is: obtaining original point cloud data containing a target to be completed; based on the original point cloud data, screening out at least one complete point cloud data matching the target to be completed from the complete point cloud data of multiple targets; based on the at least one complete point cloud data, performing point cloud completion on the missing shape of the target to be completed in the original point cloud data to obtain the target point cloud data of the target to be completed; in this way, the point cloud completion of the missing shape of the target to be completed contained in the original point cloud data is realized based on the prior shape, which improves the point cloud quality, thereby improving the robot's perception accuracy and environmental adaptability, and further improving the success rate and accuracy of task execution.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, robot and medium for completing point cloud data. Background Art

[0002] Understanding 3D objects plays an important role in AI application fields such as embodied intelligence and augmented reality. In practical applications, robots mostly acquire environmental images based on the acquisition device of the robot body, such as the acquisition device on the robot head, so as to perceive the environment based on the environmental image, and then perform corresponding tasks according to the perceived environmental information, such as object grasping and workpiece assembly. Therefore, the success rate and accuracy of the robot's execution of tasks are strongly related to the environmental information perceived by the robot.

[0003] However, the robot's visual perception is limited by occlusion, limited viewing angle and sensor resolution, so that the perceived environmental information, that is, the original point cloud data converted from the collected environmental image, is usually sparse and missing. In related technologies, point cloud completion is often used to predict the missing parts based on the global features obtained from the original point cloud data to achieve data completion. However, the global features are scattered and do not retain too much fine-grained information, which reduces the success rate and accuracy of executing tasks based on the completed point cloud data. Summary of the invention

[0004] The embodiments of the present application provide a method, device, robot and medium for completing point cloud data, which are used to screen out complete point cloud data that matches the target to be completed from the complete point cloud data of multiple targets, so as to achieve shape completion of the completion target contained in the original point cloud data, thereby improving the point cloud quality and further improving the success rate and accuracy of task execution.

[0005] The specific technical solutions provided by the embodiments of this application are as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for completing point cloud data, comprising:

[0007] Obtain the original point cloud data containing the target to be completed;

[0008] Based on the original point cloud data, at least one complete point cloud data matching the target to be completed is screened out from the complete point cloud data of multiple targets;

[0009] Based on the at least one complete point cloud data, point cloud completion is performed on the missing shape of the target to be completed in the original point cloud data to obtain target point cloud data of the target to be completed.

[0010] The above method provided in the embodiment of the present application is based on the original point cloud data containing the target to be completed, and screens out complete point cloud data matching the target to be completed from the complete point cloud data of multiple targets, thereby realizing the point cloud completion of the missing shape of the target to be completed contained in the original point cloud data based on the prior shape, improving the point cloud quality, thereby improving the robot's perception accuracy and environmental adaptability, and further improving the success rate and accuracy of task execution.

[0011] In a possible implementation, the obtaining of original point cloud data containing the target to be completed includes:

[0012] Capturing a first image containing the target to be completed by a first acquisition device of the robot body;

[0013] Acquiring a second image including the target to be completed by a second acquisition device on the end effector of the robot;

[0014] converting the first image into first point cloud data, and converting the second image into second point cloud data;

[0015] The first point cloud data and the second point cloud data are superimposed to obtain the original point cloud data.

[0016] The above method acquires images containing the target to be completed by the first acquisition device and the second acquisition device, so as to obtain the shape features of the target to be completed from multiple angles, thereby improving the shape features of the target to be completed contained in the original point cloud data obtained by converting the images acquired at multiple angles, and further improving the matching degree between the complete point cloud data screened out from the complete point cloud data of multiple targets based on the original point cloud data and the target to be completed, so as to improve the quality of the completed point cloud.

[0017] In a possible implementation, the selecting, based on the original point cloud data, at least one complete point cloud data matching the target to be completed from the complete point cloud data of multiple targets includes:

[0018] Based on the task to be processed, target recognition is performed on the original point cloud data to obtain the target to be completed;

[0019] Extracting local shape features of the target to be completed from the original point cloud data;

[0020] Perform feature matching on the local shape features and the shape features of the complete point cloud data of the multiple targets to obtain respective matching rates;

[0021] The complete point cloud data corresponding to at least one shape feature having a matching rate greater than a first matching rate threshold among the matching rates is determined as the at least one complete point cloud data.

[0022] The above method extracts local shape features and matches at least one complete point cloud data that matches the target to be completed based on the local shape features, thereby improving the quality of the subsequent completed point cloud.

[0023] In a possible implementation, based on the at least one complete point cloud data, performing point cloud completion on the missing shape of the target to be completed in the original point cloud data to obtain target point cloud data of the target to be completed includes:

[0024] Extracting a priori shape features of the at least one complete point cloud data;

[0025] Based on the inference rule set, the local shape feature is compared with the extracted prior shape feature to infer the missing shape feature of the target to be completed, wherein the inference rule set is used to ensure that the inferred missing shape feature is the shape feature of the target to be completed;

[0026] Based on the local shape features and the missing shape features of the target to be completed, point cloud completion is performed on the missing shape of the target to be completed in the original point cloud data to obtain the target point cloud data.

[0027] In the above method, since the filtered shapes should not change the input original point cloud data, that is, the original structure of the partial point cloud, in order to better utilize the retrieved shapes, through the inference rule set, it is possible to infer useful information suitable for completing the missing shape of the target to be completed based on at least one complete point cloud data that has been filtered out, that is, the missing shape features, so as to improve the point cloud quality.

[0028] In a possible implementation, the shape feature and complete point cloud data of each target are stored in the form of a key-value pair, wherein the key in each key-value pair represents the shape feature of the target, and the value represents the complete point cloud data of the target;

[0029] The step of determining the complete point cloud data corresponding to at least one shape feature having a matching rate greater than a first matching rate threshold value as the at least one complete point cloud data comprises:

[0030] Obtaining a key corresponding to at least one shape feature whose matching rates are greater than the first matching rate threshold value;

[0031] Based on the acquired key and multiple key-value pairs, complete point cloud data of the values ​​corresponding to the acquired keys are obtained.

[0032] The above method improves the retrieval speed of selecting at least one complete point cloud data matching the target to be completed from the complete point cloud data of multiple targets through key-value pair storage, thereby improving processing efficiency.

[0033] In a possible implementation, the multiple key-value pairs are obtained in the following manner:

[0034] Based on multiple training samples, at least one round of training is performed on the complete shape encoder to be trained until the convergence condition is met, and the corresponding relationships between the shape features of multiple targets and the complete point cloud data are respectively stored in the form of key-value pairs to obtain the multiple key-value pairs, each training sample includes the complete point cloud data of the training target and each part of the point cloud data, and the belonging relationship between the each part of the point cloud data and the complete point cloud data of the training target; wherein, in each round of training:

[0035] Inputting each portion of point cloud data included in each training sample into the complete shape encoder to be trained, and using the complete shape encoder to extract features of the portion of point cloud data;

[0036] Based on the features, the partial point cloud data are classified to obtain the complete point cloud data to which the partial point cloud data belongs;

[0037] Based on the obtained complete point cloud data to which the partial point cloud data belongs and various attribution relationships, a loss value is determined, and based on the loss value, the parameters of the complete shape encoder to be trained are adjusted.

[0038] The above method, by pre-training the complete shape encoder, enables the complete shape encoder to learn the dependency between partial shapes and complete shapes in the feature space, thereby improving the screening accuracy in the actual reasoning stage, and then improving the quality of the completed point cloud, improving the robot's perception accuracy and environmental adaptability, and then improving the success rate and accuracy of task execution.

[0039] In a possible implementation, each key-value pair includes a generation time; the multiple key-value pairs are updated in the following manner:

[0040] In response to the update instruction, selecting each training point cloud data corresponding to the update instruction from the acquired multiple original point cloud data and the target point cloud data;

[0041] Extracting shape features of each training point cloud data respectively, and comparing each extracted shape feature with the shape features in the multiple key-value pairs respectively;

[0042] If the matching rate between the shape feature of the first training point cloud data and the shape feature in any key-value pair is greater than a second matching rate threshold, replacing any key-value pair with a key-value pair consisting of the shape feature of the first training point cloud data and the first training point cloud data;

[0043] If the matching rate between the shape features of the second training point cloud data and the shape features in the multiple key-value pairs is less than a third matching rate threshold, the shape features of the second training point cloud data and the second training point cloud data are stored in the form of key-value pairs.

[0044] In the above method, the ability to distinguish similar shapes is achieved through the key-value pair update strategy. This method can also save unknown shapes to improve screening efficiency and screening effect.

[0045] In a second aspect, an embodiment of the present application provides a data completion device for point cloud data, comprising:

[0046] An acquisition module is used to acquire the original point cloud data containing the target to be completed;

[0047] A screening module, configured to screen out at least one complete point cloud data matching the target to be completed from the complete point cloud data of multiple targets based on the original point cloud data;

[0048] The point cloud completion module is used to complete the missing shape of the target to be completed in the original point cloud data based on the at least one complete point cloud data, so as to obtain the target point cloud data of the target to be completed.

[0049] In a possible implementation, the acquisition module is specifically used to:

[0050] Capturing a first image containing the target to be completed by a first acquisition device of the robot body;

[0051] Acquiring a second image including the target to be completed by a second acquisition device on the end effector of the robot;

[0052] converting the first image into first point cloud data, and converting the second image into second point cloud data;

[0053] The first point cloud data and the second point cloud data are superimposed to obtain the original point cloud data.

[0054] In a possible implementation, the screening module is specifically used to:

[0055] Based on the task to be processed, target recognition is performed on the original point cloud data to obtain the target to be completed;

[0056] Extracting local shape features of the target to be completed from the original point cloud data;

[0057] Perform feature matching on the local shape features and the shape features of the complete point cloud data of the multiple targets to obtain respective matching rates;

[0058] The complete point cloud data corresponding to at least one shape feature having a matching rate greater than a first matching rate threshold among the matching rates is determined as the at least one complete point cloud data.

[0059] In a possible implementation, the point cloud completion module is specifically used to:

[0060] Extracting a priori shape features of the at least one complete point cloud data;

[0061] Based on the inference rule set, the local shape feature is compared with the extracted prior shape feature to infer the missing shape feature of the target to be completed, wherein the inference rule set is used to ensure that the inferred missing shape feature is the shape feature of the target to be completed;

[0062] Based on the local shape features and the missing shape features of the target to be completed, point cloud completion is performed on the missing shape of the target to be completed in the original point cloud data to obtain the target point cloud data.

[0063] In a possible implementation, the shape feature and complete point cloud data of each target are stored in the form of a key-value pair, wherein the key in each key-value pair represents the shape feature of the target, and the value represents the complete point cloud data of the target; the screening module is specifically used to:

[0064] Obtaining a key corresponding to at least one shape feature whose matching rates are greater than the first matching rate threshold value;

[0065] Based on the acquired key and multiple key-value pairs, complete point cloud data of the values ​​corresponding to the acquired keys are obtained.

[0066] In a possible implementation, the multiple key-value pairs are obtained in the following manner:

[0067] Based on multiple training samples, at least one round of training is performed on the complete shape encoder to be trained until the convergence condition is met, and the corresponding relationships between the shape features of multiple targets and the complete point cloud data are respectively stored in the form of key-value pairs to obtain the multiple key-value pairs, each training sample includes the complete point cloud data of the training target and each part of the point cloud data, and the belonging relationship between the each part of the point cloud data and the complete point cloud data of the training target; wherein, in each round of training:

[0068] Inputting each portion of point cloud data included in each training sample into the complete shape encoder to be trained, and using the complete shape encoder to extract features of the portion of point cloud data;

[0069] Based on the features, the partial point cloud data are classified to obtain the complete point cloud data to which the partial point cloud data belongs;

[0070] Based on the obtained complete point cloud data to which the partial point cloud data belongs and various attribution relationships, a loss value is determined, and based on the loss value, the parameters of the complete shape encoder to be trained are adjusted.

[0071] In a possible implementation, each key-value pair includes a generation time; the multiple key-value pairs are updated in the following manner:

[0072] In response to the update instruction, selecting each training point cloud data corresponding to the update instruction from the acquired multiple original point cloud data and the target point cloud data;

[0073] Extracting shape features of each training point cloud data respectively, and comparing each extracted shape feature with the shape features in the multiple key-value pairs respectively;

[0074] If the matching rate between the shape feature of the first training point cloud data and the shape feature in any key-value pair is greater than a second matching rate threshold, replacing any key-value pair with a key-value pair consisting of the shape feature of the first training point cloud data and the first training point cloud data;

[0075] If the matching rate between the shape features of the second training point cloud data and the shape features in the multiple key-value pairs is less than a third matching rate threshold, the shape features of the second training point cloud data and the second training point cloud data are stored in the form of key-value pairs.

[0076] In a third aspect, an embodiment of the present application provides a robot, comprising:

[0077] A first acquisition device, deployed on the robot body;

[0078] A second acquisition device, deployed on the robot end effector, includes a depth camera with an inertial measurement unit IMU;

[0079] Memory, used to store computer programs or instructions;

[0080] A processor is used to execute the computer program or instructions in the memory so that any method in the first aspect described above is executed.

[0081] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium. When instructions in the storage medium are executed by a processor, the processor is enabled to execute any one of the methods described in the first aspect above.

[0082] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes: a computer program code, when the computer program code is run on a computer, enables the computer to execute the method as described in any one of the first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 A schematic diagram of a data completion process for point cloud data in an embodiment of the present application;

[0084] Figure 2 A schematic diagram of the distribution position of a collection device of a robot in an embodiment of the present application;

[0085] Figure 3 This is a system architecture block diagram of a second acquisition device in an embodiment of the present application;

[0086] Figure 4 This is a schematic diagram of the main appearance, key modules and field of view parameter configuration of a second acquisition device in an embodiment of the present application;

[0087] Figure 5 This is a schematic diagram of the structure of a metal hoop in an embodiment of the present application;

[0088] Figure 6 This is an intentional diagram of the assembly effect of a metal hoop in an embodiment of the present application after a second collecting device is installed;

[0089] Figure 7 This is a schematic diagram of a process of acquiring original point cloud data in an embodiment of the present application;

[0090] Figure 8 A schematic diagram of a screening process of complete point cloud data in an embodiment of the present application;

[0091] Fig. 9 A schematic diagram of a network architecture of a point cloud completion network model in an embodiment of the present application;

[0092] Fig.10 A schematic diagram of a specific screening process of complete point cloud data in an embodiment of the present application;

[0093] Fig.11 A schematic diagram of a process of obtaining target point cloud data in an embodiment of the present application;

[0094] Fig.12 Schematic diagram of the point cloud completion process of a toy airplane in an embodiment of the present application;

[0095] Fig.13 A schematic diagram of the logical architecture of a data completion device for point cloud data in an embodiment of the present application;

[0096] Fig.14 Schematic diagram of the physical structure of the robot in the embodiment of the present application. DETAILED DESCRIPTION

[0097] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0098] It should be noted that the terms "first", "second", "third", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0099] The following is a brief introduction to the design concept of the embodiments of the present application.

[0100] The present application relates to the field of data processing technology, and mainly to a method, device, robot and medium for completing point cloud data.

[0101] Usually, robots perceive the external environment through vision, such as acquiring images of the environment through acquisition devices, and then converting the images into raw point cloud data, so as to perform corresponding tasks based on the perceived raw point cloud data. However, the robot's visual perception is limited by occlusion, limited viewing angle and sensor resolution, so that the raw point cloud data is usually sparse and missing.

[0102] In related technologies, point cloud completion is often used to predict missing parts based on global features obtained from original point cloud data to complete data. However, since the shape of the target included in the original point cloud data is incomplete, detailed structural information cannot be obtained from the incomplete shape, and the global features are scattered and do not retain too much fine-grained information, which results in a decrease in the success rate and accuracy of tasks performed based on the completed point cloud data.

[0103] In view of this, in order to solve the problem of low task execution success rate and accuracy in the point cloud completion method in the related art, an embodiment of the present application provides a data completion method for point cloud data. In the embodiment of the present application, original point cloud data containing a target to be completed is obtained; based on the original point cloud data, at least one complete point cloud data matching the target to be completed is screened out from the complete point cloud data of multiple targets; based on the at least one complete point cloud data, the missing shape of the target to be completed in the original point cloud data is completed by point cloud, so as to obtain the target point cloud data of the target to be completed.

[0104] In an embodiment of the present application, complete point cloud data that matches the target to be completed is screened out from the complete point cloud data of multiple targets, and point cloud completion of the missing shape of the target to be completed contained in the original point cloud data based on the prior shape is achieved, thereby improving the point cloud quality, thereby improving the robot's perception accuracy and environmental adaptability, and further improving the success rate and accuracy of task execution.

[0105] A data completion method for point cloud data provided in an embodiment of the present application is applicable to robots, such as mobile robots, intelligent agents, multi-legged robots, bionic humans, bipedal robots, humanoid / type robots, etc.

[0106] The preferred implementation modes of the present application are further described in detail below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application, and the embodiments of the present application and the features in the embodiments may be combined with each other if there is no conflict.

[0107] See also Figure 1 As shown, the embodiment of the present application provides a method for completing point cloud data, and the specific process of the method includes but is not limited to the following:

[0108] Step 100: Acquire original point cloud data containing the target to be completed.

[0109] In some embodiments of the present application, the method can be applied to robots or other intelligent devices. The following only takes the application to robots as an example to describe in detail the data completion scheme for point cloud data in the present application.

[0110] In some embodiments, see Figure 2 As shown, the robot includes a first acquisition device deployed on the robot body, and a second acquisition device deployed on the robot end effector, wherein the second acquisition device includes a depth camera with an inertial measurement unit (IMU).

[0111] like Figure 2As shown, the first acquisition device can be deployed on the head of the robot or on other positions of the robot. It can be understood that the first acquisition device is the main acquisition device for the robot to perceive the environment; the second acquisition device can be deployed on the wrist of the robot, and can include a red, green, blue depth (RGBD) camera with IMU inertial navigation compensation, wherein the camera is implemented based on the principle of structured light, and the depth camera emits a structured laser spot through a lightweight directory access protocol (LDMP) light array, and calculates the spatial distance based on the spot distortion through an infrared (IR) lens.

[0112] In some embodiments of the present application, the second acquisition device integrates the IMU inertial navigation function, so that the end effector of the robot can be effectively compensated. Figure 2 The end of the humanoid robot's hand is shown vibrating, making distance detection and imaging more precise.

[0113] In other embodiments, the second acquisition device may also be integrated with a proximity sensor (Line Departure Prevention, LDP), which can detect the distance of a face approaching the camera for laser safety protection to prevent accidental injuries caused by the approaching face.

[0114] In some embodiments, the system architecture block diagram of the second acquisition device can be as follows: Figure 3 shown. Figure 3 In the embodiment, the system architecture of the second acquisition device includes an application specific integrated circuit (ASIC), an image signal processor (ISP), an RGB module, an IR module, an IR light emitting diode (LED), an LDP module, an LDMP module, a universal serial bus (USB), and an IMU.

[0115] In some embodiments, the second acquisition device can be arranged at the wrist of the robot, facing the operation direction of the end effector such as a hand, and can be symmetrically arranged in structure, such as Figure 2 It is shown that the second acquisition devices are deployed simultaneously on the wrists of the left and right hands of the robot.

[0116] In other embodiments, the cameras deployed on the left and right hands of the robot can be the same or different. In some embodiments, the main shape, key modules and field of view (FOV) parameters of the second acquisition device can be as follows: Figure 4 shown.

[0117] In some embodiments, the second acquisition device may be integrated with a quick-release / quick-change function, and the axial direction of the robot's forearm may be used as the rotation axis to support angle adjustment in any direction, thereby ensuring the scalability of application scenarios. The quick-release / quick-change function may be designed in a variety of ways. In some embodiments, a symmetrically designed flange metal hoop may fix the second acquisition device and other accessories at any position on the wrist and forearm.

[0118] In some application scenarios with large vibrations, the metal hoop in the present application may also include anti-slip spikes, and further locking can be achieved through the anti-slip spikes on the metal hoop, so that it can be reliably fixed both axially and radially.

[0119] In some embodiments, the metal hoop may also provide a camera gimbal fixing position, such as a fixing screw hole, for installing the corresponding gimbal. The metal hoop may be symmetrically designed, and a camera or other auxiliary equipment, such as a fill light, may also be installed on the opposite side. The structure of the flange metal hoop may be as follows: Figure 5 shown.

[0120] In other embodiments, the fixing screw holes of the camera gimbal may not be provided on the metal hoop, and the second acquisition device may be directly fixed.

[0121] It is understandable that the second acquisition device can be installed together with the metal hoop, or can be installed together with the metal hoop through a universal pan / tilt, which can more effectively perform hand-eye calibration of the second acquisition device and the end effector such as a hand. Figure 6 The figure shows the assembly effect after installing a second collection device.

[0122] After introducing the first acquisition device and the second acquisition device deployed on the robot, in some embodiments, based on the aforementioned structure, the first acquisition device and the second acquisition device are continuously used to acquire images within their respective fields of view, the environment is perceived based on the acquired images, and then corresponding tasks are performed based on the perceived environmental information. In the present application, the image acquisition cycles of the first acquisition device and the second acquisition device may be the same, and images acquired at the same or similar times are processed as a group. For ease of description, the following is a detailed description of only a group of images acquired by the first acquisition device and an image acquired by the second acquisition device as an example. The processing process of each subsequent group of images is similar to this and will not be repeated here.

[0123] In some embodiments of the present application, when executing step 100, refer to Figure 7 As shown, the following steps can be performed to obtain the original point cloud data containing the target to be completed:

[0124] Step 1001: A first image including a target to be completed is captured by a first capturing device of a robot body.

[0125] Step 1002: A second image including the object to be completed is acquired by a second acquisition device on the robot end effector.

[0126] Step 1003: Convert the first image into first point cloud data, and convert the second image into second point cloud data.

[0127] In some embodiments of the present application, when executing step 1003, a traditional data reconstruction method is used to convert the first image into first point cloud data and the second image into second point cloud data.

[0128] Step 1004: superimpose the first point cloud data and the second point cloud data to obtain original point cloud data.

[0129] In some embodiments of the present application, since the robot has been pre-calibrated for hand and eye, when executing step 1004, the first point cloud data and the second point cloud data can be directly superimposed to obtain the original point cloud data containing the target to be completed.

[0130] Step 110: Based on the original point cloud data, at least one complete point cloud data matching the target to be completed is screened out from the complete point cloud data of multiple targets.

[0131] In some embodiments of the present application, when executing step 110, refer to Figure 8 As shown, this can be achieved by executing the following steps:

[0132] Step 1101: Based on the task to be processed, target recognition is performed on the original point cloud data to obtain the target to be completed.

[0133] In some embodiments of the present application, the robot performs grabbing or operation on a certain target, etc., which is usually based on a pre-received pending task, which can be a task generated based on a received voice instruction, or a pre-issued task, and the pending task includes the target that the robot needs to process. Depending on the application scenario, the person who issues the pending task is also different. For example, in a family scenario, the pending task may be a family member who uses the robot. For another example, in a shopping guide scenario, the pending task may be a customer served by the robot; for another example, in a production scenario, the pending task may be a manager in the production scenario, etc.

[0134] In some embodiments of the present application, a point cloud completion network model can be pre-built. The network architecture of the point cloud completion network model is as follows: Fig. 9 As shown. Fig. 9 As shown, the point cloud completion network model includes a shape encoder, a memory network model, a priori shape selection model and a shape decoder, wherein:

[0135] The shape encoder may be a model based on a multimodal pre-trained neural network model (Contrastive Language-Image Pre-training, CLIP), and is used to identify the target included in the task to be processed, i.e., the target to be completed, based on the text included in the task to be processed and the original point cloud data, and extract the shape features of the target to be completed in the original point cloud data. Due to factors such as limited vision, the shapes of the target to be completed included in the original point cloud data are all partial shapes. Therefore, in order to facilitate distinction, the shape features are recorded as local shape features;

[0136] A memory network model, comprising a complete shape encoder, wherein the complete shape encoder is obtained based on training, and the dependency relationship between the partial shape and the complete shape in the feature space can be learned through training, so that the model is used to retrieve at least one complete point cloud data matching the target to be completed from the complete point cloud data of multiple targets based on the local shape features of the target to be completed;

[0137] A priori shape selection model is used to infer the missing shape features of the target to be completed based on at least one complete point cloud data and local shape features;

[0138] The shape decoder is used to complete the missing shape of the target to be completed in the original point cloud data based on the local shape features and the missing shape features, so as to obtain the target point cloud data of the target to be completed.

[0139] In some embodiments of the present application, when executing step 1001, the original point cloud data is input into the aforementioned point cloud completion network model, and the shape encoder in the point cloud completion network model is used to perform target recognition on the original point cloud data based on the task to be processed, and the target corresponding to the task to be processed is identified and determined as the target to be completed.

[0140] Step 1102: extracting local shape features of the target to be completed from the original point cloud data.

[0141] In some embodiments of the present application, when executing step 1102, the local shape features of the target to be completed are extracted from the original point cloud data through the aforementioned shape encoder.

[0142] like Fig. 9 As shown, assuming that the original point cloud data is represented as in, represents each point in the point cloud data of the target to be completed, and N is the number of points of the shape; then, by Fig. 9 The shape encoder shown in Figure 2 can express the local shape features of the target to be completed as

[0143] Step 1103: perform feature matching on the local shape features and the shape features of the complete point cloud data of multiple targets to obtain respective matching rates.

[0144] In some embodiments of the present application, in the memory network model, the shape features and complete point cloud data of each target are stored in the form of key (Key)-value (Value) pairs, wherein the key in each key-value pair represents the shape features of the target, and the value represents the complete point cloud data of the target. In some embodiments, the aforementioned shape features may include the complete shape features of the target and partial shape features at different angles. Fig. 9 As shown, in some embodiments of the present application, each Ki (i.e., Key i) is extracted from Vi (i.e., Value i) by a complete shape encoder in the memory network model, which can be expressed as F Vi .

[0145] Then, in some embodiments of the present application, in executing step 1102, the local shape features of the target to be completed are obtained. After that, it is output to the memory network model, and step 1103 is executed to convert the local shape features into The shape features F of the complete point cloud data of multiple targets are Vi Perform feature matching to obtain the matching rates, where the shape feature F of any target Vi It is extracted from the corresponding complete point cloud data through the complete shape encoder in the memory network model.

[0146] In some embodiments of the present application, the cosine similarity of two features can be calculated to obtain the matching rate between the two features, thereby obtaining the matching rate between the target to be completed and any target in the multiple targets. and the shape feature F in the memory network Vi The matching rate between them can be expressed by the following formula:

[0147]

[0148] Among them, Sim() represents cosine similarity, that is, matching rate; i is the sequence number; Represents local shape features; F Vi Represents the shape features of the i-th complete point cloud data.

[0149] Step 1104: Determine the complete point cloud data corresponding to at least one shape feature with a matching rate greater than a first matching rate threshold as the aforementioned at least one complete point cloud data.

[0150] In some embodiments of the present application, when executing step 1104, refer to Fig.10 As shown, the following steps are specifically performed to obtain at least one complete point cloud data:

[0151] Step 1000: Obtain a key corresponding to at least one shape feature whose matching rate is greater than a first matching rate threshold.

[0152] In some embodiments of the present application, when executing step 1000, the key corresponding to at least one shape feature with a matching rate greater than a first matching rate threshold is obtained, such as obtaining the top-K keys with the maximum matching rate.

[0153] Step 1010: Based on the acquired key and multiple key-value pairs, complete point cloud data of the value corresponding to the acquired key is obtained.

[0154] In some embodiments of the present application, when executing step 1010, according to the obtained key, a value corresponding to the sequence number of the obtained key is obtained from multiple key-value pairs, thereby obtaining complete point cloud data corresponding to each value. Continuing with the above example, if the top-K keys with the maximum matching rate are obtained, the obtained complete point cloud data can be expressed by the following formula:

[0155]

[0156] Where V represents the complete point cloud dataset obtained; V ni Represents the i-th complete point cloud data obtained; i is the sequence number, i∈(1, K) in the above formula; ni represents the i-th memory item in multiple key-value pairs; Represents the local shape features of the target to be completed; F Vi represents the shape feature of the target; Sim() represents the cosine similarity, that is, the matching rate is determined by the cosine similarity method; argmaxSim() represents the set that makes Sim() take the maximum value; the memory item in the aforementioned memory network model can be expressed as in is the number of memory items, A i Indicates the time when the corresponding "key-value" pair is established, recorded as the generation time.

[0157] In some embodiments of the present application, the complete shape encoder to be trained can be trained for at least one round based on multiple training samples until the convergence condition is met, and the corresponding relationships between the shape features of the multiple targets and the complete point cloud data are stored in the form of key-value pairs to obtain the above-mentioned key-value pairs. Each training sample includes the complete point cloud data and each part of the point cloud data of the training target, as well as the attribution relationship between the each part of the point cloud data and the complete point cloud data of the training target.

[0158] Among them, in each round of training, the following operations can be performed: Operation 1, input each partial point cloud data included in each training sample into the complete shape encoder to be trained, and use the complete shape encoder to extract the features of the partial point cloud data; Operation 2, based on the features, classify the partial point cloud data to obtain the complete point cloud data to which the partial point cloud data belongs; Operation 3, determine the loss value based on the complete point cloud data to which the obtained partial point cloud data belongs, and each attribution relationship, and adjust the parameters of the complete shape encoder to be trained based on the loss value.

[0159] In some embodiments of the present application, each key-value pair may also include a generation time; then, multiple key-value pairs may be updated in the following manner to enhance the memory network model's ability to receive similar shapes, save unknown shapes, and update multiple key-value pairs.

[0160] In some embodiments of the present application, the updating of the obtained multiple key-value pairs can be achieved by performing the following operations: Operation 1, responding to the update instruction, selecting each training point cloud data corresponding to the update instruction from the multiple original point cloud data and target point cloud data obtained; Operation 2, respectively extracting the shape features of each training point cloud data, and respectively comparing the extracted shape features with the shape features in the multiple key-value pairs; Operation 3, if the matching rate between the shape features of the first training point cloud data and the shape features in any key-value pair is greater than the second matching rate threshold, then replacing the any key-value pair with the key-value pair consisting of the shape features of the first training point cloud data and the first training point cloud data; Operation 4, if the matching rate between the shape features of the second training point cloud data and the shape features in the multiple key-value pairs is less than the third matching rate threshold, then storing the shape features of the second training point cloud data and the second training point cloud data in the form of key-value pairs.

[0161] In some embodiments of the present application, the above-mentioned matching rate can be determined by the aforementioned cosine similarity calculation method, or other feature matching methods, which will not be described in detail here.

[0162] In some embodiments of the present application, when determining to update or add a new key-value pair, the incomplete point cloud feature "key-value" pairs can be associated from multiple angles, thereby enhancing the memory network's geometric perception of incomplete shapes.

[0163] Step 120: Based on at least one complete point cloud data, point cloud completion is performed on the missing shape of the target to be completed in the original point cloud data to obtain target point cloud data of the target to be completed.

[0164] In some embodiments of the present application, when executing step 120, refer to Fig.11 As shown, perform the following operations to implement the corresponding functions:

[0165] Step 1201: extracting a priori shape features of at least one complete point cloud data.

[0166] In some embodiments of the present application, when executing step 1201, if Fig. 9 As shown, a prior shape feature of at least one complete point cloud data is extracted through a prior shape encoder in a prior shape selection model, wherein the prior shape feature focuses on the boundary features of the target and pays less attention to its internal features.

[0167] Step 1202: Based on the inference rule set, the local shape features are compared with the extracted prior shape features to infer the missing shape features of the target to be completed, wherein the inference rule set is used to ensure that the inferred prior shape features are the shape features of the target to be completed.

[0168] In some embodiments of the present application, in order to obtain better prior knowledge information, a causal reasoning module is pre-constructed in the prior shape selection model, and the causal reasoning module is constructed by an inference rule set, wherein the inference rule set is used to ensure that the inferred prior shape features are the shape features of the target to be completed, so as to prevent the inferred missing shape features of the target to be completed from being irrelevant to the shape of the target to be completed. Then, when executing step 1202, the local shape features are compared with the extracted prior shape features through the causal reasoning module to obtain the missing shape features of the target to be completed.

[0169] Step 1203: Based on the local shape features and the missing shape features of the target to be completed, point cloud completion is performed on the missing shape of the target to be completed in the original point cloud data to obtain target point cloud data.

[0170] In some embodiments of the present application, after executing step 1202, the causal reasoning module outputs the missing shape features of the completed target to the shape decoder, and the shape decoder restores the shape features back to point cloud data. Specifically, when executing step 1203, the shape decoder decodes the received missing shape features and local shape features of the completed target into the target point cloud data of the target to be completed.

[0171] The following is an example to introduce the data completion method of point cloud data in some embodiments of the present application.

[0172] For example, in a home scenario, the user sends a task to be processed to the robot, such as, "Get me a toy plane". Then, based on the task to be processed, the robot collects images of the environment in which the robot is located through the first acquisition device. At this time, since the second acquisition device is deployed on the end effector, such as the wrist, assuming that the end effector of the robot, such as the arm, is in a vertical state, the images collected by the second acquisition device are mostly ground images, which are not very useful for the robot to perform tasks. In this case, the images collected by the second acquisition device may not be further processed. At this time, it is identified whether there is a target to be completed in the image. If so, the corresponding point cloud data is obtained based on the image conversion collected by the first acquisition device. Since the distance to the cup is far at this time, the point cloud data may not be input into the point cloud completion network model. Based on the point cloud data and the current position of the robot, motion planning is performed to guide the robot to approach the target.

[0173] In some embodiments, after identifying whether there is a target to be completed in the image, if it is determined that there is no target, the steering of the robot body is adjusted and the field of view of the first acquisition device is adjusted to search for the target until the target to be completed is found. The corresponding point cloud data is obtained based on the image conversion acquired by the first acquisition device, and motion planning is performed based on the point cloud data and the current position of the robot to guide the robot to approach the target.

[0174] In other embodiments, after obtaining the aforementioned point cloud data, the point cloud data can also be input into the point cloud completion network model. The point cloud completion network model can identify whether there is a target to be completed, that is, a toy airplane, based on the task to be processed. If not, the target is searched by adjusting the steering of the robot body and the field of view of the first acquisition device until the target to be completed is found, and motion planning is performed to guide the robot to approach the target to be completed.

[0175] After the robot approaches the target, the first acquisition device continuously acquires the first image containing the target to be completed, and the second acquisition device acquires the second image containing the target to be completed, and the aforementioned point cloud data completion method is performed for each group of the first image and the second image.

[0176] Specifically, the first image is converted into first point cloud data, and the second image is converted into second point cloud data, and the first point cloud data and the second point cloud data are superimposed to obtain the original point cloud data of the target to be completed, that is, Fig.12 The raw point cloud data of the toy airplane shown.

[0177] See also Fig.12 As shown, the original point cloud data is input into the shape encoder included in the point cloud completion network model. The shape encoder performs target recognition on the original point cloud data based on the task to be processed, recognizes the toy cup, and then extracts the local shape features of the toy airplane from the original point cloud data.

[0178] The local shape features of the toy airplane are input into the memory network model, and the memory network model retrieves at least one complete point cloud data matching the toy airplane from the complete point cloud data associated with multiple key-value pairs based on the local shape features of the toy airplane. Specifically, the memory network model calculates the cosine similarity between the local shape features of the toy airplane and the shape features in multiple key-value pairs to obtain the matching rate between the toy airplane and each target, and based on the obtained matching rates, the top-K complete point cloud data with matching rates greater than the first matching rate threshold are screened out from the complete point cloud data associated with multiple key-value pairs. Assuming that the value of K is 3, the obtained complete point cloud data is as follows: Fig.12 Shown are 3 complete point cloud data matching the toy airplane.

[0179] The memory network model inputs the retrieved three complete point cloud data into the prior shape selection model, extracts the prior shape features of the three complete point cloud data through the prior shape encoding included in the prior shape selection model, inputs the extracted three prior shape features into the causal reasoning module, compares the local shape features of the toy airplane with the three prior shape features through the causal reasoning module, infers the missing shape features of the toy airplane, and sends the missing shape features of the toy airplane corresponding to the inferred three complete point cloud data to the shape decoder. Fig.12 As shown, the point cloud data corresponding to the missing shape features of the toy airplane in the three complete point cloud data.

[0180] The shape decoder completes the missing shape features of the toy airplane in the original point cloud data based on the missing shape features of the toy airplane corresponding to the three received complete point cloud data and the local shape features of the toy airplane, thereby obtaining the following: Fig.12 The target point cloud data of the toy airplane is shown.

[0181] Based on the same inventive concept, refer to Fig.13 As shown, in an embodiment of the present application, a data completion device for point cloud data is provided, comprising:

[0182] An acquisition module 1310 is used to acquire original point cloud data including the target to be completed;

[0183] A screening module 1320 is used to screen out at least one complete point cloud data matching the target to be completed from the complete point cloud data of multiple targets based on the original point cloud data;

[0184] The point cloud completion module 1330 is used to complete the missing shape of the target to be completed in the original point cloud data based on the at least one complete point cloud data, so as to obtain the target point cloud data of the target to be completed.

[0185] In a possible implementation, the acquisition module 1310 is specifically configured to:

[0186] Capturing a first image containing the target to be completed by a first acquisition device of the robot body;

[0187] Acquiring a second image including the target to be completed by a second acquisition device on the end effector of the robot;

[0188] converting the first image into first point cloud data, and converting the second image into second point cloud data;

[0189] The first point cloud data and the second point cloud data are superimposed to obtain the original point cloud data.

[0190] In a possible implementation, the screening module 1320 is specifically used to:

[0191] Based on the task to be processed, target recognition is performed on the original point cloud data to obtain the target to be completed;

[0192] Extracting local shape features of the target to be completed from the original point cloud data;

[0193] Perform feature matching on the local shape features and the shape features of the complete point cloud data of the multiple targets to obtain respective matching rates;

[0194] The complete point cloud data corresponding to at least one shape feature having a matching rate greater than a first matching rate threshold among the matching rates is determined as the at least one complete point cloud data.

[0195] In a possible implementation, the point cloud completion module 1330 is specifically used to:

[0196] Extracting a priori shape features of the at least one complete point cloud data;

[0197] Based on the inference rule set, the local shape feature is compared with the extracted prior shape feature to infer the missing shape feature of the target to be completed, wherein the inference rule set is used to ensure that the inferred missing shape feature is the shape feature of the target to be completed;

[0198] Based on the local shape features and the missing shape features of the target to be completed, point cloud completion is performed on the missing shape of the target to be completed in the original point cloud data to obtain the target point cloud data.

[0199] In a possible implementation, the shape feature and complete point cloud data of each target are stored in the form of a key-value pair, wherein the key in each key-value pair represents the shape feature of the target, and the value represents the complete point cloud data of the target; the screening module 1320 is specifically used to:

[0200] Obtaining a key corresponding to at least one shape feature whose matching rates are greater than the first matching rate threshold value;

[0201] Based on the acquired key and multiple key-value pairs, complete point cloud data of the values ​​corresponding to the acquired keys are obtained.

[0202] In a possible implementation, the multiple key-value pairs are obtained in the following manner:

[0203] Based on multiple training samples, at least one round of training is performed on the complete shape encoder to be trained until the convergence condition is met, and the corresponding relationships between the shape features of multiple targets and the complete point cloud data are respectively stored in the form of key-value pairs to obtain the multiple key-value pairs, each training sample includes the complete point cloud data of the training target and each part of the point cloud data, and the belonging relationship between the each part of the point cloud data and the complete point cloud data of the training target; wherein, in each round of training:

[0204] Inputting each portion of point cloud data included in each training sample into the complete shape encoder to be trained, and using the complete shape encoder to extract features of the portion of point cloud data;

[0205] Based on the features, the partial point cloud data are classified to obtain the complete point cloud data to which the partial point cloud data belongs;

[0206] Based on the obtained complete point cloud data to which the partial point cloud data belongs and various attribution relationships, a loss value is determined, and based on the loss value, the parameters of the complete shape encoder to be trained are adjusted.

[0207] In a possible implementation, each key-value pair includes a generation time; the multiple key-value pairs are updated in the following manner:

[0208] In response to the update instruction, selecting each training point cloud data corresponding to the update instruction from the acquired multiple original point cloud data and the target point cloud data;

[0209] Extracting shape features of each training point cloud data respectively, and comparing each extracted shape feature with the shape features in the multiple key-value pairs respectively;

[0210] If the matching rate between the shape feature of the first training point cloud data and the shape feature in any key-value pair is greater than a second matching rate threshold, replacing any key-value pair with a key-value pair consisting of the shape feature of the first training point cloud data and the first training point cloud data;

[0211] If the matching rate between the shape features of the second training point cloud data and the shape features in the multiple key-value pairs is less than a third matching rate threshold, the shape features of the second training point cloud data and the second training point cloud data are stored in the form of key-value pairs.

[0212] Based on the same inventive concept, refer to Fig.14 As shown, an embodiment of the present application provides a robot, comprising:

[0213] A first acquisition device 141 is deployed on the robot body;

[0214] A second acquisition device 142, deployed on the robot end effector, includes a depth camera with an inertial measurement unit IMU;

[0215] Memory 143, for storing computer programs or instructions;

[0216] The processor 144 is used to execute the computer program or instructions in the memory 143 so that any one of the methods in the above embodiments is executed.

[0217] The processor 144 may include one or more central processing units (CPUs) or digital processing units, etc., for executing computer programs or instructions in the memory 143, so that any of the methods in the above embodiments is executed;

[0218] It should be noted that the specific connection medium between the memory 143 and the processor 144 is not limited in the embodiment of the present application. Fig.14 In the embodiment, the memory 143 and the processor 144 are connected via a bus 145, and the connection between other components is only for illustrative purposes and is not intended to be limiting. The bus 145 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.14 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0219] Based on the same inventive concept, the embodiment of the present application provides a computer-readable storage medium. When the instructions in the storage medium are executed by a processor, the processor is enabled to execute any one of the methods in the above embodiments. Since the principle of solving the problem by the above computer-readable storage medium is similar to a method for completing point cloud data, the implementation of the above computer-readable storage medium can refer to the implementation of the method, and the repeated parts will not be repeated.

[0220] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0221] The present application is described with reference to the flowchart and / or block diagram of the method, device (system), and computer program product according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the process and / or box in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0222] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0223] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0224] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for completing point cloud data, characterized in that: include: Obtain the original point cloud data containing the target to be completed; Based on the original point cloud data, at least one complete point cloud data matching the target to be completed is screened out from the complete point cloud data of multiple targets; Based on the at least one complete point cloud data, point cloud completion is performed on the missing shape of the target to be completed in the original point cloud data to obtain target point cloud data of the target to be completed.

2. The method according to claim 1, characterized in that The obtaining of original point cloud data containing the target to be completed includes: Capturing a first image containing the target to be completed by a first acquisition device of the robot body; Acquiring a second image including the target to be completed by a second acquisition device on the end effector of the robot; converting the first image into first point cloud data, and converting the second image into second point cloud data; The first point cloud data and the second point cloud data are superimposed to obtain the original point cloud data.

3. The method according to claim 1 or 2, characterized in that The step of selecting at least one complete point cloud data matching the target to be completed from the complete point cloud data of multiple targets based on the original point cloud data includes: Based on the task to be processed, target recognition is performed on the original point cloud data to obtain the target to be completed; Extracting local shape features of the target to be completed from the original point cloud data; Perform feature matching on the local shape features and the shape features of the complete point cloud data of the multiple targets to obtain respective matching rates; The complete point cloud data corresponding to at least one shape feature having a matching rate greater than a first matching rate threshold among the matching rates is determined as the at least one complete point cloud data.

4. The method according to claim 3, characterized in that The step of performing point cloud completion on the missing shape of the target to be completed in the original point cloud data based on the at least one complete point cloud data to obtain target point cloud data of the target to be completed includes: Extracting a priori shape features of the at least one complete point cloud data; Based on the inference rule set, the local shape feature is compared with the extracted prior shape feature to infer the missing shape feature of the target to be completed, wherein the inference rule set is used to ensure that the inferred missing shape feature is the shape feature of the target to be completed; Based on the local shape features and the missing shape features of the target to be completed, point cloud completion is performed on the missing shape of the target to be completed in the original point cloud data to obtain the target point cloud data.

5. The method according to claim 3, characterized in that The shape feature and complete point cloud data of each target are stored in the form of key-value pairs, wherein the key in each key-value pair represents the shape feature of the target, and the value represents the complete point cloud data of the target; The step of determining the complete point cloud data corresponding to at least one shape feature having a matching rate greater than a first matching rate threshold value as the at least one complete point cloud data comprises: Obtaining a key corresponding to at least one shape feature whose matching rates are greater than the first matching rate threshold value; Based on the acquired key and multiple key-value pairs, complete point cloud data of the values ​​corresponding to the acquired keys are obtained.

6. The method according to claim 5, characterized in that The multiple key-value pairs are obtained in the following manner: Based on multiple training samples, at least one round of training is performed on the complete shape encoder to be trained until the convergence condition is met, and the corresponding relationships between the shape features of multiple targets and the complete point cloud data are respectively stored in the form of key-value pairs to obtain the multiple key-value pairs, each training sample includes the complete point cloud data of the training target and each part of the point cloud data, and the belonging relationship between the each part of the point cloud data and the complete point cloud data of the training target; wherein, in each round of training: Inputting each portion of point cloud data included in each training sample into the complete shape encoder to be trained, and using the complete shape encoder to extract features of the portion of point cloud data; Based on the features, the partial point cloud data are classified to obtain the complete point cloud data to which the partial point cloud data belongs; Based on the obtained complete point cloud data to which the partial point cloud data belongs and various attribution relationships, a loss value is determined, and based on the loss value, the parameters of the complete shape encoder to be trained are adjusted.

7. The method according to claim 5, characterized in that Each key-value pair includes a generation time; the multiple key-value pairs are updated in the following manner: In response to the update instruction, selecting each training point cloud data corresponding to the update instruction from the acquired multiple original point cloud data and the target point cloud data; Extracting shape features of each training point cloud data respectively, and comparing each extracted shape feature with the shape features in the multiple key-value pairs respectively; If the matching rate between the shape feature of the first training point cloud data and the shape feature in any key-value pair is greater than a second matching rate threshold, replacing any key-value pair with the key-value pair consisting of the shape feature of the first training point cloud data and the first training point cloud data; If the matching rate between the shape features of the second training point cloud data and the shape features in the multiple key-value pairs is less than a third matching rate threshold, the shape features of the second training point cloud data and the second training point cloud data are stored in the form of key-value pairs.

8. A data completion device for point cloud data, characterized in that: include: An acquisition module is used to acquire the original point cloud data containing the target to be completed; A screening module, configured to screen out at least one complete point cloud data matching the target to be completed from the complete point cloud data of multiple targets based on the original point cloud data; The point cloud completion module is used to complete the missing shape of the target to be completed in the original point cloud data based on the at least one complete point cloud data, so as to obtain the target point cloud data of the target to be completed.

9. A robot, characterized in that: include: A first acquisition device, deployed on the robot body; A second acquisition device, deployed on the robot end effector, includes a depth camera with an inertial measurement unit IMU; Memory, used to store computer programs or instructions; A processor is used to execute the computer program or instructions in the memory so that the method according to any one of claims 1 to 7 is executed.

10. A computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor, the processor is enabled to execute the method according to any one of claims 1 to 7.

Images