Mechanical arm grabbing method and device based on point cloud completion

Abstract

The embodiment of the invention provides a mechanical arm grabbing method and device based on point cloud completion. Relates to the field of robot grabbing. The method comprises the following steps: adjusting a current shooting angle, and obtaining point cloud data of a to-be-grabbed object at the current shooting angle; registering and combining the point cloud data with previously obtained point cloud data at other shooting angles; inputting the merged point cloud data into a point cloud completion network to generate a complete geometric structure of the to-be-grabbed object; according to the generated complete geometric structure, utilizing a grabbing posture detection method to generate a grabbing posture; the mechanical arm is controlled to grab the to-be-grabbed object in the grabbing posture through a motion planning method; and if grabbing fails, the grabbing process is repeated until grabbing succeeds. In this way, dependence on multi-view-angle or complex sensor arrangement can be greatly reduced, the grabbing posture is continuously optimized through a dynamic updating mechanism, and the grabbing success rate and robustness and the applicability and practicability of the mechanical arm in the complex environment are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of robot grasping, and in particular to a robot arm grasping method and device based on point cloud completion. Background Art

[0002] Robot grasping has many applications in daily life. For example, industrial sorting robots can sort items. Robot grasping posture estimation is particularly important in the field of robot grasping. Generally, the robot grasping posture is estimated by reconstructing the 3D scene. In actual scenes, there is usually only incomplete geometric information from a single or a few viewpoints, which will result in a low grasping success rate. There are currently two solutions to this problem. One solution is to obtain a complete 3D object scene. This method requires obtaining a feasible camera path around the object, which is very time-consuming; the other solution is to place multiple sensors around the object of interest. This method not only requires fine calibration of each sensor, but also does not have the advantage of being cost-effective; therefore, it is very necessary for robots to achieve intelligent grasping by predicting the missing geometric structure from partial point cloud data obtained from a single or a few viewpoints. Summary of the invention

[0003] The present invention provides a robot arm grasping method and device based on point cloud completion, equipment and storage medium.

[0004] According to a first aspect of the present disclosure, a robotic arm grasping method based on point cloud completion is provided. The method comprises: Adjust the current shooting angle to obtain the point cloud data of the object to be captured at the current shooting angle; Register and merge the point cloud data with the point cloud data obtained previously at other shooting angles; The merged point cloud data is input into the point cloud completion network to generate the complete geometric structure of the object to be grasped; Generate a grasping posture using a grasping posture detection method according to the generated complete geometric structure; The motion planning method is used to control the robot arm to adopt a grasping posture to grasp the object to be grasped; If the crawling fails, repeat the above crawling process until the crawling is successful.

[0005] In some implementations of the first aspect, adjusting the current shooting angle to obtain point cloud data of the object to be captured at the current shooting angle includes: Adjust the current shooting angle to obtain the RGB image and depth image of the object to be captured at the current shooting angle; Perform instance segmentation on the RGB image to obtain the pixel coordinates of the grasping area corresponding to the object to be grasped; Obtaining the depth value of the grasping area corresponding to the object to be grasped according to the depth image; The pixel coordinates and depth values ​​are converted into three-dimensional coordinates corresponding to the real scene, and the point cloud data of the object to be grasped at the current shooting angle is obtained according to the three-dimensional coordinates.

[0006] In some implementations of the first aspect, registering and merging the point cloud data with previously acquired point cloud data at other shooting angles includes: Through the rigid transformation method, the point cloud data at the current shooting angle and the point cloud data at other shooting angles previously acquired are respectively converted into point cloud data in a unified preset coordinate system, and the converted point cloud data are merged.

[0007] In some implementations of the first aspect, the merged point cloud data is input into a point cloud completion network to generate a complete geometric structure of the object to be grasped, including: Normalize the merged point cloud data and input it into the point cloud completion network; The local feature encoder is used to extract the local features of each point in the normalized point cloud data, and the local geometric structure is constructed for each point; According to the local geometric structure corresponding to each point, the global features are obtained through the Transformer Encoder and pooling layer; Concatenate the local features of each point with the global features to generate local-global features; Input local-global features into the decomposition network to obtain the missing structure; The detail optimization network is used to optimize the missing structure and obtain the complete geometric structure of the object to be grasped.

[0008] In some implementations of the first aspect, inputting the local-global features into a decomposition network to obtain a missing structure includes: Use the scoring function to score each point in the local-global feature and arrange the scores from high to low; A preset number of points with the highest scores are selected, and a geometric patch formed by the selected points is used as a repeated geometric structure; Perform geometric transformation on the repeated geometric structure to restore the missing structure corresponding to the repeated geometric structure.

[0009] In some implementations of the first aspect, optimizing the missing structure using a detail optimization network to obtain a complete geometric structure of the object to be grasped includes: Using a multi-layer perceptron to learn local-global features, we can obtain error points in the missing structure where the error between the corresponding point in the local-global features is greater than a preset error. The position of the error point in the repeated geometric structure is moved to the position of the corresponding target point, and the repeated geometric structure after the error point is moved is geometrically transformed to obtain the optimized missing structure corresponding to the repeated geometric structure; The optimized missing structure is integrated with the local geometric structure corresponding to each point to generate the complete geometric structure of the object to be grasped.

[0010] In some implementations of the first aspect, optimizing the missing structure using the detail optimization network to obtain the complete geometric structure of the object to be grasped further includes: A global optimization strategy is used to eliminate unnatural overlapping geometric blocks in point cloud areas without repeated geometric structures; According to the optimized missing structure and the local geometric structure corresponding to each point, combined with the eliminated overlapping geometric blocks, the complete geometric structure of the object to be grasped is generated.

[0011] According to a second aspect of the present disclosure, a robotic arm grasping device based on point cloud completion is provided. The device comprises: The point cloud data acquisition module is used to adjust the current shooting angle and acquire the point cloud data of the object to be captured at the current shooting angle; A point cloud data registration and merging module is used to register and merge the point cloud data with the point cloud data previously acquired at other shooting angles; The complete geometry generation module is used to input the merged point cloud data into the point cloud completion network to generate the complete geometry of the object to be grasped; A grasping posture generation module, used for generating a grasping posture using a grasping posture detection method according to the generated complete geometric structure; The grasping module is used to control the robot arm to grasp the object with a grasping posture by using the motion planning method; if the grasping fails, the above grasping process is repeated until the grasping is successful.

[0012] According to a third aspect of the present disclosure, an electronic device is provided. The electronic device includes: at least one processor; and a memory connected to the at least one processor in communication; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the method described above.

[0013] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute the method as described above.

[0014] In the present disclosure, the current shooting angle is adjusted to obtain the point cloud data of the object to be grasped at the current shooting angle; the point cloud data is registered and merged with the point cloud data previously acquired at other shooting angles; the merged point cloud data is input into the point cloud completion network to generate the complete geometric structure of the object to be grasped; the grasping posture is generated according to the generated complete geometric structure using the grasping posture detection method; the motion planning method is used to control the robot arm to adopt the grasping posture to grasp the object to be grasped; if the grasping fails, the above grasping process is repeated until the grasping is successful. In this way, the dependence on multi-viewing angles or complex sensor arrangements can be greatly reduced, and the grasping posture can be continuously optimized using a dynamic update mechanism to improve the success rate and robustness of the grasping, as well as the applicability and practicality of the robot arm in complex environments.

[0015] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, among which: Figure 1 A flowchart of a robotic arm grasping method based on point cloud completion provided by an embodiment of the present disclosure is shown; Figure 2 A structural diagram of a robotic arm grasping device based on point cloud completion provided by an embodiment of the present disclosure is shown; Figure 3 A block diagram of an exemplary electronic device capable of implementing an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0018] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0019] In response to the problems arising in the background technology, the embodiments of the present disclosure provide a robotic arm grasping method and device based on point cloud completion. Adjust the current shooting angle to obtain the point cloud data of the object to be grasped at the current shooting angle; align and merge the point cloud data with the point cloud data previously acquired at other shooting angles; input the merged point cloud data into the point cloud completion network to generate a complete geometric structure of the object to be grasped; generate a grasping posture using a grasping posture detection method based on the generated complete geometric structure; use a motion planning method to control the robotic arm to adopt a grasping posture to perform a grasping operation on the object to be grasped; if the grasping fails, repeat the above grasping process until the grasping is successful. In this way, the reliance on multiple viewing angles or complex sensor arrangements can be greatly reduced, and the grasping posture can be continuously optimized using a dynamic update mechanism to improve the success rate and robustness of the grasping, as well as the applicability and practicality of the robotic arm in complex environments.

[0020] The following is a detailed description of the point cloud completion-based robotic arm grasping method and device provided by the embodiments of the present disclosure through specific embodiments in conjunction with the accompanying drawings.

[0021] Figure 1 A flowchart of a robotic arm grasping method based on point cloud completion provided by an embodiment of the present disclosure is shown. The method 100 includes the following steps: S110, adjusting the current shooting angle to obtain point cloud data of the object to be captured at the current shooting angle.

[0022] In some embodiments, adjusting the current shooting angle to obtain point cloud data of the object to be captured at the current shooting angle includes: Adjust the current shooting angle to obtain the RGB image and depth image of the object to be captured at the current shooting angle; Perform instance segmentation on the RGB image to obtain the pixel coordinates of the grasping area corresponding to the object to be grasped; Obtaining the depth value of the grasping area corresponding to the object to be grasped according to the depth image; Convert pixel coordinates and depth values ​​into three-dimensional coordinates corresponding to the real scene, and obtain point cloud data of the object to be grasped at the current shooting angle according to the three-dimensional coordinates; Furthermore, an RGB image and a depth image of the object to be captured at the current shooting angle are obtained through an RGB-D camera.

[0023] In some embodiments, performing instance segmentation on an RGB image includes: Use instance segmentation models to perform instance segmentation on RGB images; instance segmentation models include Mask R-CNN, Polar Mask, and YOLACT; Furthermore, the instance segmentation model used in the present invention is YOLACT. The specific steps of using YOLACT to perform instance segmentation on RGB images are as follows: YOLACT divides the instance segmentation task into the prototype mask generation task and the mask coefficient generation task; According to the RGB image, a fully convolutional network is used to generate some prototype masks. These prototype masks have the size of the entire RGB image and are not related to any instance. On the basis of the existing target detection branch, YOLACT adds an additional branch, which is used to predict the mask coefficient of each detected object. The mask coefficient can be used to represent the corresponding detected object in the mask space in an encoded way; For each detected object, its corresponding prototype mask is matrix multiplied with the mask coefficient, and then the operation result is activated by the Sigmoid function to generate the final instance mask.

[0024] In some embodiments, converting the pixel coordinates and the depth value into three-dimensional coordinates corresponding to the real scene includes: The pixel coordinates are converted into three-dimensional coordinates corresponding to the real scene through the camera intrinsic and extrinsic matrix of the shooting device (such as an RGB-D camera).

[0025] S120: align and merge the point cloud data with the point cloud data previously acquired at other shooting angles.

[0026] In some embodiments, registering and merging the point cloud data with previously acquired point cloud data at other shooting angles includes: By using a rigid transformation method, the point cloud data at the current shooting angle and the point cloud data at other shooting angles previously acquired are respectively converted into point cloud data in a unified preset coordinate system, and the converted point cloud data are merged; Furthermore, rigid transformation includes translation transformation, rotation transformation and the like.

[0027] S130, inputting the merged point cloud data into a point cloud completion network to generate a complete geometric structure of the object to be grasped.

[0028] In some embodiments, the merged point cloud data is input into a point cloud completion network to generate a complete geometric structure of the object to be grasped, including: Normalize the merged point cloud data and input it into the point cloud completion network; The local feature encoder is used to extract the local features of each point in the normalized point cloud data, and the local geometric structure is constructed for each point; According to the local geometric structure corresponding to each point, the global features are obtained through the Transformer Encoder and pooling layer; Concatenate the local features of each point with the global features to generate local-global features; Input local-global features into the decomposition network to obtain the missing structure; Utilize the detail optimization network to optimize the missing structure and obtain the complete geometric structure of the object to be grasped; Furthermore, normalization includes scaling and centering.

[0029] In some embodiments, extracting the local features of each point in the normalized point cloud data using a local feature encoder includes: Using the local feature encoder, the local features of each point in the normalized point cloud data are extracted through two sets of multi-layer perceptrons and a pooling layer.

[0030] In some embodiments, constructing a local geometry for each point includes: The K-nearest neighbor algorithm based on point-by-point Euclidean distance is used to obtain the K neighboring points corresponding to each point, and the local geometric structure is constructed for each point based on these K neighboring points.

[0031] In some embodiments, the local-global features are input into a decomposition network to obtain a missing structure, including: Use the scoring function to score each point in the local-global feature and arrange the scores from high to low; A preset number of points with the highest scores are selected, and a geometric patch formed by the selected points is used as a repeated geometric structure; Perform geometric transformation on the repeated geometric structure to restore the missing structure corresponding to the repeated geometric structure.

[0032] In some embodiments, performing geometric transformation on the repeated geometric structure to restore the missing structure corresponding to the repeated geometric structure includes: Performing geometric transformation on the repeated geometric structure; wherein the geometric transformation includes at least one of the following: symmetry, rotation, translation, and uniform scaling; According to the completion formula, the missing structure corresponding to the repeated geometric structure is restored; the completion formula is as follows: g m = A m × p m + D m； g m Indicates m The missing structure corresponding to the repeated geometric structure, A m Indicates m The affine transformation matrix corresponding to the repeated geometric structure, D m Indicates m The translation matrix corresponding to the repeated geometric structure is p m Indicates m A repeating geometric structure.

[0033] In some embodiments, since the scales of the local geometric structures corresponding to each point are different, the number of points in different repeated geometric structures is arbitrary. In order to improve efficiency, the present invention sets the number of points in all repeated geometric structures to a fixed number of points, so that large-scale local geometric structures can become dense after multiple iterations, which not only ensures the accuracy of the acquired repeated geometric structures, but also improves the calculation efficiency.

[0034] In some embodiments, the missing structure is optimized using a detail optimization network to obtain a complete geometric structure of the object to be grasped, including: Using a multi-layer perceptron to learn local-global features, we can obtain error points in the missing structure where the error between the corresponding point in the local-global features is greater than a preset error. The position of the error point in the repeated geometric structure is moved to the position of the corresponding target point, and the repeated geometric structure after the error point is moved is geometrically transformed to obtain the optimized missing structure corresponding to the repeated geometric structure; The optimized missing structure is integrated with the local geometric structure corresponding to each point to generate the complete geometric structure of the object to be grasped.

[0035] In some embodiments, optimizing the missing structure using the detail optimization network to obtain the complete geometric structure of the object to be grasped further includes: A global optimization strategy is used to eliminate unnatural overlapping geometric blocks in point cloud areas without repeated geometric structures; According to the optimized missing structure and the local geometric structure corresponding to each point, combined with the eliminated overlapping geometric blocks, the complete geometric structure of the object to be grasped is generated.

[0036] In some embodiments, a global optimization strategy is used to eliminate unnatural overlapping geometric blocks in point cloud regions without repeated geometric structures, including: Using uniformity constraint function Optimize the positions of points in unnaturally overlapping geometric blocks in areas of the point cloud without repeated geometric structure; the uniformity constraint function is as follows: ; Indicates the first points, P represents the set of points in the overlapping geometric cells, Indicate point The K-nearest neighbors of Indicate point The first points; The uniformity constraint function constrains the distance between adjacent points in overlapping geometric blocks, which allows the points in the generated complete geometric structure to maintain a smooth transition.

[0037] In some embodiments, the point cloud completion network is used to generate the complete geometric structure of the object to be grasped, which not only ensures that the original geometric structure will not be destroyed during the completion process, but also greatly reduces the complexity of point cloud data acquisition, and accurate grasping can be achieved using only point cloud data from a single or a small number of shooting perspectives.

[0038] S140: Generate a grasping posture using a grasping posture detection method according to the generated complete geometric structure.

[0039] S150, using a motion planning method to control the robot arm to adopt a grasping posture to perform a grasping operation on the grasped object; if the grasping fails, repeat the above grasping process until the grasping is successful.

[0040] In some embodiments, the motion planning method includes RRT (Rapidly-exploring Random Trees) and PRM (Probabilistic Roadmap Methods).

[0041] According to the embodiments of the present disclosure, the current shooting angle is adjusted to obtain the point cloud data of the object to be grasped at the current shooting angle; the point cloud data is registered and merged with the point cloud data previously acquired at other shooting angles; the merged point cloud data is input into the point cloud completion network to generate the complete geometric structure of the object to be grasped; the grasping posture is generated according to the generated complete geometric structure using the grasping posture detection method; the motion planning method is used to control the robotic arm to adopt the grasping posture to grasp the object to be grasped; if the grasping fails, the above grasping process is repeated until the grasping is successful. In this way, the dependence on multi-viewing angles or complex sensor arrangements can be greatly reduced, and the grasping posture can be continuously optimized using a dynamic update mechanism to improve the success rate and robustness of the grasping, as well as the applicability and practicality of the robotic arm in complex environments.

[0042] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

[0043] The above is an introduction to the method embodiment. The following is a further explanation of the scheme disclosed in the present invention through an apparatus embodiment.

[0044] Figure 2 The structure diagram of a robot arm grasping device based on point cloud completion provided by an embodiment of the present disclosure is shown. The device 200 includes: The point cloud data acquisition module 210 is used to adjust the current shooting angle and acquire the point cloud data of the object to be captured at the current shooting angle.

[0045] In some embodiments, the point cloud data acquisition module 210 is specifically used to: Adjust the current shooting angle to obtain the point cloud data of the object to be captured at the current shooting angle, including: Adjust the current shooting angle to obtain the RGB image and depth image of the object to be captured at the current shooting angle; Perform instance segmentation on the RGB image to obtain the pixel coordinates of the grasping area corresponding to the object to be grasped; Obtaining the depth value of the grasping area corresponding to the object to be grasped according to the depth image; The pixel coordinates and depth values ​​are converted into three-dimensional coordinates corresponding to the real scene, and the point cloud data of the object to be grasped at the current shooting angle is obtained according to the three-dimensional coordinates.

[0046] The point cloud data registration and merging module 220 is used to register and merge the point cloud data with the point cloud data previously acquired at other shooting angles.

[0047] In some embodiments, the point cloud data registration and merging module 220 is specifically used to: Register and merge the point cloud data with the previously acquired point cloud data from other shooting angles, including: Through the rigid transformation method, the point cloud data at the current shooting angle and the point cloud data at other shooting angles previously acquired are respectively converted into point cloud data in a unified preset coordinate system, and the converted point cloud data are merged.

[0048] The complete geometric structure generation module 230 is used to input the merged point cloud data into the point cloud completion network to generate the complete geometric structure of the object to be grasped.

[0049] In some embodiments, the complete geometry generation module 230 is specifically configured to: The merged point cloud data is input into the point cloud completion network to generate the complete geometric structure of the object to be grasped, including: Normalize the merged point cloud data and input it into the point cloud completion network; The local feature encoder is used to extract the local features of each point in the normalized point cloud data, and the local geometric structure is constructed for each point; According to the local geometric structure corresponding to each point, the global features are obtained through the Transformer Encoder and pooling layer; Concatenate the local features of each point with the global features to generate local-global features; Input local-global features into the decomposition network to obtain the missing structure; The detail optimization network is used to optimize the missing structure and obtain the complete geometric structure of the object to be grasped.

[0050] In some embodiments, the complete geometry generation module 230 is further configured to: The local-global features are fed into the decomposition network to obtain the missing structure, including: Use the scoring function to score each point in the local-global feature and arrange the scores from high to low; A preset number of points with the highest scores are selected, and a geometric patch formed by the selected points is used as a repeated geometric structure; Perform geometric transformation on the repeated geometric structure to restore the missing structure corresponding to the repeated geometric structure.

[0051] In some embodiments, the complete geometry generation module 230 is further configured to: The detail optimization network is used to optimize the missing structure and obtain the complete geometric structure of the object to be grasped, including: Using a multi-layer perceptron to learn local-global features, we can obtain error points in the missing structure where the error between the corresponding point in the local-global features is greater than a preset error. The position of the error point in the repeated geometric structure is moved to the position of the corresponding target point, and the repeated geometric structure after the error point is moved is geometrically transformed to obtain the optimized missing structure corresponding to the repeated geometric structure; The optimized missing structure is integrated with the local geometric structure corresponding to each point to generate the complete geometric structure of the object to be grasped.

[0052] In some embodiments, the complete geometry generation module 230 is further configured to: Utilize the detail optimization network to optimize the missing structure and obtain the complete geometric structure of the object to be grasped, including: A global optimization strategy is used to eliminate unnatural overlapping geometric blocks in point cloud areas without repeated geometric structures; According to the optimized missing structure and the local geometric structure corresponding to each point, combined with the eliminated overlapping geometric blocks, the complete geometric structure of the object to be grasped is generated.

[0053] The grasping posture generating module 240 is used to generate a grasping posture using a grasping posture detecting method according to the generated complete geometric structure.

[0054] The grasping module 250 is used to control the robot arm to grasp the object with the grasping posture by using the motion planning method; if the grasping fails, the above grasping process is repeated until the grasping is successful.

[0055] Understandably, Figure 2 Each module / unit in the illustrated device 200 has the function of implementing each step in the method 100 provided in the embodiment of the present disclosure and can achieve its corresponding technical effect. For the sake of brevity, they will not be described in detail here.

[0056] Figure 3 A block diagram of an exemplary electronic device capable of implementing an embodiment of the present disclosure is shown. The electronic device 300 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device 300 may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0057] like Figure 3 As shown, the electronic device 300 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 302 or a computer program loaded from a storage unit 308 to a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 can also be stored. The computing unit 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An I / O interface 305 is also connected to the bus 304.

[0058] A number of components in the electronic device 300 are connected to the I / O interface 305, including: an input unit 306, such as a keyboard, a mouse, etc.; an output unit 307, such as various types of displays, speakers, etc.; a storage unit 308, such as a disk, an optical disk, etc.; and a communication unit 309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 309 allows the electronic device 300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0059] The computing unit 301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 301 performs the various methods and processes described above, such as method 100. For example, in some embodiments, the method 100 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 300 via the ROM 302 and / or the communication unit 309. When the computer program is loaded into the RAM 303 and executed by the computing unit 301, one or more steps of the method 100 described above may be performed. Alternatively, in other embodiments, the computing unit 301 may be configured to perform the method 100 in any other appropriate manner (e.g., by means of firmware).

[0060] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0061] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0062] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0063] It should be noted that the present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute method 100 and achieve the corresponding technical effect achieved by executing the method in the embodiment of the present disclosure. For the sake of concise description, they are not repeated here.

[0064] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0065] The systems and techniques described herein can be implemented in a computing system that includes a backend component (e.g., as a data server), or a computing system that includes a middleware component (e.g., an application server), or a computing system that includes a frontend component (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with the systems and techniques described herein). The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0066] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0067] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0068] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A robotic arm grasping method based on point cloud completion, It is characterized in that include: Adjust the current shooting angle to obtain the point cloud data of the object to be captured at the current shooting angle; Registering and merging the point cloud data with the point cloud data previously acquired at other shooting angles; The merged point cloud data is input into the point cloud completion network to generate the complete geometric structure of the object to be grasped; Generate a grasping posture using a grasping posture detection method according to the generated complete geometric structure; Using a motion planning method to control the robot arm to adopt the grasping posture to perform a grasping operation on the grasped object; If the crawling fails, repeat the above crawling process until the crawling is successful.

2. The method according to claim 1, It is characterized in that The step of adjusting the current shooting angle to obtain point cloud data of the object to be captured at the current shooting angle includes: Adjust the current shooting angle to obtain the RGB image and depth image of the object to be captured at the current shooting angle; Perform instance segmentation on the RGB image to obtain pixel coordinates of a capture area corresponding to the object to be captured; Acquire a depth value of a capture area corresponding to the object to be captured according to the depth image; The pixel coordinates and depth values ​​are converted into three-dimensional coordinates corresponding to the real scene, and point cloud data of the object to be captured at the current shooting angle is obtained according to the three-dimensional coordinates.

3. The method according to claim 1, It is characterized in that The registering and merging the point cloud data with the point cloud data previously acquired at other shooting angles includes: Through the rigid transformation method, the point cloud data at the current shooting angle and the point cloud data at other shooting angles previously acquired are respectively converted into point cloud data in a unified preset coordinate system, and the converted point cloud data are merged.

4. The method according to claim 1, It is characterized in that The merged point cloud data is input into the point cloud completion network to generate the complete geometric structure of the object to be grasped, including: Normalize the merged point cloud data and input it into the point cloud completion network; The local feature encoder is used to extract the local features of each point in the normalized point cloud data, and the local geometric structure is constructed for each point; According to the local geometric structure corresponding to each point, the global features are obtained through the Transformer Encoder and pooling layer; Concatenate the local features of each point with the global features to generate local-global features; Input local-global features into the decomposition network to obtain the missing structure; The missing structure is optimized using a detail optimization network to obtain the complete geometric structure of the object to be grasped.

5. The method according to claim 4, It is characterized in that The step of inputting the local-global features into the decomposition network to obtain the missing structure includes: Use the scoring function to score each point in the local-global feature and arrange the scores from high to low; A preset number of points with the highest scores are selected, and a geometric patch formed by the selected points is used as a repeated geometric structure; Performing geometric transformation on the repeated geometric structure to restore the missing structure corresponding to the repeated geometric structure.

6. The method according to claim 5, It is characterized in that The step of optimizing the missing structure using a detail optimization network to obtain a complete geometric structure of the object to be grasped includes: Using a multi-layer perceptron to learn local-global features, we can obtain error points in the missing structure where the error between the corresponding point in the local-global features is greater than a preset error. The position of the error point in the repeated geometric structure is moved to the position of the corresponding target point, and the repeated geometric structure after the error point is moved is geometrically transformed to obtain the optimized missing structure corresponding to the repeated geometric structure; The optimized missing structure is integrated with the local geometric structure corresponding to each point to generate the complete geometric structure of the object to be grasped.

7. The method according to claim 6, It is characterized in that The optimizing the missing structure by using the detail optimization network to obtain the complete geometric structure of the object to be grasped also includes: A global optimization strategy is used to eliminate unnatural overlapping geometric blocks in point cloud areas without repeated geometric structures; According to the optimized missing structure and the local geometric structure corresponding to each point, combined with the eliminated overlapping geometric blocks, the complete geometric structure of the object to be grasped is generated.

8. A robotic arm grasping device based on point cloud completion, It is characterized in that include: The point cloud data acquisition module is used to adjust the current shooting angle and acquire the point cloud data of the object to be captured at the current shooting angle; A point cloud data registration and merging module, used to register and merge the point cloud data with point cloud data previously acquired at other shooting angles; The complete geometry generation module is used to input the merged point cloud data into the point cloud completion network to generate the complete geometry of the object to be grasped; A grasping posture generation module, used for generating a grasping posture using a grasping posture detection method according to the generated complete geometric structure; The grasping module is used to control the robot arm to grasp the object by adopting the grasping posture using the motion planning method; if the grasping fails, the above grasping process is repeated until the grasping is successful.

9. An electronic device, It is characterized in that include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 7.

Images