A method, apparatus, terminal device, and storage medium for grasping objects.
By acquiring single-view point cloud images of the target object, predicting and selecting the optimal placement pose, the problem of the robot being unable to place the target object in a single placement is solved, and accurate object grasping and placement is achieved under incomplete image conditions.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SOUTHERN UNIVERSITY OF SCIENCE AND TECHNOLOGY
- Filing Date
- 2025-04-27
- Publication Date
- 2026-07-17
AI Technical Summary
When moving objects, robots often cannot grasp and place objects in the target posture in one go due to workspace and robot structure limitations. They need to grasp and place multiple times. Moreover, existing technologies rely on complete images of the object to accurately determine the placement posture and cannot handle incomplete images.
By acquiring a single-view point cloud image of the target object, multiple candidate placement poses are predicted, and the optimal placement pose is selected based on the differences in point cloud information corresponding to the candidate poses. The robot arm is then controlled to grasp and place the target object, thus achieving accurate placement.
Even without a complete image of the object, it can accurately determine and achieve the optimal placement pose of the object, reduce grasping errors, and improve the accuracy of object movement.
Smart Images
Figure CN120244970B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of robot control technology, and in particular relates to an object grasping method, device, terminal equipment and storage medium. Background Technology
[0002] When a robot performs a task of moving an object, it first needs to grasp the object and then place it in the target position with the desired orientation. However, due to limitations such as workspace and robot structure, the robot often cannot place the object in the target orientation in a single grasping and placing operation. It often requires multiple grasping and placing operations to complete the movement. Therefore, how to more accurately determine the object's orientation after it has grasped it is a problem that needs to be solved. Summary of the Invention
[0003] This application provides an object grasping method, apparatus, terminal device, and storage medium, which can more accurately determine the placement posture of the object after the robot grasps it.
[0004] In a first aspect, embodiments of this application provide an object grasping method, including:
[0005] During the process of the robotic arm moving the target object to the target pose, if it is necessary to grasp the target object for the i-th time, the i-th point cloud image of the target object collected at the current time is obtained, wherein the target object is in the i-th placement pose at the current time, i≥1;
[0006] If the object structure in the i-th point cloud image does not match the preset object structure of the target object, multiple candidate placement poses are generated based on the i-th point cloud image. The candidate placement pose is the predicted pose in which the robot can place the target object after the robot grasps the target object for the i-th time.
[0007] Based on the difference between the first point cloud information corresponding to each candidate placement pose and the second point cloud information of the target object in the i-th point cloud image, the optimal placement pose is selected from multiple candidate placement poses, wherein the first point cloud information is the point cloud information of the target object that can be collected when the target object is in the candidate placement pose.
[0008] Based on the optimal placement pose, the robotic arm is controlled to perform the i-th grasping and placement operation, placing the target object in the optimal placement pose, where the optimal placement position is the (i+1)-th placement pose of the target object.
[0009] In one possible implementation of the first aspect, after acquiring the i-th point cloud image of the target object at the current moment, the method further includes:
[0010] If the object structure in the i-th point cloud image matches the preset object structure of the target object, the optimal placement pose is determined based on the i-th placement pose and the preset target pose of the target object.
[0011] In one possible implementation of the first aspect, acquiring the i-th point cloud image of the target object at the current moment includes:
[0012] The camera device is controlled to acquire the i-th image information of the target object, wherein the camera device is fixed in a preset area and the image information is a single-view RGB-D image;
[0013] Based on the i-th image information, generate the i-th point cloud image of the target object.
[0014] In one possible implementation of the first aspect, generating the i-th point cloud image of the target object based on the i-th image information includes:
[0015] Based on the i-th image information, generate the current point cloud image of the target object;
[0016] The i-1th point cloud image of the target object is obtained, wherein the i-1th point cloud image is obtained based on the i-1th image information collected, and the i-1th image information is the image information of the target object collected when the target object is in the i-1th placement pose;
[0017] The i-1th point cloud image is fused with the current point cloud image to obtain the i-th point cloud image of the target object.
[0018] In one possible implementation of the first aspect, the step of selecting the optimal placement pose from multiple candidate placement poses based on the difference between the first point cloud information corresponding to each candidate placement pose and the second point cloud information of the target object in the i-th point cloud image includes:
[0019] For each candidate placement pose, predict the first point cloud information of the target object when the target object is in the candidate placement pose;
[0020] Extract the second point cloud information of the target object from the i-th point cloud image;
[0021] Point cloud information that differs between the first feature information and the second feature information is filtered to obtain the difference feature information;
[0022] By filtering out identical point cloud information from the first and second point cloud information, overlapping feature information is obtained.
[0023] Based on the difference feature information and the overlap feature information, a score is calculated for each of the candidate placement poses;
[0024] The optimal placement pose is selected from the multiple candidate placement poses based on the score of each candidate placement pose.
[0025] In one possible implementation of the first aspect, calculating a score for each candidate placement pose based on the difference feature information and the overlap feature information includes:
[0026] Based on the difference feature information, calculate the information gain of the first point cloud information relative to the second point cloud information;
[0027] Based on the overlapping feature information, the information overlap degree between the first point cloud information and the second point cloud information is calculated;
[0028] Based on the information gain and the information overlap, a score is calculated for each candidate placement pose.
[0029] In one possible implementation of the first aspect, the first point cloud information of the target object acquired when predicting that the target object is in the candidate placement pose includes:
[0030] Based on the candidate placement pose, the virtual placement position of the virtual camera device relative to the target object is determined, so that the target object observed by the virtual camera device in the i-th placement pose is the same as the target object observed by the camera device in the candidate placement pose. The camera device is a device fixed in a preset area for capturing the target object at the current moment.
[0031] A virtual space is constructed, in which the target object is in the i-th placement pose, and the virtual camera device is set at the virtual placement position;
[0032] In the virtual space, the virtual camera device is simulated to emit ray signals toward the target object;
[0033] Based on the return signal of the ray signal, the first point cloud information of the target object is obtained when the target object is in the candidate placement pose.
[0034] Secondly, embodiments of this application provide an object grasping device, including:
[0035] The image acquisition module is used to acquire the i-th point cloud image of the target object at the current moment if the target object needs to be grasped for the i-th time during the process of the robot arm moving the target object to the target pose, wherein the target object is in the i-th placement pose at the current moment, i≥1;
[0036] The position prediction module is used to generate multiple candidate placement poses based on the i-th point cloud image if the i-th point cloud image does not include a complete image of the target object. The candidate placement poses are the predicted poses in which the robot can place the target object after the robot grasps the target object for the i-th time.
[0037] The optimal pose selection module is used to select the optimal placement pose from multiple candidate placement poses based on the difference between the first point cloud information corresponding to each candidate placement pose and the second point cloud information of the target object in the i-th point cloud image. The first point cloud information is the point cloud information of the target object that can be collected when the target object is in the candidate placement pose.
[0038] The control module is used to control the robotic arm to perform the i-th grasping and placement operation based on the optimal placement pose, and to place the target object in the optimal placement pose, wherein the optimal placement position is the (i+1)-th placement pose of the target object.
[0039] Thirdly, embodiments of this application provide a terminal device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the object grasping method described in any one of the first aspects above.
[0040] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the object grasping method described in any one of the first aspects above.
[0041] Fifthly, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute the object grasping method described in any of the first aspects above.
[0042] The beneficial effects of the first aspect of this application compared with the prior art are as follows: When the robotic arm performs a re-grabbing operation, if it is necessary to grasp the target object for the i-th time, the i-th point cloud image of the target object collected at the current moment is first acquired. If the object structure in the i-th point cloud image does not match the preset object structure, it indicates that there is a missing part in the point cloud image of the target object collected at the current time. Therefore, multiple candidate placement poses that the target object can present in the next time can be predicted based on the i-th point cloud image. Based on the difference between the first point cloud information of the target object that can be collected at each candidate placement pose and the second point cloud information in the i-th point cloud image, the optimal placement pose is selected from multiple candidate placement poses. Finally, the robotic arm is controlled to grasp and place the target object so that the target object presents the optimal placement pose. This application's object grasping does not depend on a complete image of the object. When a complete image of the object is not obtained, this application can first predict the candidate placement poses in which the object can be placed, and then obtain the optimal placement pose through screening. This allows for object grasping based on an incomplete image of the object, and also allows the object to be placed in the optimal pose after grasping, making object grasping more accurate.
[0043] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description
[0044] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0045] Figure 1 This is a schematic diagram of a scenario where an object cannot be placed in the target pose using a single grasping action, according to an embodiment of this application.
[0046] Figure 2 This is a schematic diagram of a scenario provided by an embodiment of this application, in which an object can be placed in a target pose by using multiple grasps;
[0047] Figure 3 This is a flowchart illustrating an embodiment of the object grasping method provided in this application;
[0048] Figure 4 This is a comparative schematic diagram of an incomplete image and a completed image of an object provided in an embodiment of this application;
[0049] Figure 5This is a schematic flowchart of a point cloud image acquisition method provided in an embodiment of this application;
[0050] Figure 6 This is a flowchart illustrating a method for acquiring point cloud images according to another embodiment of this application;
[0051] Figure 7 This is a flowchart illustrating a method for selecting the optimal placement pose according to an embodiment of this application;
[0052] Figure 8 This is a schematic diagram illustrating the constructed virtual space provided in one embodiment of this application;
[0053] Figure 9 This is a schematic diagram of the structure of an object grasping device provided in an embodiment of this application;
[0054] Figure 10 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation
[0055] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.
[0056] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."
[0057] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0058] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.
[0059] Object repositioning is a common task in robot operations. Object repositioning refers to the process by which a robot adjusts an object from its initial pose to a target pose.
[0060] Specifically, when a robotic arm needs to grasp and move an object, limitations in space and the arm's own performance often prevent it from placing the object in the target position with the desired posture after grasping it once. For example... Figure 1 As shown, due to the limitations of the robotic arm itself, an object cannot be flattened in a single grasp. In this case, the robotic arm needs to perform object repositioning operations, repeatedly grasping and placing the object to its target orientation and location. For example... Figure 2 As shown, the robotic arm flattens the object by grasping it twice.
[0061] Currently, when a robotic arm performs multiple grasping operations on an object, it relies on a complete image of the object. If the image is incomplete, it cannot calculate the object's orientation after grasping. Therefore, before grasping an object, it is necessary to acquire images of the object from multiple perspectives to obtain complete images from each angle. This method is relatively complex and cannot accurately move objects with complex structures. Without a complete image of the object, grasping and movement are impossible.
[0062] Therefore, this application proposes an object grasping method. This application utilizes images of objects captured from a single viewpoint to more accurately grasp and place objects when the image cannot fully reproduce the object's structure. Specifically, when the captured object's structure is incomplete, multiple placement poses for the object after grasping can be predicted first. Then, the optimal placement pose is selected from these poses, and finally, the grasped object is placed in the optimal pose. Through multiple grasping and placement processes, the effect of grasping and placing objects more accurately is achieved even without a complete image of the object.
[0063] The following combination Figure 3 The object grasping method of the embodiments of this application will be described in detail.
[0064] Figure 3 A schematic flowchart of the object grasping method provided in this application is shown, with reference to... Figure 3 The method is described in detail below:
[0065] S101, during the process of the robotic arm moving the target object to the target pose, if it is necessary to grasp the target object for the i-th time, the i-th point cloud image of the target object collected at the current time is obtained, wherein the target object is in the i-th placement pose at the current time, i≥1.
[0066] In this embodiment, the i-th point cloud image is the point cloud image of the target object at the current moment, acquired by a camera device fixed in a preset area. The point cloud image of the object obtained by the camera device fixed in the preset area is a point cloud image from a single viewpoint.
[0067] Placement pose includes position and posture.
[0068] When i=1, the i-th placement pose is the initial pose of the target object. At this time, the target object needs to be grasped for the first time, and the point cloud image of the target object acquired at this time is the first point cloud image.
[0069] When i ≠ 1, the current placement pose of the target object is the pose at which it was placed after the last grasp. For example, if i = 3, the target object is currently in its third placement pose, which is the pose at which it was placed after the second grasp. In this case, a third grasp is needed, and the point cloud image of the target object acquired at the current moment is the third point cloud image.
[0070] S102, if the object structure in the i-th point cloud image does not match the preset object structure of the target object, generate multiple candidate placement poses based on the i-th point cloud image, wherein the candidate placement pose is the predicted posture in which the robot can place the target object after the robot grasps the target object for the i-th time.
[0071] In this embodiment, the object structure of the target object is pre-stored, and the stored object structure represents the complete structure of the object. If the object structure in the i-th point cloud image does not match the pre-defined object structure of the target object, it indicates that the point cloud image of the object acquired at this time cannot completely represent the object structure. For example... Figure 4 As shown, Figure 4 Only the last image is a complete image of the object. Figure 4 The first three images in the middle cannot fully represent the object structure; Figure 4 The last image in the image can be the object structure of the preset target object.
[0072] Since the structure of the target object is missing in the i-th point cloud image, the pose of the target object after this grasp cannot be determined based on the target pose. However, the pose in which the robot can place the target object after grasping it can be predicted based on the existing structure of the target object in the i-th point cloud image. In this application, the predicted pose in which the target object can be placed is recorded as the candidate placement pose.
[0073] In one approach, the i-th point cloud image is input into a trained convolutional neural network to obtain multiple candidate placement poses.
[0074] In another approach, based on the i-th point cloud image, the performance parameters of the robotic arm, etc., we search for poses that the robotic arm can achieve and that the target object can stably maintain after placement, thus obtaining multiple candidate placement poses.
[0075] S103, based on the difference between the first point cloud information corresponding to each candidate placement pose and the second point cloud information of the target object in the i-th point cloud image, the optimal placement pose is selected from multiple candidate placement poses, wherein the first point cloud information is the point cloud information of the target object that can be collected when the target object is in the candidate placement pose.
[0076] In this embodiment, it is first predicted that if the target object is placed according to the candidate placement pose, the point cloud information of the target object collected by the camera device fixed in the preset area is recorded as the first point cloud information. In order to collect a more complete image of the target object next time, the optimal placement pose is selected according to the difference between the first point cloud information corresponding to the candidate placement pose and the second point cloud information of the target object in the i-th point cloud image.
[0077] In one approach, the proportions of missing, newly added, and overlapping point clouds in the first point cloud information relative to the second point cloud information are calculated. Based on these proportions, the optimal placement pose is selected. The missing point cloud proportion is the ratio of the number of point clouds in the second point cloud information that have no corresponding counterparts in the first point cloud information to the total number of point clouds in the second point cloud information. The newly added point cloud proportion is the ratio of the number of point clouds in the first point cloud information that have no corresponding counterparts in the second point cloud information to the total number of point clouds in the first point cloud information. The overlapping point cloud proportion is the ratio of the number of point clouds in the first point cloud information that have corresponding counterparts in the second point cloud information to the total number of point clouds in the first point cloud information.
[0078] Specifically, the first weight corresponding to the missing point cloud proportion, the second weight corresponding to the newly added point cloud proportion, and the third weight corresponding to the overlapping point cloud proportion are obtained. A weighted algorithm is then used to obtain the score of the candidate placement pose. The candidate placement pose with the highest score is selected as the optimal placement pose.
[0079] S104, based on the optimal placement pose, control the robotic arm to perform the i-th grasping and placement operation, and place the target object in the optimal placement pose, wherein the optimal placement position is the (i+1)-th placement pose of the target object.
[0080] In this embodiment, the candidate placement pose may further include the translation distance and rotation angle that the robot arm should move from the current position of the target object to the candidate placement pose. Therefore, the optimal placement position includes the translation distance and rotation angle that the robot arm should move during this grasping action. The robot arm is controlled to perform its actions according to the translation distance and rotation angle to make the target object reach the optimal placement pose.
[0081] Alternatively, based on the optimal placement pose, the i-th placement pose of the target object, and the performance parameters of the robotic arm, the translation distance and rotation angle that the robotic arm should move in this grasping operation can be determined. Then, the robotic arm's actions can be controlled according to the determined translation distance and rotation angle.
[0082] In this application, when the robotic arm performs a re-grasping, if it is necessary to grasp the target object for the i-th time, it first acquires the i-th point cloud image of the target object at the current moment. If the object structure in the i-th point cloud image does not match the preset object structure, it indicates that there is a missing part in the point cloud image of the target object currently acquired. Therefore, it can first predict multiple candidate placement poses that the target object can present next based on the i-th point cloud image. Then, based on the difference between the first point cloud information of the target object that can be acquired at each candidate placement pose and the second point cloud information in the i-th point cloud image, the optimal placement pose is selected from multiple candidate placement poses. Finally, the robotic arm is controlled to grasp and place the target object so that the target object presents the optimal placement pose. This application's object grasping does not depend on a complete image of the object. When a complete image of the object is not obtained, this application can first predict the candidate placement poses in which the object can be placed, and then obtain the optimal placement pose through screening. This allows for object grasping based on an incomplete image of the object, and also allows the object to be placed in the optimal pose after grasping, making object grasping more accurate.
[0083] Since this application is based on the point cloud image of the target object, and the RGB-D image includes the depth and color of the target object, the point cloud image of the target object can be determined based on the RGB-D image.
[0084] Therefore, as Figure 5 As shown, the implementation process of step S101 may include:
[0085] S1011, control the camera device to acquire the i-th image information of the target object, wherein the camera device is fixed in a preset area and the image information is a single-view RGB-D image.
[0086] In this embodiment, the camera device can be a depth camera or a radar system, etc. Since the camera device is fixed in the preset area and its position does not change, the target object captured by the camera device is a single-view image.
[0087] S1012, Generate the i-th point cloud image of the target object based on the i-th image information.
[0088] In this embodiment, the i-th image information is the i-th RGB-D image. The RGB-D image includes color visual information and depth information. Therefore, a point cloud image of the target object can be generated based on the i-th image information.
[0089] Since the image information of the target object acquired each time is a single-view image, in order to make the point cloud image more accurate, after acquiring the point cloud image of the target object at the current moment, the point cloud image at the current moment can be fused with the previous point cloud images to obtain a point cloud image containing more features of the target object, and thus obtain a complete point cloud image of the target object.
[0090] Specifically, such as Figure 6 As shown, the implementation process of step S1012 above may further include:
[0091] S201, Generate the current point cloud image of the target object based on the i-th image information.
[0092] S202, acquire the (i-1)th point cloud image of the target object, wherein the (i-1)th point cloud image is obtained based on the (i-1)th image information acquired, and the (i-1)th image information is the image information of the target object acquired when the target object is in the (i-1)th placement pose.
[0093] In this embodiment, when i=1, there is no (i-1)th point cloud image. Therefore, when i=1, the obtained (i-1)th point cloud image is 0.
[0094] S203, perform image fusion between the (i-1)th point cloud image and the current point cloud image to obtain the i-th point cloud image of the target object.
[0095] In this embodiment, when i=2, the first point cloud image of the target object is acquired, and the first point cloud image is fused with the currently acquired point cloud image to obtain the second point cloud image. The first point cloud image is the point cloud image acquired when the target object is in its initial pose.
[0096] When i=3, the second point cloud image of the target object is acquired, and the second point cloud image is fused with the currently acquired point cloud image to obtain the third point cloud image.
[0097] Following this pattern, after several captures, a complete point cloud image of the target object can be obtained, such as... Figure 4 The last image in the series is a complete point cloud image of the target object.
[0098] In this application, a complete point cloud image of the target object can be obtained by continuously fusing point cloud images. By grasping the target object based on increasingly complete images, the robotic arm can grasp the target object more and more accurately, reducing the error of the robotic arm in grasping the target object.
[0099] like Figure 7 As shown, in one possible implementation, step S103 may include:
[0100] S1031, for each of the candidate placement poses, predict the first point cloud information of the target object collected when the target object is in the candidate placement pose.
[0101] In one approach, a scenario is simulated where the target object is placed in a candidate placement pose, and then a camera is used to simulate capturing an image of the target object to obtain the first point cloud information of the target object.
[0102] In another approach, the first point cloud information is determined using the Truncated Signed Distance Function (TSDF) principle.
[0103] Specifically, in S11, based on the candidate placement pose, the virtual placement position of the virtual camera device relative to the target object is determined, so that the target object observed by the virtual camera device in the i-th placement pose is the same as the target object observed by the camera device in the candidate placement pose. The camera device is a device fixed in a preset area for capturing the target object at the current moment.
[0104] In this embodiment, the current pose of the target object is not changed; that is, the target object remains in the i-th placement pose. Changing the position of the virtual camera device and finding the location where the virtual camera device captures the image of the target object is equivalent to using the camera device to capture the image when the target object is set as a candidate placement pose, thus obtaining the virtual placement position of the virtual camera device.
[0105] S12, construct a virtual space, in which the target object is in the i-th placement pose, and the virtual camera device is set at the virtual placement position.
[0106] For example, such as Figure 8As shown, the target object is placed according to the i-th placement pose. A is a camera device fixed in a preset area. B is the virtual placement position of the virtual camera device determined based on the first candidate placement position. C is the virtual placement position of the virtual camera device determined based on the second candidate placement position. D is the virtual placement position of the virtual camera device determined based on the third candidate placement position. E is the virtual placement position of the virtual camera device determined based on the fourth candidate placement position.
[0107] S13, In the virtual space, the virtual camera device is simulated to emit ray signals toward the target object.
[0108] S14, based on the return signal of the ray signal, obtain the first point cloud information of the target object when the target object is in the candidate placement pose.
[0109] In this embodiment, if the ray signal can hit the target object, then the ray signal has a return signal. If the ray signal cannot hit the target object, then the ray signal does not have a return signal.
[0110] S1032, Extract the second point cloud information of the target object from the i-th point cloud image.
[0111] In this embodiment, the point cloud information includes data for each point cloud in the point cloud image, and the point cloud information may include geometric information, intensity information, and color information of each point cloud.
[0112] S1033, filter out the point cloud information that differs from the first point cloud information and the second point cloud information to obtain the difference feature information.
[0113] In this embodiment, the point cloud information that differs between the first point cloud information and the second point cloud information is the difference feature information.
[0114] Specifically, each point cloud in the first point cloud information is compared with each point cloud in the second information to obtain the point cloud information that differs between the first point cloud information and the second point cloud information.
[0115] Specifically, if point cloud A in the first point cloud information does not exist in the second point cloud information, then point cloud A is a difference point cloud. If point cloud B in the second point cloud information does not exist in the first point cloud information, then point cloud B is also a difference point cloud.
[0116] S1034, filter out the same point cloud information in the first point cloud information and the second point cloud information to obtain overlapping feature information.
[0117] In this embodiment, the same point cloud information in the first point cloud information and the second point cloud information is the overlapping feature information.
[0118] S1035, Based on the difference feature information and the overlap feature information, calculate the score for each of the candidate placement poses.
[0119] In this embodiment, the score of the candidate placement pose is calculated based on the number of difference point clouds in the difference feature information and the number of coincidence point clouds in the coincidence feature information.
[0120] In one approach, based on the number of differing point clouds, a preset interval containing that number of differing point clouds is queried, with each preset interval corresponding to a score. The score corresponding to the preset interval containing the number of differing point clouds is determined as the first score. Based on the number of overlapping point clouds, a preset interval containing the number of overlapping point clouds is queried, and the score corresponding to the preset interval containing the number of overlapping point clouds is determined as the second score. The sum of the first score and the second score is the score for the candidate placement pose.
[0121] In another approach, the scoring method may also include:
[0122] Based on the difference feature information, calculate the information gain of the first point cloud information relative to the second point cloud information; based on the overlap feature information, calculate the information overlap degree between the first point cloud information and the second point cloud information; based on the information gain and the information overlap degree, calculate the score for each candidate placement pose.
[0123] In this embodiment, information gain characterizes the newly added information in the first point cloud information compared to the second point cloud information, i.e., the features of the newly acquired target object. Information overlap characterizes the degree to which the point cloud in the second point cloud information is retained in the first point cloud information, i.e., the degree to which the features of the previously acquired target object are retained. The optimal placement position is determined jointly based on information gain and information overlap. This takes into account that after the target object is captured and placed this time, the point cloud information of the target object should be collected as much as possible in the next image acquisition of the target object, so that the constructed point cloud image of the target object is more complete.
[0124] Specifically, query the difference feature information to find the number of newly added point clouds in the first point cloud information. Divide the number of first point clouds by the total number of point clouds in the first point cloud information to obtain the information gain.
[0125] The query identifies the number of second point clouds that overlap with the first and second point cloud information, based on the overlapping feature information. The overlap degree is then calculated by multiplying this number by a preset parameter. Alternatively, the total number of point clouds in the first and second point cloud information can be calculated, and the overlap degree is obtained by dividing the second point cloud number by the total number.
[0126] The weights of information gain and information overlap are obtained, and a weighted algorithm is used to obtain the score of the candidate placement pose.
[0127] S1036, Based on the score of each of the candidate placement poses, select the optimal placement pose from the multiple candidate placement poses.
[0128] In this embodiment, the highest score is found among all scores, and the candidate placement pose corresponding to the highest score is determined as the optimal placement pose.
[0129] In this application, a score for each candidate placement pose is calculated by comparing the first point cloud information and the second point cloud information, and then the optimal placement pose is selected based on the score of each candidate placement pose. This application provides a method for selecting the optimal placement pose, so that the final determined optimal placement pose is more in line with the requirements.
[0130] The above describes a method for selecting the optimal placement pose when a complete point cloud image of the target object is not available, based on the difference between the first point cloud information corresponding to each candidate placement pose and the second point cloud information of the target object in the i-th point cloud image. If a complete point cloud image of the target object is available, the placement pose of the target object after grasping it can be determined based on the current pose and the target pose.
[0131] Specifically, if the object structure in the i-th point cloud image matches the preset object structure of the target object, the (i+1)-th placement pose of the target object is determined based on the i-th placement pose and the preset target pose of the target object.
[0132] In this embodiment, if the object structure in the i-th point cloud image matches the object structure of the preset target object, it indicates that the i-th point cloud image contains a complete point cloud image of the target object.
[0133] Based on the i-th point cloud image and the performance parameters of the robotic arm, it can be predicted whether the target object can be placed in the target pose after this grasping. If the target object can be placed in the target pose after this grasping, the target pose can be determined as the pose that the target object should present after this grasping. If the target object cannot be placed in the target pose after this grasping, a virtual model of the target object and the robotic arm can be established. The grasping process of the target object can be simulated through the virtual model to determine how many grasping attempts are needed to place the target object in the target pose, and thus determine the placement pose of the target object after each grasping.
[0134] Alternatively, the i-th point cloud image and the target pose of the target object can be input into the trained convolutional neural network to obtain the (i+1)-th placement pose.
[0135] The following describes another implementation of the method in this application.
[0136] S21, acquire the initial image of the target object (the first image information), and generate the first point cloud image based on the initial image.
[0137] S22, if the structure of the target object in the first point cloud image does not match the preset object structure (that is, the structure of the target object in the first point cloud image is incomplete), generate multiple candidate placement poses based on the information of the first image.
[0138] S23, based on the difference between the first point cloud information of the target object corresponding to the candidate placement pose and the second point cloud information in the first point cloud image, the optimal placement pose is selected from multiple candidate placement poses.
[0139] S24, control the robotic arm to perform the first grasp and placement, so that the target object is in the optimal placement pose (second placement pose).
[0140] S25: Acquire the second image information of the target object when it is in the second placement pose, and fuse the point cloud image generated from the second image information with the first point cloud image to generate the second point cloud image.
[0141] S26. If the structure of the target object in the second point cloud image does not match the preset object structure, repeat the above operations S22 to S25.
[0142] S27, if the structure of the target object in the acquired Nth point cloud image matches the preset object structure, determine the (N+1)th placement pose of the target object based on the Nth placement pose of the target object and the preset target pose of the target object. Control the robot arm to perform the Nth grasping and placement, so that the target object is in the (N+1)th placement pose, until the target object is moved to the target pose.
[0143] In this application, by fusing point cloud data of the target object and selecting the optimal placement pose, the structure of the target object is reconstructed, and a complete image of the target object is finally obtained, reducing errors in the object grasping process and making object grasping more accurate.
[0144] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0145] Corresponding to the object grasping method described in the above embodiments, Figure 9 A structural block diagram of the object grasping device provided in the embodiments of this application is shown. For ease of explanation, only the parts related to the embodiments of this application are shown.
[0146] Reference Figure 9The device 300 may include: an image acquisition module 310, a position prediction module 320, an optimal pose selection module 330, and a control module 340.
[0147] The image acquisition module 310 is used to acquire the i-th point cloud image of the target object at the current moment if the target object needs to be grasped for the i-th time during the process of the robot moving the target object to the target pose, wherein the target object is in the i-th placement pose at the current moment, i≥1.
[0148] The position prediction module 320 is used to generate multiple candidate placement poses based on the i-th point cloud image if the i-th point cloud image does not include a complete image of the target object. The candidate placement poses are the predicted poses in which the robot can place the target object after the robot grasps the target object for the i-th time.
[0149] The optimal pose selection module 330 is used to select the optimal placement pose from multiple candidate placement poses based on the difference between the first point cloud information corresponding to each candidate placement pose and the second point cloud information of the target object in the i-th point cloud image. The first point cloud information is the point cloud information of the target object that can be collected when the target object is in the candidate placement pose.
[0150] The control module 340 is used to control the robotic arm to perform the i-th grasping and placement operation based on the optimal placement pose, and to place the target object in the optimal placement pose, wherein the optimal placement position is the (i+1)-th placement pose of the target object.
[0151] In one possible implementation, the control module 340 can also be used for:
[0152] If the object structure in the i-th point cloud image matches the preset object structure of the target object, the (i+1)-th placement pose of the target object is determined based on the i-th placement pose and the preset target pose of the target object.
[0153] In one possible implementation, the image acquisition module 310 can specifically be used for:
[0154] The camera device is controlled to acquire the i-th image information of the target object, wherein the camera device is fixed in a preset area and the image information is a single-view RGB-D image;
[0155] Based on the i-th image information, generate the i-th point cloud image of the target object.
[0156] In one possible implementation, the image acquisition module 310 can specifically be used for:
[0157] Based on the i-th image information, generate the current point cloud image of the target object;
[0158] The i-1th point cloud image of the target object is obtained, wherein the i-1th point cloud image is obtained based on the i-1th image information collected, and the i-1th image information is the image information of the target object collected when the target object is in the i-1th placement pose;
[0159] The i-1th point cloud image is fused with the current point cloud image to obtain the i-th point cloud image of the target object.
[0160] In one possible implementation, the optimal pose selection module 330 can specifically be used for:
[0161] For each candidate placement pose, predict the first point cloud information of the target object when the target object is in the candidate placement pose;
[0162] Extract the second point cloud information of the target object from the i-th point cloud image;
[0163] Filter out point cloud information that differs from the first point cloud information and the second point cloud information to obtain difference feature information;
[0164] By filtering out identical point cloud information from the first and second point cloud information, overlapping feature information is obtained.
[0165] Based on the difference feature information and the overlap feature information, a score is calculated for each of the candidate placement poses;
[0166] The optimal placement pose is selected from the multiple candidate placement poses based on the score of each candidate placement pose.
[0167] In one possible implementation, the optimal pose selection module 330 can specifically be used for:
[0168] Based on the difference feature information, calculate the information gain of the first point cloud information relative to the second point cloud information;
[0169] Based on the overlapping feature information, the information overlap degree between the first point cloud information and the second point cloud information is calculated;
[0170] Based on the information gain and the information overlap, a score is calculated for each candidate placement pose.
[0171] In one possible implementation, the optimal pose selection module 330 can specifically be used for:
[0172] Based on the candidate placement pose, the virtual placement position of the virtual camera device relative to the target object is determined, so that the target object observed by the virtual camera device in the i-th placement pose is the same as the target object observed by the camera device in the candidate placement pose. The camera device is a device fixed in a preset area for capturing the target object at the current moment.
[0173] A virtual space is constructed, in which the target object is in the i-th placement pose, and the virtual camera device is set at the virtual placement position;
[0174] In the virtual space, the virtual camera device is simulated to emit ray signals toward the target object;
[0175] Based on the return signal of the ray signal, the first point cloud information of the target object is obtained when the target object is in the candidate placement pose.
[0176] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.
[0177] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0178] This application also provides a terminal device, see [link to relevant documentation] Figure 10 The terminal device 400 may include: at least one processor 410, a memory 420, and a computer program stored in the memory 420 and executable on the at least one processor 410. When the processor 410 executes the computer program, it implements the steps in any of the above method embodiments, for example... Figure 3Steps S101 to S104 in the illustrated embodiment. Alternatively, when the processor 410 executes the computer program, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 9 The functions of the image acquisition module 310 to the control module 340 are shown.
[0179] For example, a computer program may be divided into one or more modules / units, one or more of which are stored in memory 420 and executed by processor 410 to complete this application. The one or more modules / units may be a series of computer program segments capable of performing a specific function, which are used to describe the execution process of the computer program in terminal device 400.
[0180] Those skilled in the art will understand that Figure 10 This is merely an example of a terminal device and does not constitute a limitation on the terminal device. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, buses, etc.
[0181] The processor 410 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.
[0182] The memory 420 can be an internal storage unit of the terminal device or an external storage device, such as a plug-in hard drive, a smart media card (SMC), a secure digital card (SD), or a flash card. The memory 420 is used to store the computer program and other programs and data required by the terminal device. The memory 420 can also be used to temporarily store data that has been output or will be output.
[0183] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.
[0184] The object grasping method, apparatus, terminal device, and storage medium provided in this application can be applied to terminal devices such as computers, tablets, laptops, netbooks, and personal digital assistants (PDAs). This application does not impose any restrictions on the specific type of terminal device.
[0185] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0186] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0187] In the embodiments provided in this application, it should be understood that the disclosed terminal devices, apparatuses, and methods can be implemented in other ways. For example, the terminal device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, apparatuses, or units, and may be electrical, mechanical, or other forms.
[0188] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0189] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0190] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by one or more processors, it can implement the steps of the various method embodiments described above.
[0191] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by one or more processors, it can implement the steps of the various method embodiments described above.
[0192] Similarly, as a computer program product, when the computer program product is run on a terminal device, it enables the terminal device to implement the steps in the above-described method embodiments.
[0193] The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.
[0194] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.
Claims
1. A method for grasping an object, characterized in that, include: During the process of the robotic arm moving the target object to the target pose, if it is necessary to grasp the target object for the i-th time, the i-th point cloud image of the target object collected at the current time is obtained, wherein the target object is in the i-th placement pose at the current time, i≥1; If the object structure in the i-th point cloud image does not match the preset object structure of the target object, multiple candidate placement poses are generated based on the i-th point cloud image. The candidate placement pose is the predicted pose in which the robot can place the target object after the robot grasps the target object for the i-th time. Based on the difference between the first point cloud information corresponding to each candidate placement pose and the second point cloud information of the target object in the i-th point cloud image, the optimal placement pose is selected from multiple candidate placement poses, wherein the first point cloud information is the point cloud information of the target object that can be collected when the target object is in the candidate placement pose. Based on the optimal placement pose, the robotic arm is controlled to perform the i-th grasping and placement operation, placing the target object in the optimal placement pose, which is the (i+1)-th placement pose of the target object. The step of selecting the optimal placement pose from multiple candidate placement poses based on the difference between the first point cloud information corresponding to each candidate placement pose and the second point cloud information of the target object in the i-th point cloud image includes: For each candidate placement pose, predict the first point cloud information of the target object when the target object is in the candidate placement pose; Extract the second point cloud information of the target object from the i-th point cloud image; Filter out point cloud information that differs from the first point cloud information and the second point cloud information to obtain difference feature information; By filtering out identical point cloud information from the first and second point cloud information, overlapping feature information is obtained. Based on the difference feature information and the overlap feature information, a score is calculated for each of the candidate placement poses; The optimal placement pose is selected from the multiple candidate placement poses based on the score of each candidate placement pose.
2. The object grasping method as described in claim 1, characterized in that, After acquiring the i-th point cloud image of the target object at the current moment, the method further includes: If the object structure in the i-th point cloud image matches the preset object structure of the target object, the (i+1)-th placement pose of the target object is determined based on the i-th placement pose and the preset target pose of the target object.
3. The object grasping method as described in claim 1, characterized in that, The step of acquiring the i-th point cloud image of the target object at the current moment includes: The camera device is controlled to acquire the i-th image information of the target object, wherein the camera device is fixed in a preset area and the image information is a single-view RGB-D image; Based on the i-th image information, generate the i-th point cloud image of the target object.
4. The object grasping method as described in claim 3, characterized in that, The step of generating the i-th point cloud image of the target object based on the i-th image information includes: Based on the i-th image information, generate the current point cloud image of the target object; The i-1th point cloud image of the target object is obtained, wherein the i-1th point cloud image is obtained based on the i-1th image information collected, and the i-1th image information is the image information of the target object collected when the target object is in the i-1th placement pose; The i-1th point cloud image is fused with the current point cloud image to obtain the i-th point cloud image of the target object.
5. The object grasping method as described in claim 1, characterized in that, The step of calculating a score for each candidate placement pose based on the difference feature information and the overlap feature information includes: Based on the difference feature information, calculate the information gain of the first point cloud information relative to the second point cloud information; Based on the overlapping feature information, the information overlap degree between the first point cloud information and the second point cloud information is calculated; Based on the information gain and the information overlap, a score is calculated for each candidate placement pose.
6. The object grasping method as described in claim 1, characterized in that, The first point cloud information of the target object collected when predicting that the target object is in the candidate placement pose includes: Based on the candidate placement pose, the virtual placement position of the virtual camera device relative to the target object is determined, so that the target object observed by the virtual camera device in the i-th placement pose is the same as the target object observed by the camera device in the candidate placement pose. The camera device is a device fixed in a preset area for capturing the target object at the current moment. A virtual space is constructed, in which the target object is in the i-th placement pose, and the virtual camera device is set at the virtual placement position; In the virtual space, the virtual camera device is simulated to emit ray signals toward the target object; Based on the return signal of the ray signal, the first point cloud information of the target object is obtained when the target object is in the candidate placement pose.
7. An object grasping device, characterized in that, include: The image acquisition module is used to acquire the i-th point cloud image of the target object at the current moment if the target object needs to be grasped for the i-th time during the process of the robot arm moving the target object to the target pose, wherein the target object is in the i-th placement pose at the current moment, i≥1; The position prediction module is used to generate multiple candidate placement poses based on the i-th point cloud image if the i-th point cloud image does not include a complete image of the target object. The candidate placement poses are the predicted poses in which the robot can place the target object after the robot grasps the target object for the i-th time. The optimal pose selection module is used to: predict the first point cloud information of the target object when it is in the candidate placement pose for each candidate placement pose; extract the second point cloud information of the target object from the i-th point cloud image; filter point cloud information that differs between the first and second point cloud information to obtain difference feature information; filter point cloud information that is the same between the first and second point cloud information to obtain overlap feature information; calculate a score for each candidate placement pose based on the difference feature information and the overlap feature information; and select the optimal placement pose from multiple candidate placement poses according to the score of each candidate placement pose, wherein the first point cloud information is the point cloud information of the target object that can be collected when the target object is in the candidate placement pose. The control module is used to control the robotic arm to perform the i-th grasping and placement operation based on the optimal placement pose, and to place the target object in the optimal placement pose, wherein the optimal placement pose is the (i+1)-th placement pose of the target object.
8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the object grasping method as described in any one of claims 1 to 6.
9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the object grasping method as described in any one of claims 1 to 6.