Object grasping method, device, robot, readable storage medium and chip

Abstract

The application provides an object grasping method, device, robot, readable storage medium and chip, and the method comprises the following steps: acquiring scene information of an environment in which an object to be grasped is located; determining first point cloud information corresponding to the object to be grasped according to the scene information; determining a plurality of grasping postures for grasping the object to be grasped according to the first point cloud information; determining a target posture from the plurality of grasping postures, and controlling a mechanical arm to grasp the object to be grasped according to the target posture.

Description

Technical Field

[0001] This invention relates to the field of computers, and more specifically, to an object grasping method, an object grasping device, a robot, a readable storage medium, and a chip. Background Technology

[0002] In related technologies, when recognizing the grasping posture during object grasping, it is usually necessary to mark the grasping posture of the object in space. Using simulation annotation or manual annotation takes a long time and cannot be analyzed and grasped in a timely manner. Summary of the Invention

[0003] In order to solve or improve at least one of the above-mentioned technical problems, one object of the present invention is to provide an object grasping method.

[0004] Another object of the present invention is to provide an object grasping device.

[0005] Another object of the present invention is to provide a robot.

[0006] Another object of the present invention is to provide a readable storage medium.

[0007] Another object of the present invention is to provide a chip.

[0008] To achieve the above objectives, the first aspect of the present invention provides an object grasping method, comprising: acquiring scene information of the environment in which the object to be grasped is located; determining first point cloud information corresponding to the object to be grasped based on the scene information; determining multiple grasping postures for grasping the object to be grasped based on the first point cloud information; determining a target posture among the multiple grasping postures, and controlling a robotic arm to grasp the object to be grasped according to the target posture.

[0009] According to the object grasping method provided by the present invention, before grasping an object, it is necessary to acquire scene information of the environment in which the object to be grasped is located. This information may include the object's position, orientation, size, and the positions of surrounding objects. By processing the acquired scene information, first point cloud information corresponding to the object to be grasped can be obtained. It can be understood that a point cloud is a set of three-dimensional coordinate points, which can be used to represent information such as the shape and position of an object. Based on the first point cloud information, multiple possible grasping postures can be determined. These grasping postures may include information such as the position and orientation of the robotic arm's end effector, used to control the robotic arm to grasp. Finally, an optimal target posture is selected from the multiple possible grasping postures to control the robotic arm to grasp. This process typically considers factors such as the object's position, orientation, and relationship with surrounding objects. After determining the target posture, the movement of the robotic arm's end effector can be controlled to move the robotic arm to a suitable position and adjust parameters such as orientation, ultimately completing the grasping operation of the object to be grasped. Overall, this object grasping method utilizes robot vision and intelligent control technologies, combined with point cloud information processing and posture adjustment technologies, to achieve automated grasping of the object to be grasped.

[0010] It should be emphasized that this application directly uses the 3D geometric information of the object's point cloud to perform attitude calculation, without the need to annotate the object's grasping attitude in 3D space, which greatly saves manpower for annotation and material resources for simulation machines.

[0011] In addition, the technical solution provided by the present invention may also have the following additional technical features:

[0012] In some technical solutions, optionally, scene information of the environment in which the object to be grasped is located can be obtained, specifically including: obtaining scene information of the current environment through a 3D camera, the scene information including image information and depth information of the scene, the image information and depth information being the information after registration.

[0013] In this technical solution, the 3D camera can simultaneously acquire color images and depth images of objects in the scene. The color image provides appearance information of the objects, while the depth image provides distance information. This information can be used for subsequent point cloud processing and object pose estimation.

[0014] Furthermore, 3D cameras typically acquire depth information using technologies such as infrared or lasers, enabling them to quickly and accurately obtain information such as the position and shape of objects in a scene under different lighting conditions.

[0015] In summary, acquiring image and depth information of the current scene using a 3D camera can provide important reference for subsequent object grasping operations.

[0016] It's important to note that the image and depth information acquired using the 3D camera has already been registered, allowing for direct analysis. Registration aligns and calibrates the image and depth information to ensure they are in the same coordinate system. This allows us to find the corresponding depth value in the image using pixel coordinates, thus obtaining the object's accurate position in 3D space. Registration can be calculated and calibrated using the camera's intrinsic and extrinsic parameters, as well as the parameters of the depth sensor.

[0017] In some technical solutions, optionally, the first point cloud information corresponding to the object to be grasped is determined based on scene information, specifically including: determining the first point cloud information based on the intrinsic parameter matrix of the 3D camera, image information, and depth information.

[0018] In this technical solution, determining the first point cloud information requires first acquiring the camera's intrinsic parameter matrix and then determining the first point cloud information corresponding to the object to be grasped based on scene information. The intrinsic parameter matrix is ​​a parameter matrix describing the camera, containing information such as the camera's focal length, principal point coordinates, and distortion coefficients. It can convert the image coordinates acquired by the camera into real-world coordinates in the camera coordinate system. Using the image and depth information acquired by the camera, point cloud information in the scene can be obtained through a 3D point cloud reconstruction algorithm. Then, based on this point cloud information, techniques such as object recognition and target detection can be used to determine the position and pose of the object to be grasped in the point cloud. Different point cloud processing algorithms may be required for different object shapes and sizes, such as global matching algorithms and local model reconstruction algorithms. Finally, by mapping the image coordinates acquired by the camera onto the point cloud, the first point cloud information corresponding to the object to be grasped can be obtained. After obtaining the first point cloud information corresponding to the object to be grasped, some point cloud filtering operations may be required to remove irrelevant or noisy point clouds in order to further determine parameters such as the grasping pose and target pose. In summary, by obtaining the camera's intrinsic parameter matrix and determining the first point cloud information corresponding to the object to be grasped based on scene information, an important data foundation can be provided for subsequent point cloud processing and object grasping operations.

[0019] In some technical solutions, optionally, the first point cloud information is determined based on the intrinsic parameter matrix of the 3D camera, image information, and depth information, including: acquiring an object detection network model and a segmentation network model; identifying the image information through the object detection network model and the segmentation network model to determine the coordinate information of the first region; determining the depth information corresponding to the first region based on the coordinate information of the first region; and converting the depth information of the first region into the first point cloud information based on the intrinsic parameter matrix.

[0020] In this technical solution, when determining the first point cloud information, it is first necessary to acquire network models suitable for object detection and segmentation tasks. These models can be used to detect and locate objects to be grasped in images and provide bounding boxes or pixel-level segmentation results. The acquired object detection network model and segmentation network model are used to process the input image information. The object detection model can detect the position of the object to be grasped, while the segmentation model can segment the object pixels to form object regions. Based on the object detection or segmentation results, the coordinate information of the first region of the object to be grasped is obtained. The first region can be the bounding box or segmented region of the object. Based on the coordinate information of the first region, the depth value of the corresponding region is extracted from the depth information. The corresponding depth value is found in the depth map using pixel coordinates. Using the intrinsic parameter matrix of the 3D camera, the depth information of the first region is converted from the image coordinate system to 3D point coordinates in the camera coordinate system, and then the converted 3D point coordinates are combined to form the first point cloud information. Each point consists of its 3D coordinates and possible other attributes (such as color, normal, etc.).

[0021] Optionally, some technical solutions may further include: filtering the first point cloud information to determine the filtered second point cloud information, wherein determining the filtered second point cloud information includes: acquiring at least one granular geometric filtering structure, each granular geometric filtering structure including an anomaly filtering algorithm and a clustering algorithm; removing anomaly point clouds from the first point cloud information according to the anomaly filtering algorithm, and clustering the filtered first point cloud information according to the clustering algorithm; and determining the point cloud information of the largest cluster after clustering as the second point cloud information.

[0022] In this technical solution, the first point cloud information is filtered using a granular geometric filtering method. This method is based on parameters such as point cloud density and distance. By setting different filtering parameters and thresholds, point cloud filtering and clustering operations can be performed. For example, a coarse filter based on Euclidean distance or a Gaussian curvature filter based on a statistical model can be used. The acquired first point cloud information is input into the granular geometric filtering structure. Anomaly filtering algorithms are used to remove abnormal point clouds, while clustering algorithms are used to cluster the filtered point cloud information. Anomaly filtering algorithms can identify and remove point clouds that do not match the surrounding environment or the shape of the object, such as noisy point clouds and outliers. Clustering algorithms can divide the filtered point cloud information into different clusters, each representing an object or part of an object. After clustering, the largest cluster needs to be selected as the point cloud information for the object to be grasped. This process typically involves calculating parameters such as the number of points and density in different clusters to determine the largest cluster, which is then passed as the second point cloud information to the robotic arm controller for precise positioning and attitude adjustment operations. In summary, using a granular geometry filtering method to filter the first point cloud information can achieve a certain degree of automated processing, while improving the accuracy and efficiency of object grasping.

[0023] In some technical solutions, optionally, there are multiple granular geometric filter structures, and the radius distance between the point cloud and the surrounding neighborhood is different in each granular geometric filter structure.

[0024] In this technical solution, multiple granular geometric filtering structures are set up, and the radius distance between the point cloud and the surrounding neighborhood is limited in each structure. In order to better adapt to the characteristics of different object shapes and sizes, the flexibility and accuracy of point cloud processing are improved.

[0025] For example, larger objects can be clustered using a larger radius distance to ensure that the point cloud information covers the entire object surface; while smaller objects can be clustered using a smaller radius distance to avoid misclassifying multiple objects as the same. Different coarse filters and clustering algorithms can also be selected and adjusted based on the object's shape, surface texture, and other features to achieve more accurate point cloud processing and grasping operations.

[0026] In some technical solutions, the method may optionally include: determining the application order of multiple granular geometric filter structures when there are multiple granular geometric filter structures; arranging multiple granular geometric filter structures according to the application order, wherein the neighborhood distance of the granular geometric filter structure applied earlier is smaller than the neighborhood distance of the granular geometric filter structure applied later.

[0027] In this technical solution, determining the application order of multiple granular geometric filtering structures requires considering the neighborhood distance. The neighborhood distance of the first applied granular geometric filtering structure should be smaller, while the neighborhood distance of the later applied structure should be larger. This ensures that outliers in the nearest neighborhood are removed first, followed by clustering of the point cloud in the more distant neighborhood, resulting in more accurate and stable second point cloud information. Furthermore, applying the granular geometric filtering structure with a smaller neighborhood distance first can more meticulously remove outliers from the first point cloud information while retaining valid point clouds in the nearest neighborhood. Applying the granular geometric filtering structure with a larger neighborhood distance later can cluster the point clouds in the more distant neighborhood together, forming larger clusters, and removing unnecessary small clusters, thus obtaining the target point cloud.

[0028] In this sequence, filtering with smaller neighborhood distances first removes most outliers, followed by filtering with larger neighborhood distances, clustering the valid point cloud into the target point cloud. This processing order maximizes the preservation of the target point cloud while removing unnecessary noise and outliers, improving the quality and accuracy of the point cloud data.

[0029] Furthermore, the algorithm incorporates three different levels of anomaly filtering and clustering, resulting in a three-tiered granular geometric filtering structure. The algorithm uses the radius distance between the point cloud and its surrounding neighborhood as a tier division to remove points from more distant neighbors. The neighborhood distances for the first, second, and third levels of anomaly filtering gradually increase, ensuring effective removal of obvious anomalies in the initial filtering stage. The filtering effectiveness gradually decreases as the filtering progresses, ensuring the geometric structure of the target point cloud is effectively preserved. The filtered point cloud is then clustered, with the largest cluster selected as the target point cloud, further eliminating outlier regions generated during the filtering process.

[0030] In some technical solutions, optionally, multiple grasping postures for grasping the object to be grasped are determined based on the first point cloud information. Specifically, this includes: segmenting the second point cloud information based on a geometric segmentation structure to determine the target point cloud; and determining multiple grasping postures corresponding to a target point cloud whose number is less than a stopping threshold.

[0031] In this technical solution, when determining multiple grasping postures, the second point cloud information is primarily segmented based on a geometric segmentation structure. A geometric segmentation algorithm is used to divide the second point cloud information into multiple clusters, each representing a potential grasping target. The target point cloud is then determined by selecting points from the segmented clusters. This selection can be based on the point cloud's characteristics, shape, and size to ensure that the selected point cloud represents the object to be grasped. After obtaining the second point cloud information, it needs to be separated from the first point cloud information. This process typically employs techniques based on Euclidean distance or normal vectors for segmentation and extraction to determine the target point cloud. After determining the target point cloud, multiple possible grasping postures are generated based on the selected target point cloud. This can be achieved by combining a posture generation algorithm with the geometric features of the target point cloud. The generated grasping postures can include the position and posture information of the robotic arm, as well as the parameter settings of the grasping tool. Furthermore, the robotic arm controller can calculate multiple possible grasping postures. These postures can be calculated based on parameters such as the position, orientation, and size of the target point cloud to ensure that the robotic arm can safely and effectively grasp the object.

[0032] When calculating the grasping posture, it is important to set a stop threshold to avoid excessive calculation time. Once the number of calculated grasping postures reaches the stop threshold, the calculation can be stopped, and an optimal grasping posture can be selected for subsequent robotic arm control operations.

[0033] In summary, determining multiple grasping postures for grasping the object based on the second cloud information can provide important reference for subsequent robotic arm control and object grasping operations.

[0034] In some technical solutions, optionally, the second point cloud information is segmented based on a geometric segmentation structure to determine the target point cloud, including: obtaining the geometric segmentation structure and clustering structure; separating the second point cloud information according to the geometric segmentation structure to determine the separated point cloud information; and clustering the separated point cloud information according to the clustering structure to determine the largest cluster as the target point cloud.

[0035] In this technical solution, the determination of the target point cloud is mainly achieved through segmentation and clustering algorithms. Specifically, a geometric segmentation structure and a clustering structure are first obtained. The geometric segmentation structure can be a set of predefined geometric features or segmentation algorithms used to segment the point cloud; the clustering structure can be a set of predefined clustering algorithms or parameter settings used to cluster the point cloud. Then, using the selected geometric segmentation structure, the second point cloud information is segmented to obtain separate point cloud sets. The segmentation process can operate according to the geometric features of the point cloud (e.g., distance, normal vectors, etc.) to divide the point cloud into different segmentation clusters. For each separate point cloud set, the selected clustering algorithm or parameter settings are applied to cluster the point cloud. The clustering algorithm can group points based on their position, density, features, etc., forming different clusters. Based on the clustering results, the cluster with the largest number of points is determined as the target point cloud. By comparing the number of points in each cluster, the cluster with the largest number of points is selected as the point cloud set representing the target object.

[0036] Through the above steps, multiple point cloud sets can be separated from the second point cloud information, and the largest cluster can be determined as the target point cloud through clustering operations for subsequent capture operations.

[0037] In some technical solutions, optionally, determining multiple grasping postures corresponding to a target point cloud quantity less than a stopping threshold includes: determining the direction vectors of multiple target point clouds; determining the matrix corresponding to the direction vectors; determining the grasping position based on the centroid position of each target point cloud; determining the grasping posture corresponding to the target point cloud based on the grasping position and the matrix; determining the quantity of target point clouds; iteratively determining the target point clouds and grasping postures until the quantity of target point clouds is less than a stopping threshold, at which point the iteration stops and multiple grasping postures are determined.

[0038] In this technical solution, for each target point cloud, its normal vector or surface normal is calculated as the direction vector of that target point cloud. A rotation matrix is ​​calculated for the direction vector of each target point cloud, which describes the orientation of the robotic arm during grasping. The centroid position of each target point cloud, i.e., the average position of all points in the point cloud, is calculated as the grasping position of that target point cloud. The grasping position and the corresponding rotation matrix are combined to determine the posture of the robotic arm during grasping, including position and orientation. The number of target point clouds is counted, and it is determined whether it is less than a stopping threshold. If the number of target point clouds is greater than the stopping threshold, the iteration continues for the next round of point cloud segmentation and grasping posture determination, repeating the above steps. If the number of target point clouds is less than the stopping threshold, the iteration stops, and multiple grasping postures are determined.

[0039] By following the steps above, the number of target point clouds and their corresponding grasping postures can be determined iteratively until the number of target point clouds is less than the stopping threshold, thereby obtaining multiple grasping postures that meet the requirements.

[0040] In some technical solutions, optionally, determining the direction vector of the target point cloud includes: determining the direction vectors of multiple surface normals in the target point cloud; determining the average value of the direction vectors of multiple surface normals corresponding to a target point cloud, and using the average value as the direction vector of the target point cloud.

[0041] In this technical solution, for a target point cloud, the direction vectors of multiple surface normals are calculated. Each point has a normal vector, representing the direction of the normal to the point on the surface. Multiple surface normal vectors in the target point cloud are collected. The collected surface normal vectors are averaged, and their average value is calculated. The resulting average normal vector is used as the direction vector of the target point cloud.

[0042] By following the steps above, the direction vectors of multiple surface normals in the target point cloud can be calculated, and their average value can be used as the direction vector of the target point cloud. This yields a direction vector representing the overall orientation of the target point cloud.

[0043] In some technical solutions, optionally, the matrix corresponding to the direction vector is determined, including: determining the corresponding Euler angles based on the direction vector; or determining the matrix based on multiple Euler angles.

[0044] In this technical solution, when determining the matrix, the corresponding Euler angles are first calculated based on the given direction vector. Euler angles are a common way to describe rotation, usually represented by three angles. Then, the rotation matrix is ​​determined based on the multiple Euler angles. The rotation matrix is ​​a 3×3 matrix used to represent rotational transformations in three-dimensional space. The formula for the rotation matrix can be used to calculate the rotation based on the obtained Euler angles.

[0045] It's important to note that the specific transformation from Euler angles to rotation matrices depends on the coordinate system and rotation order used. Common rotation orders include ZYX, XYZ, and YXZ, while coordinate systems typically include Euler coordinates and rotation matrix coordinates. Choose the appropriate rotation order and coordinate system based on your specific needs, and then use the corresponding transformation formula for calculation.

[0046] Due to different conventions and representations, the conversion from Euler angles to rotation matrices can take many different forms. Therefore, in your implementation, please refer to the relevant functions and conversion methods provided in the mathematical library or documentation you are using.

[0047] A second aspect of the present invention provides an object grasping device, comprising: a scene acquisition module for acquiring scene information of the environment in which the object to be grasped is located; a point cloud acquisition module for determining first point cloud information corresponding to the object to be grasped based on the scene information; a posture determination module for determining multiple grasping postures for grasping the object to be grasped based on the second point cloud information; and a grasping module for determining a target posture among the multiple grasping postures and controlling a robotic arm to grasp the object to be grasped according to the target posture.

[0048] The technical solution of the object grasping device according to the present invention mainly includes a scene acquisition module, a point cloud acquisition module, a posture determination module, and a grasping module. Before grasping an object, it is necessary to acquire scene information of the environment in which the object to be grasped is located. This information may include the object's position, posture, size, and the positions of surrounding objects. By processing the acquired scene information, first point cloud information corresponding to the object to be grasped can be obtained. It can be understood that a point cloud is a set of three-dimensional coordinate points, which can be used to represent information such as the shape and position of an object. Based on the first point cloud information, multiple possible grasping postures can be determined. These grasping postures may include information such as the position and posture of the end effector of the robotic arm, used to control the robotic arm to grasp. Finally, the optimal target posture is selected from multiple possible grasping postures to control the robotic arm to grasp. This process usually considers factors such as the object's position, posture, and relationship with surrounding objects. After determining the target posture, the movement of the end effector of the robotic arm can be controlled to move the robotic arm to a suitable position and adjust posture and other parameters to finally complete the grasping operation of the object to be grasped. Overall, this object grasping method utilizes robot vision and intelligent control technologies, combined with point cloud information processing and posture adjustment technologies, to achieve automated grasping of the object to be grasped.

[0049] A third aspect of the present invention provides a robot, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein when the program or instructions are executed by the processor, the steps of the object grasping method in any of the above technical solutions are implemented.

[0050] A fourth aspect of the present invention provides a readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the object grasping method in any of the above-described technical solutions.

[0051] The fifth aspect of the present invention provides a chip, the chip including a processor and a communication interface, the communication interface and the processor being coupled, the processor being used to run programs or instructions to implement the steps of the object grasping method in any of the above technical solutions.

[0052] Additional aspects and advantages of the technical solutions of the present invention will become apparent in the following description or may be learned by practice of the invention. Attached Figure Description

[0053] Figure 1 A schematic diagram of an object grasping method according to an embodiment of the present invention is shown;

[0054] Figure 2 A schematic diagram of an object grasping method according to an embodiment of the present invention is shown;

[0055] Figure 3 A schematic diagram of an object grasping method according to an embodiment of the present invention is shown;

[0056] Figure 4 A schematic diagram of an object grasping method according to an embodiment of the present invention is shown;

[0057] Figure 5 A schematic diagram of an object grasping method according to an embodiment of the present invention is shown;

[0058] Figure 6 A schematic diagram of the structure of an object grasping device according to an embodiment of this application is shown;

[0059] Figure 7 A schematic diagram of a robot according to an embodiment of the present invention is shown;

[0060] Figure 8 A flowchart illustrating a grasping posture determination method according to an embodiment of the present invention is shown;

[0061] Figure 9 It shows Figure 8 A schematic diagram illustrating the generation of grasping postures using an automatic refined set decomposition structure.

[0062] in, Figures 6 to 7 The correspondence between the reference numerals and component names in the attached drawings is as follows:

[0063] 100: Robot; 200: Object grasping device; 202: Scene acquisition module; 204: Point cloud acquisition module; 206: Filtering module; 208: Attitude determination module; 210: Grasping module; 212: Memory; 214: Processor. Detailed Implementation

[0064] To better understand the above-described objectives, features, and advantages of the embodiments of the present invention, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0065] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, embodiments of the invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of this application is not limited to the specific embodiments disclosed below.

[0066] The following reference Figures 1 to 9 This invention describes an object grasping method, object grasping device, robot, readable storage medium, and chip provided according to some embodiments of the present invention.

[0067] This embodiment provides a method for grasping objects, such as... Figure 1 As shown, it includes:

[0068] Step S102: Obtain scene information of the environment in which the object to be grabbed is located;

[0069] Step S104: Determine the first point cloud information corresponding to the object to be captured based on the scene information;

[0070] Step S108: Determine multiple grasping postures for grasping the object to be grasped based on the first point cloud information;

[0071] Step S110: Determine the target posture among multiple grasping postures, and control the robotic arm to grasp the object to be grasped according to the target posture.

[0072] The object grasping method provided in this embodiment requires acquiring scene information about the environment in which the object to be grasped is located before grasping it. This information may include the object's position, orientation, size, and the positions of surrounding objects. By processing the acquired scene information, first point cloud information corresponding to the object to be grasped can be obtained. A point cloud is a set of three-dimensional coordinate points that can be used to represent information such as the shape and position of an object. Based on the second point cloud information, multiple possible grasping postures can be determined. These postures may include information such as the position and orientation of the robotic arm's end effector, used to control the robotic arm to grasp. Finally, the optimal target posture is selected from the multiple possible grasping postures to control the robotic arm to grasp. This process typically considers factors such as the object's position, orientation, and relationship with surrounding objects. After determining the target posture, the movement of the robotic arm's end effector can be controlled to move the robotic arm to a suitable position and adjust parameters such as orientation, ultimately completing the grasping operation of the object to be grasped. Overall, this object grasping method utilizes robot vision and intelligent control technologies, combined with point cloud information processing and posture adjustment technologies, to achieve automated grasping of the object to be grasped.

[0073] It should be emphasized that this application directly uses the 3D geometric information of the object's point cloud to perform attitude calculation, without the need to annotate the object's grasping attitude in 3D space, which greatly saves manpower for annotation and material resources for simulation machines.

[0074] Optionally, such as Figure 2 As shown, the scene information of the environment in which the object to be grabbed is located is obtained, specifically including:

[0075] Step S1022: Obtain scene information of the current scene through a 3D camera. The scene information includes image information and depth information of the scene.

[0076] Image information and depth information are the information obtained after registration.

[0077] In this embodiment, the 3D camera can simultaneously acquire color images and depth images of objects in the scene. The color image provides appearance information of the objects, while the depth image provides distance information. This information can be used for subsequent point cloud processing and object pose estimation.

[0078] Furthermore, 3D cameras typically acquire depth information using technologies such as infrared or lasers, enabling them to quickly and accurately obtain information such as the position and shape of objects in a scene under different lighting conditions.

[0079] In summary, acquiring image and depth information of the current scene using a 3D camera can provide important reference for subsequent object grasping operations.

[0080] Optionally, such as Figure 3 As shown, the first point cloud information corresponding to the object to be grasped is determined based on the scene information, specifically including:

[0081] Step S1042: Determine the first point cloud information based on the intrinsic parameter matrix of the 3D camera, image information, and depth information.

[0082] In this embodiment, before performing the object grasping operation, it is necessary to first obtain the camera's intrinsic parameter matrix and determine the first point cloud information corresponding to the object to be grasped based on the scene information. The intrinsic parameter matrix is ​​a parameter matrix used to describe the camera, containing information such as the camera's focal length, principal point coordinates, and distortion coefficients. It can convert the image coordinates acquired by the camera into real-world coordinates in the camera coordinate system. Using the image information and depth information acquired by the camera, the point cloud information in the scene can be obtained through a 3D point cloud reconstruction algorithm. Then, based on this point cloud information, techniques such as object recognition and target detection can be used to determine the position and pose of the object to be grasped in the point cloud. Different point cloud processing algorithms may be required for different object shapes and sizes, such as global matching algorithms and local model reconstruction algorithms. Finally, by mapping the image coordinates acquired by the camera to the point cloud, the first point cloud information corresponding to the object to be grasped can be obtained. After obtaining the first point cloud information corresponding to the object to be grasped, some point cloud filtering operations may be required to remove irrelevant or noisy point clouds in order to further determine parameters such as grasping pose and target pose. In summary, by obtaining the camera's intrinsic parameter matrix and determining the first point cloud information corresponding to the object to be grasped based on scene information, an important data foundation can be provided for subsequent point cloud processing and object grasping operations.

[0083] Further, the first point cloud information is determined based on the intrinsic parameter matrix of the 3D camera, image information, and depth information, including: acquiring an object detection network model and a segmentation network model; identifying the image information through the object detection network model and the segmentation network model to determine the coordinate information of the first region; determining the depth information corresponding to the first region based on the coordinate information of the first region; and converting the depth information of the first region into the first point cloud information based on the intrinsic parameter matrix.

[0084] To determine the first point cloud information, it is first necessary to acquire network models suitable for object detection and segmentation tasks. These models can be used to detect and locate objects to be grasped in images and provide bounding boxes or pixel-level segmentation results. The acquired object detection and segmentation network models are then used to process the input image information. The object detection model can detect the location of the object to be grasped, while the segmentation model can segment the object pixels to form object regions. Based on the object detection or segmentation results, the coordinate information of the first region of the object to be grasped is obtained. The first region can be the bounding box or segmented region of the object. Based on the coordinate information of the first region, the depth value of the corresponding region is extracted from the depth information. The corresponding depth value is found in the depth map using the pixel coordinates. Using the intrinsic parameter matrix of the 3D camera, the depth information of the first region is converted from the image coordinate system to 3D point coordinates in the camera coordinate system. The converted 3D point coordinates are then combined to form the first point cloud information. Each point consists of its 3D coordinates and possible other attributes (such as color, normals, etc.).

[0085] Optionally, such as Figure 4 As shown, it also includes: step S106: filtering the first point cloud information to determine the filtered second point cloud information.

[0086] Step S108: Determine multiple grasping postures for grasping the object to be grasped based on the first point cloud information, and further includes: Step S1082: Determine multiple grasping postures for grasping the object to be grasped based on the second point cloud information.

[0087] like Figure 5 As shown, step S106: Filtering the first point cloud information to determine the filtered second point cloud information specifically includes:

[0088] Step S1062: Obtain at least one granular geometric filter structure, each granular geometric filter structure including an anomaly filtering algorithm and a clustering algorithm;

[0089] Step S1064: Remove abnormal point clouds from the first point cloud information according to the anomaly filtering algorithm, and cluster the filtered first point cloud information according to the clustering algorithm;

[0090] Step S1066: Determine the point cloud information of the largest cluster after clustering as the second point cloud information.

[0091] In this embodiment, the first point cloud information is filtered using a granular geometric filtering method. This method is based on parameters such as point cloud density and distance. By setting different filtering parameters and thresholds, point cloud filtering and clustering operations can be performed. For example, a coarse filter based on Euclidean distance or a Gaussian curvature filter based on a statistical model can be used. The acquired first point cloud information is input into the granular geometric filtering structure. Anomaly filtering algorithms are used to remove abnormal point clouds, while clustering algorithms are used to cluster the filtered point cloud information. Anomaly filtering algorithms can identify and remove point clouds that do not match the surrounding environment or the shape of the object, such as noisy point clouds and outliers. Clustering algorithms can divide the filtered point cloud information into different clusters, each representing an object or part of an object. After clustering, the largest cluster needs to be selected as the point cloud information for the object to be grasped. This process typically involves calculating parameters such as the number of points and density in different clusters to determine the largest cluster, which is then passed as the second point cloud information to the robotic arm controller for precise positioning and attitude adjustment operations. In summary, using a granular geometry filtering method to filter the first point cloud information can achieve a certain degree of automated processing, while improving the accuracy and efficiency of object grasping.

[0092] Furthermore, there are multiple granular geometric filter structures, each with a different radius distance between the point cloud and its surrounding neighborhood. By setting multiple granular geometric filter structures and limiting the radius distance between the point cloud and its surrounding neighborhood in each structure, the flexibility and accuracy of point cloud processing can be improved to better adapt to the characteristics of different object shapes and sizes.

[0093] For example, larger objects can be clustered using a larger radius distance to ensure that the point cloud information covers the entire object surface; while smaller objects can be clustered using a smaller radius distance to avoid misclassifying multiple objects as the same. Different coarse filters and clustering algorithms can also be selected and adjusted based on the object's shape, surface texture, and other features to achieve more accurate point cloud processing and grasping operations.

[0094] In one embodiment, the method further includes: when there are multiple granular geometric filter structures, determining the application order of the multiple granular geometric filter structures; arranging the multiple granular geometric filter structures according to the application order, wherein the neighborhood distance of the granular geometric filter structure applied earlier is less than the neighborhood distance of the granular geometric filter structure applied later.

[0095] Determining the application order of multiple granular geometric filtering structures requires considering the neighborhood distance. The neighborhood distance of the first applied granular geometric filtering structure should be smaller, while the neighborhood distance of the later applied structure should be larger. This ensures that outliers in the nearest neighborhood are removed first, followed by clustering of the point cloud in the more distant neighborhood, resulting in more accurate and stable second point cloud information. Furthermore, applying the granular geometric filtering structure with smaller neighborhood distance first can more meticulously remove outliers from the first point cloud information while retaining valid points in the nearest neighborhood. Applying the granular geometric filtering structure with larger neighborhood distance later can cluster the point clouds in the more distant neighborhood together, forming larger clusters, and removing unnecessary small clusters, thus obtaining the target point cloud.

[0096] In this sequence, filtering with smaller neighborhood distances first removes most outliers, followed by filtering with larger neighborhood distances, clustering the valid point cloud into the target point cloud. This processing order maximizes the preservation of the target point cloud while removing unnecessary noise and outliers, improving the quality and accuracy of the point cloud data.

[0097] Furthermore, the algorithm incorporates three different levels of anomaly filtering and clustering. The algorithm uses the radius distance between the point cloud and its surrounding neighborhood as a gradation point to remove points from more distant neighbors. The neighborhood distances for the first, second, and third levels of anomaly filtering gradually increase, ensuring effective removal of obvious outliers in the initial filtering phase. The filtering effectiveness gradually decreases as the filtering level increases, ensuring the geometric structure of the target point cloud is effectively preserved. The filtered point cloud is then clustered, with the largest cluster selected as the target point cloud, further eliminating outlier regions generated during the filtering process.

[0098] Based on any of the above embodiments, multiple grasping postures for grasping the object to be grasped are determined according to the second point cloud information, specifically including: segmenting the second point cloud information based on a geometric segmentation structure to determine the target point cloud; and determining multiple grasping postures corresponding to a target point cloud having a number less than a stopping threshold.

[0099] When determining multiple grasping postures, the second point cloud information is first separated from the first point cloud information to determine the target point cloud. After obtaining the second point cloud information, it needs to be separated from the first point cloud information. This process typically uses techniques based on Euclidean distance or normal vectors for segmentation and extraction to determine the target point cloud. After determining the target point cloud, the robotic arm controller can calculate multiple possible grasping postures. These postures can be calculated based on parameters such as the position, orientation, and size of the target point cloud to ensure that the robotic arm can safely and effectively grasp the object.

[0100] The process of segmenting the second point cloud information based on the geometric segmentation structure to determine the target point cloud includes: acquiring the geometric segmentation structure and clustering structure; separating the second point cloud information according to the geometric segmentation structure to determine the separated point cloud information; and clustering the separated point cloud information according to the clustering structure to determine the largest cluster as the target point cloud.

[0101] Determining the target point cloud primarily involves segmentation and clustering algorithms. Specifically, a geometric segmentation structure and a clustering structure are first obtained. The geometric segmentation structure can be a set of predefined geometric features or segmentation algorithms used to segment the point cloud; the clustering structure can be a set of predefined clustering algorithms or parameter settings used to cluster the point cloud. Then, using the selected geometric segmentation structure, the second set of point cloud information is segmented, resulting in separate point cloud sets. The segmentation process can operate based on the geometric features of the point cloud (e.g., distance, normal vectors), dividing the point cloud into different clusters. For each separate point cloud set, the selected clustering algorithm or parameter settings are applied to cluster the point cloud. The clustering algorithm can group points based on their location, density, features, etc., forming different clusters. Based on the clustering results, the cluster with the largest number of points is determined as the target point cloud. By comparing the number of points in each cluster, the cluster with the largest number of points is selected as the point cloud set representing the target object.

[0102] Through the above steps, multiple point cloud sets can be separated from the second point cloud information, and the largest cluster can be determined as the target point cloud through clustering operations for subsequent capture operations.

[0103] When calculating the grasping posture, it is important to set a stop threshold to avoid excessive calculation time. Once the number of calculated grasping postures reaches the stop threshold, the calculation can be stopped, and an optimal grasping posture can be selected for subsequent robotic arm control operations.

[0104] In summary, determining multiple grasping postures for grasping the object based on the second cloud information can provide important reference for subsequent robotic arm control and object grasping operations.

[0105] Determining multiple grasping postures corresponding to a target point cloud quantity less than a stopping threshold includes: determining the direction vectors of multiple target point clouds; determining the matrix corresponding to the direction vectors; determining the grasping position based on the centroid position of each target point cloud; determining the grasping posture corresponding to the target point cloud based on the grasping position and the matrix; determining the quantity of target point clouds; iteratively determining the target point clouds and grasping postures until the quantity of target point clouds is less than the stopping threshold, at which point the iteration stops and multiple grasping postures are determined.

[0106] For each target point cloud, calculate its normal vector or surface normal, which serves as the direction vector for that target point cloud. Calculate a rotation matrix for each target point cloud's direction vector; this matrix describes the robot arm's orientation during grasping. Calculate the centroid position of each target point cloud, i.e., the average position of all points in the point cloud, as the grasping position for that target point cloud. Combine the grasping position with the corresponding rotation matrix to determine the robot arm's posture during grasping, including position and orientation. Count the number of target point clouds and determine if it is less than a stopping threshold. If the number of target point clouds is greater than the stopping threshold, continue iterating for the next round of point cloud segmentation and grasping posture determination, repeating the above steps. If the number of target point clouds is less than the stopping threshold, stop the iteration and determine multiple grasping postures.

[0107] By following the steps above, the number of target point clouds and their corresponding grasping postures can be determined iteratively until the number of target point clouds is less than the stopping threshold, thereby obtaining multiple grasping postures that meet the requirements.

[0108] Determining the direction vector of the target point cloud includes: determining the direction vectors of multiple surface normals in the target point cloud; determining the average value of the direction vectors of multiple surface normals corresponding to a target point cloud, and using the average value as the direction vector of the target point cloud.

[0109] For a target point cloud, calculate the direction vectors of multiple surface normals. Each point has a normal vector representing the direction of the normal to the surface. Collect multiple surface normal vectors from the target point cloud. Average the collected surface normal vectors and calculate their average value. Use the resulting average normal vector as the direction vector of the target point cloud.

[0110] By following the steps above, the direction vectors of multiple surface normals in the target point cloud can be calculated, and their average value can be used as the direction vector of the target point cloud. This yields a direction vector representing the overall orientation of the target point cloud.

[0111] Determining the matrix corresponding to the direction vector includes: determining the corresponding Euler angles based on the direction vector; and determining the matrix based on multiple Euler angles.

[0112] When determining the rotation matrix, first calculate the corresponding Euler angles based on the given direction vector. Euler angles are a common way to describe rotation, usually represented by three angles. Then, determine the rotation matrix based on the multiple Euler angles. The rotation matrix is ​​a 3×3 matrix used to represent rotational transformations in three-dimensional space. The rotation matrix can be calculated using the formula given by the obtained Euler angles.

[0113] It's important to note that the specific transformation from Euler angles to rotation matrices depends on the coordinate system and rotation order used. Common rotation orders include ZYX, XYZ, and YXZ, while coordinate systems typically include Euler coordinates and rotation matrix coordinates. Choose the appropriate rotation order and coordinate system based on your specific needs, and then use the corresponding transformation formula for calculation.

[0114] Due to different conventions and representations, the conversion from Euler angles to rotation matrices can take many different forms. Therefore, in your implementation, please refer to the relevant functions and conversion methods provided in the mathematical library or documentation you are using.

[0115] In one specific embodiment, the grasping posture in 3D space is calculated by performing geometric structure analysis on the point cloud within the target area, such as... Figure 8 As shown, a general object grasping algorithm based on visual geometry is proposed. The pose of the current object in 3D space is obtained by feeding RGB and Depth images into different structures. The current scene is captured by an external 3D camera, obtaining RGB and Depth images, denoted as S1 and S2 respectively, where S1 and S2 have already been registered.

[0116] Specifically, S1 and S2 are fed into Model 1 to obtain the target point cloud P1. Model 1 can be a conventional 2D object detection and segmentation network, such as the object detection network model of the YOLO series, the semantic segmentation model of the FCN series, or the instance segmentation model of the Mask RNNN series. By placing the coordinates of the region identified by the detection and segmentation network on S1 onto S2, the depth information of the target region is obtained, and then the depth information of the region is converted into the target point cloud P1 using the intrinsic parameters of the 3D camera. Model 1 can also be a point cloud recognition and segmentation algorithm, such as a conventional 3D object detection algorithm or an unsupervised clustering algorithm, which can segment the geometric point cloud of the object from the scene point cloud to obtain the target point cloud P1. In this invention, the type of 2D / 3D object detection and segmentation model used in Model 1 is not limited, as long as it can segment the target point cloud.

[0117] The target point cloud P1 is input into a multi-level granular geometric filtering structure to obtain the filtered target point cloud P2. The multi-level granular geometric filtering structure includes three different levels of anomaly filtering algorithms and clustering algorithms. The level is determined by setting the radius distance between the point cloud and its surrounding neighborhood, thus deleting points from more distant neighbors. The neighborhood distances set for the first, second, and third levels of anomaly filtering gradually increase, ensuring effective removal of obvious anomaly points in the initial filtering stage. The filtering effect gradually decreases as the filtering progresses, ensuring the effective preservation of the target point cloud's geometric structure. The filtered point cloud is then clustered, and the largest cluster is selected as the target point cloud P2 to further eliminate outlier regions generated during the filtering process.

[0118] The target point cloud P2 is fed into an automatic refined geometry decomposition structure to obtain multiple grasping poses M1, M2, ... M for the current object. n Specifically, such as Figure 9 As shown, the target point cloud P2 is first fed into a geometric segmentation structure. This structure can be composed of segmentation algorithms such as planar segmentation, Delaunay triangulation segmentation, and curvature segmentation, or a mixture of different algorithms. This separates a portion of the point cloud from the target point cloud P2. Then, a clustering algorithm in the clustering structure selects the largest cluster as a candidate for capturing the target point cloud P2. 1 This eliminates interference from other clusters. Calculate the target point cloud. The direction vectors of the m surface normals Then, the mean of the direction vectors of the m surface normals is calculated as the target point cloud. Direction vector Direction vector Convert to Euler angles of rotation [α] * ,β * ,γ * The symbol ] indicates that this facilitates subsequent operations. At this point, a matrix composed of k randomly generated Euler angles arranged row by row is introduced:

[0119]

[0120] In L1, each value is sampled from a uniform distribution U(-5°, 5°). Similarly, for L2...L... n Each value is sampled from a uniform distribution U(-5°, 5°) to calculate the target point cloud. Using the centroid of the object as the grasping position [x1, y1, z1], the object's grasping posture is as follows:

[0121]

[0122] Each row represents a grasping pose of the point cloud in that region, and each region's point cloud contains k grasping poses.

[0123] Then target point cloud Separate from the target point cloud P2 to obtain the punctuation cloud. The grasping posture is obtained through the same process described above. By continuously iterating and acquiring new grasping postures and target point clouds, until the separated target point cloud P2 is obtained. n If the number of points is less than the set threshold, the iteration stops, and at this point, n sets of grasping postures {M1, M2, ..., M} can be obtained. n The feedback is sent to the robotic arm to plan and select a more advantageous grasping posture to pick up the object.

[0124] According to the embodiments provided in this application, the 3D geometric information of the object point cloud can be directly used to calculate the grasping posture without additional grasping posture annotation, greatly saving manpower for annotation and material resources for simulation machines. Furthermore, this technical solution uses logical derivation, without using high-performance computing resources such as GPUs, and is fast with minimal memory overhead, making it easy to port to devices. It has broad applicability to common object grasping estimation and can be flexibly adapted to existing common recognition algorithms.

[0125] like Figure 6 As shown in the figure, this application embodiment provides an object grasping device 200. The object grasping device 200 includes a scene acquisition module 202, a point cloud acquisition module 204, a filtering module 206, a posture determination module 208, and a grasping module 210.

[0126] The system includes a scene acquisition module 202 for acquiring scene information of the environment in which the object to be grasped is located; a point cloud acquisition module 204 for determining the first point cloud information corresponding to the object to be grasped based on the scene information; a filtering module 206 for filtering the first point cloud information to determine the filtered second point cloud information; a posture determination module 208 for determining multiple grasping postures for grasping the object to be grasped based on the second point cloud information; and a grasping module 210 for determining the target posture among the multiple grasping postures and controlling the robotic arm to grasp the object to be grasped according to the target posture.

[0127] Before grasping an object, this embodiment needs to acquire scene information about the environment in which the object to be grasped is located. This information may include the object's position, posture, size, and the positions of surrounding objects. By processing the acquired scene information, first point cloud information corresponding to the object to be grasped can be obtained. A point cloud is a collection of three-dimensional coordinate points that can be used to represent information such as the shape and position of an object. Based on the first point cloud information, filtering operations can be performed, such as removing point clouds of irrelevant objects, to obtain filtered second point cloud information. Based on the second point cloud information, multiple possible grasping postures can be determined. These postures may include information such as the position and posture of the robotic arm's end effector, used to control the robotic arm to grasp. Finally, the optimal target posture is selected from the multiple possible grasping postures to control the robotic arm to grasp. This process typically considers factors such as the object's position, posture, and relationship with surrounding objects. After determining the target posture, the movement of the robotic arm's end effector can be controlled to move the robotic arm to a suitable position and adjust posture and other parameters to ultimately complete the grasping operation of the object to be grasped. Overall, this object grasping method utilizes robot vision and intelligent control technologies, combined with point cloud information processing and posture adjustment technologies, to achieve automated grasping of the object to be grasped.

[0128] The object grasping device 200 in this application embodiment can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not impose specific limitations.

[0129] The object grasping device 200 in this embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this embodiment does not specifically limit its use.

[0130] The object grasping device 200 provided in this application embodiment can achieve... Figures 1 to 5 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0131] Optionally, such as Figure 7 As shown, this application embodiment also provides a robot 100, including a processor 214, a memory 212, and a program or instructions stored in the memory 212 and executable on the processor 214. When the program or instructions are executed by the processor 214, they implement the various processes of the above-described object grasping method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0132] It should be noted that the robot in the embodiments of this application includes both electronic and non-electronic devices.

[0133] The processor 214 is used to: acquire scene information of the environment in which the object to be grasped is located; determine the first point cloud information corresponding to the object to be grasped based on the scene information; filter the first point cloud information to determine the filtered second point cloud information; determine multiple grasping postures for grasping the object to be grasped based on the second point cloud information; determine the target posture among the multiple grasping postures, and control the robotic arm to grasp the object to be grasped according to the target posture.

[0134] The above approach requires acquiring scene information about the environment in which the object to be grasped exists before object grasping can be performed. This information may include the object's position, orientation, size, and the positions of surrounding objects. Processing the acquired scene information yields the first point cloud information corresponding to the object to be grasped. A point cloud is a collection of three-dimensional coordinate points that can represent the shape and position of an object. Based on the first point cloud information, filtering operations can be performed, such as removing point clouds containing irrelevant objects, resulting in a filtered second point cloud. Based on the second point cloud information, multiple possible grasping postures can be determined. These postures may include information such as the position and orientation of the robotic arm's end effector, used to control the robotic arm to grasp the object. Finally, the optimal target posture is selected from the multiple possible postures to control the robotic arm to grasp the object. This process typically considers factors such as the object's position, orientation, and relationship with surrounding objects. After determining the target posture, the movement of the robotic arm's end effector can be controlled to move the robotic arm to a suitable position and adjust its orientation and other parameters, ultimately completing the grasping operation. Overall, this object grasping method utilizes robot vision and intelligent control technologies, combined with point cloud information processing and posture adjustment technologies, to achieve automated grasping of the object to be grasped.

[0135] Optionally, the processor 214 is also configured to perform the following steps: obtain the intrinsic parameter matrix of the 3D camera; determine the first point cloud information based on the intrinsic parameter matrix, image information, and depth information.

[0136] Optionally, the processor 214 is further configured to acquire at least one granular geometric filtering structure, each granular geometric filtering structure including an anomaly filtering algorithm and a clustering algorithm; remove abnormal point clouds from the first point cloud information according to the anomaly filtering algorithm, and cluster the filtered first point cloud information according to the clustering algorithm; determine the point cloud information of the largest cluster after clustering as the second point cloud information.

[0137] Optionally, the processor 214 is further configured to separate the second point cloud information from the first point cloud information to determine the target point cloud; and to determine multiple grasping postures corresponding to a number of target point clouds that are less than a stopping threshold.

[0138] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described object grasping method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0139] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes readable storage media such as computer read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0140] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described object grasping method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0141] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0142] According to embodiments of the object grasping method, object grasping device, robot, readable storage medium, and chip of the present invention, when performing attitude calculation, the 3D geometric information of the object point cloud is directly used, without the need to annotate the object's grasping attitude in 3D space, which greatly saves manpower for annotation and material resources for simulation machines.

[0143] It should be noted that the solution designed in this application is applicable to the deployment and use of various product platforms, such as intelligent robots, mobile phones, surveillance cameras, and smart appliances. The technical solution provided in this application can be used with various sensors as inputs, such as RGB-D cameras, RGB cameras + LiDAR, RGB cameras + millimeter-wave radar, RGB cameras + infrared cameras, and other components capable of capturing RGB images and performing ranging imaging.

[0144] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0145] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0146] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for grasping an object, characterized in that, include: Obtain scene information about the environment in which the object to be grabbed is located; Determine the first point cloud information corresponding to the object to be grasped based on the scene information; The first point cloud information is filtered to determine the filtered second point cloud information. Based on the second point cloud information, multiple grasping postures for grasping the object to be grasped are determined; A target posture is determined among multiple grasping postures, and the robotic arm is controlled to grasp the object to be grasped according to the target posture. The determination of the filtered second point cloud information includes: obtaining at least one granular geometric filtering structure, each of the granular geometric filtering structures including an anomaly filtering algorithm and a clustering algorithm; Abnormal point clouds are removed from the first point cloud information according to the anomaly filtering algorithm, and the filtered first point cloud information is clustered according to the clustering algorithm; the point cloud information of the largest cluster after clustering is determined as the second point cloud information.

2. The object grasping method according to claim 1, characterized in that, The process of obtaining scene information about the environment in which the object to be grasped is located includes: The scene information of the current environment is acquired by a 3D camera. The scene information includes image information and depth information of the scene. The image information and the depth information are registered information.

3. The object grasping method according to claim 2, characterized in that, Determining the first point cloud information corresponding to the object to be grasped based on the scene information includes: The first point cloud information is determined based on the intrinsic parameter matrix of the 3D camera, the image information, and the depth information.

4. The object grasping method according to claim 3, characterized in that, The first point cloud information is determined based on the intrinsic parameter matrix of the 3D camera, the image information, and the depth information, including: Obtain the object detection network model and the segmentation network model; The image information is identified using the target detection network model and the segmentation network model to determine the coordinate information of the first region; Determine the depth information corresponding to the first region based on the coordinate information of the first region; The depth information of the first region is converted into first point cloud information based on the intrinsic parameter matrix.

5. The object grasping method according to claim 2, characterized in that, The number of granular geometric filtering structures is multiple, and the radius distance between the point cloud and the surrounding neighborhood in each granular geometric filtering structure is different.

6. The object grasping method according to claim 5, characterized in that, Also includes: When there are multiple granular geometric filter structures, the application order of the multiple granular geometric filter structures is determined; Multiple granular geometric filter structures are arranged according to the application order, wherein the neighborhood distance of the granular geometric filter structure applied earlier is smaller than the neighborhood distance of the granular geometric filter structure applied later.

7. The object grasping method according to claim 2, characterized in that, The step of determining multiple grasping postures for grasping the object to be grasped based on the second point cloud information specifically includes: The second point cloud information is segmented based on the geometric segmentation structure to determine the target point cloud; Determine multiple grasping postures corresponding to a target point cloud whose number is less than a stopping threshold.

8. The object grasping method according to claim 7, characterized in that, The segmentation of the second point cloud information based on the geometric segmentation structure to determine the target point cloud includes: Obtain the geometric segmentation structure and clustering structure; The second point cloud information is separated according to the geometric segmentation structure to determine the separated point cloud information; The separated point cloud information is clustered according to the clustering structure, and the largest cluster is determined as the target point cloud.

9. The object grasping method according to claim 8, characterized in that, The determination of multiple grasping postures corresponding to the number of target point clouds being less than a stopping threshold includes: Determine the direction vectors of multiple target point clouds; Determine the matrix corresponding to the direction vector; The grab position is determined based on the centroid position of each target point cloud; The grasping posture corresponding to the target point cloud is determined based on the grasping position and the matrix; Determine the number of the target point cloud; The target point cloud and the grasping posture are determined iteratively until the number of target point clouds is less than a stopping threshold. Then the iteration stops and multiple grasping postures are determined.

10. The object grasping method according to claim 9, characterized in that, Determining the direction vectors of the plurality of target point clouds includes: Determine the direction vectors of multiple surface normals in the target point cloud; The average value of the direction vectors of multiple surface normals corresponding to a target point cloud is determined, and the average value is used as the direction vector of the target point cloud.

11. The object grasping method according to claim 9, characterized in that, Determining the matrix corresponding to the direction vector includes: Determine the corresponding Euler angles based on the direction vector; The matrix is ​​determined based on the multiple Euler angles.

12. A robot, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the object grasping method as described in any one of claims 1 to 11.

13. An object grasping device, characterized in that, include: The scene acquisition module is used to acquire scene information of the environment in which the object to be captured is located; The point cloud acquisition module is used to determine the first point cloud information corresponding to the object to be captured based on the scene information; The filtering module is used to filter the first point cloud information and determine the filtered second point cloud information. The attitude determination module is used to determine multiple grasping attitudes for grasping the object to be grasped based on the second point cloud information. The grasping module is used to determine a target posture among multiple grasping postures and control the robotic arm to grasp the object to be grasped according to the target posture. The determination of the filtered second point cloud information includes: obtaining at least one granular geometric filtering structure, each of the granular geometric filtering structures including an anomaly filtering algorithm and a clustering algorithm; Abnormal point clouds are removed from the first point cloud information according to the anomaly filtering algorithm, and the filtered first point cloud information is clustered according to the clustering algorithm; the point cloud information of the largest cluster after clustering is determined as the second point cloud information.

14. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the object grasping method as described in any one of claims 1 to 11.

15. A chip, characterized in that, The chip includes a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the steps of the object grasping method as described in any one of claims 1 to 11.

Images