Object grabbing method and apparatus, robot, readable storage medium, and chip

Abstract

An object grabbing method and apparatus, a robot, a readable storage medium, and a chip. The method comprises: acquiring scene information of an environment where an object to be grabbed is located (S102); on the basis of the scene information, determining first point cloud information corresponding to the object (S104); on the basis of the first point cloud information, determining a plurality of grabbing orientations for grabbing the object (S108); and determining a target orientation among the plurality of grabbing orientations, and controlling a robotic arm to grab the object on the basis of the target orientation (S110).

Description

Object grasping method, device, robot, readable storage medium and chip

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on July 14, 2023, with application number "202310871127.X" and application name "Object grasping method, device, robot, readable storage medium and chip", all contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computers, and in particular to an object grasping method, an object grasping device, a robot, a readable storage medium, and a chip. Background Art

[0003] In related technologies, when performing grasping posture recognition during object grasping, it is usually necessary to mark the grasping posture of the object in space. Using simulation marking or manual marking takes a long time and cannot perform timely analysis and grasping.

[0004] Summary of the Invention

[0005] In order to solve or improve at least one of the above technical problems, one object of the present application is to provide an object grasping method.

[0006] Another object of the present application is to provide an object grasping device.

[0007] Another object of the present application is to provide a robot.

[0008] Another object of the present application is to provide a readable storage medium.

[0009] Another object of the present application is to provide a chip.

[0010] Another object of the present application is to provide a computer program product.

[0011] To achieve the above-mentioned objectives, the first aspect of the present application provides an object grasping method, including: obtaining scene information of the environment in which the object to be grasped is located; determining first point cloud information corresponding to the object to be grasped based on the scene information; determining multiple grasping postures for grasping the object to be grasped based on the first point cloud information; determining a target posture among the multiple grasping postures, and controlling the robotic arm to grasp the object to be grasped according to the target posture.

[0012] The second aspect of the present application provides an object grasping device, including: a scene acquisition module, used to acquire scene information of the environment in which the object to be grasped is located; a point cloud acquisition module, used to determine first point cloud information corresponding to the object to be grasped based on the scene information; a posture determination module, used to determine multiple grasping postures for grasping the object to be grasped based on the second point cloud information; a grasping module, used to determine a target posture among multiple grasping postures, and control the robotic arm to grasp the object to be grasped according to the target posture.

[0013] The third aspect of the present application provides a robot comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor. When the program or instruction is executed by the processor, the steps of the object grasping method in any of the above-mentioned technical solutions are implemented.

[0014] In a fourth aspect, the present application provides a readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the object grasping method in any of the above technical solutions are implemented.

[0015] In a fifth aspect, the present application provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the steps of the object grasping method in any of the above technical solutions.

[0016] In a sixth aspect, the present application provides a computer program product, comprising a computer program, which implements the steps of any one of the methods in the first aspect when executed by a processor.

[0017] Additional aspects and advantages of the technical solution of the present application will become apparent in the following description or be understood through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] FIG1 is a schematic diagram showing a method for grasping an object according to an embodiment of the present application;

[0019] FIG2 is a schematic diagram showing a method for grasping an object according to an embodiment of the present application;

[0020] FIG3 shows a schematic diagram of an object grasping method according to an embodiment of the present application;

[0021] FIG4 shows a schematic diagram of an object grasping method according to an embodiment of the present application;

[0022] FIG5 is a schematic diagram showing a method for grasping an object according to an embodiment of the present application;

[0023] FIG6 shows a schematic structural diagram of an object grasping device according to an embodiment of the present application;

[0024] FIG7 shows a schematic diagram of a robot according to an embodiment of the present application;

[0025] FIG8 is a schematic diagram showing a flow chart of a method for determining a grasping posture according to an embodiment of the present application;

[0026] FIG9 shows a schematic diagram of generating a grasping posture by using the automatic refinement set decomposition structure in FIG8 .

[0027] The corresponding relationship between the reference numerals and component names in FIG6 and FIG7 is as follows:

[0028] 100: robot; 200: object grasping device; 202: scene acquisition module; 204: point cloud acquisition module; 206: filtering module; 208: posture determination module; 210: grasping module; 212: memory; 214: processor. DETAILED DESCRIPTION

[0029] In order to more clearly understand the above-mentioned purposes, features and advantages of the embodiments of the present application, the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that the embodiments of the present application and the features therein can be combined with each other in the absence of conflict.

[0030] In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the embodiments of the present application may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present application is not limited to the specific embodiments disclosed below.

[0031] The object grasping method, object grasping device, robot, readable storage medium and chip provided according to some embodiments of the present application are described below with reference to Figures 1 to 9.

[0032] This embodiment provides an object grasping method, as shown in FIG1 , including:

[0033] Step S102: Acquire scene information of the environment in which the object to be grasped is located;

[0034] Step S104: determining first point cloud information corresponding to the object to be grasped according to the scene information;

[0035] Step S108: determining multiple grasping postures for grasping the object to be grasped according to the first point cloud information;

[0036] Step S110: determining a target posture among multiple grasping postures, and controlling the robotic arm to grasp the object according to the target posture.

[0037] The object grasping method provided in this embodiment requires obtaining scene information about the environment in which the object to be grasped resides before grasping. This information may include the object's position, posture, size, and the positions of surrounding objects. By processing the acquired scene information, first point cloud information corresponding to the object to be grasped can be obtained. It can be understood that a point cloud is a collection of three-dimensional coordinate points that can be used to represent information such as the object's shape and position. Based on the second point cloud information, multiple possible grasping postures can be determined. These grasping postures may include information such as the position and posture of the robot arm's end effector, which is used to control the robot arm for grasping. Ultimately, an optimal target posture is selected from the multiple possible grasping postures and used to control the robot arm for grasping. This process generally takes into account factors such as the object's position, posture, and relationship to surrounding objects. After determining the target posture, the robot arm can be moved to the appropriate position by controlling the movement of the end effector and adjusting parameters such as the posture to ultimately complete the grasping operation of the object to be grasped. Generally speaking, this object grasping method uses robot vision and intelligent control technology, combined with point cloud information processing and posture adjustment technologies, to achieve automated grasping of the object to be grasped.

[0038] It should be emphasized that this application directly uses the 3D geometric information of the object point cloud to perform posture calculation, without the need to mark the object's grasping posture in 3D space, which greatly saves manpower for marking and material resources for simulation machines.

[0039] Optionally, as shown in FIG2 , obtaining scene information of the environment in which the object to be grasped is located specifically includes:

[0040] Step S1022: Acquire scene information of the current scene through a 3D camera, where the scene information includes image information and depth information of the scene.

[0041] Image information and depth information are the information after registration.

[0042] In this embodiment, the 3D camera can simultaneously capture color and depth images of objects in the scene. The color image provides information about the object's appearance, while the depth image provides information about the object's distance. This information can be used for subsequent operations such as point cloud processing and object pose estimation.

[0043] Furthermore, three-dimensional cameras usually obtain depth information through technologies such as infrared or laser, and can quickly and accurately obtain information such as the position and shape of objects in the scene under different lighting conditions.

[0044] In short, by using a 3D camera to obtain image information and depth information of the current scene, it can provide important reference for subsequent object grasping operations.

[0045] Optionally, as shown in FIG3 , determining first point cloud information corresponding to the object to be grasped according to the scene information specifically includes:

[0046] Step S1042: Determine first point cloud information according to the intrinsic parameter matrix, image information, and depth information of the three-dimensional camera.

[0047] In this embodiment, before performing an object grasping operation, the camera's intrinsic parameter matrix must be obtained. Based on the scene information, the first point cloud information corresponding to the object to be grasped must be determined. The intrinsic parameter matrix is ​​a parameter matrix used to describe the camera. It contains information such as the camera's focal length, principal point coordinates, and distortion coefficients. It can convert the image coordinates captured by the camera into real-world coordinates in the camera coordinate system. Using the image and depth information captured by the camera, a 3D point cloud reconstruction algorithm can be used to obtain point cloud information about the scene. Based on this point cloud information, techniques such as object recognition and target detection can then be used to determine information such as the position and pose of the object to be grasped in the point cloud. Different point cloud processing algorithms, such as global matching algorithms and local model reconstruction algorithms, may be required for different object shapes and sizes. Ultimately, by mapping the image coordinates captured by the camera into the point cloud, the first point cloud information corresponding to the object to be grasped can be obtained. After obtaining the first point cloud information corresponding to the object to be grasped, operations such as point cloud filtering may be required to remove irrelevant or noisy point clouds to further determine parameters such as the grasping pose and target pose. In summary, by obtaining the camera's intrinsic parameter matrix and determining the first point cloud information corresponding to the object to be grasped based on the scene information, an important data basis can be provided for subsequent point cloud processing and object grasping operations.

[0048] Furthermore, the first point cloud information is determined based on the intrinsic parameter matrix, image information and depth information of the three-dimensional camera, including: obtaining a target detection network model and a segmentation network model; identifying the image information through the target detection network model and the segmentation network model to determine the coordinate information of the first area; determining the depth information corresponding to the first area based on the coordinate information of the first area; and converting the depth information of the first area into first point cloud information based on the intrinsic parameter matrix.

[0049] When determining the first point cloud information, it is first necessary to obtain network models suitable for target detection and segmentation tasks. These models can be used to detect and locate the object to be grasped in the image and provide the object's bounding box or pixel-level segmentation results. The input image information is processed using the acquired target detection network model and segmentation network model. The target detection model can detect the location of the object to be grasped, while the segmentation model can segment the object's pixels to form the object's region. Based on the target detection or segmentation results, the coordinate information of the first region of the object to be grasped is obtained. The first region can be the object's bounding box or segmented region. Based on the coordinate information of the first region, the depth value of the corresponding region is extracted from the depth information. The corresponding depth value is searched in the depth map using the pixel coordinates. Using the intrinsic parameter matrix of the 3D camera, the depth information of the first region is converted from the image coordinate system to 3D point coordinates in the camera coordinate system. The converted 3D point coordinates are then combined to form the first point cloud information. Each point consists of its 3D coordinates and possible other attributes (such as color, normal, etc.).

[0050] Optionally, as shown in FIG4 , the method further includes: Step S106 : filtering the first point cloud information to determine filtered second point cloud information.

[0051] Step S108: determining multiple grasping postures for grasping the object to be grasped according to the first point cloud information, and also includes: Step S1082: determining multiple grasping postures for grasping the object to be grasped according to the second point cloud information.

[0052] As shown in FIG5 , step S106 : filtering the first point cloud information to determine the filtered second point cloud information specifically includes:

[0053] Step S1062: obtaining at least one granular geometric filtering structure, each granular geometric filtering structure including an anomaly filtering algorithm and a clustering algorithm;

[0054] Step S1064: removing abnormal point clouds in the first point cloud information according to an abnormal filtering algorithm, and clustering the filtered first point cloud information according to a clustering algorithm;

[0055] Step S1066: Determine the point cloud information of the largest cluster after clustering as the second point cloud information.

[0056] In this embodiment, the first point cloud information can be filtered using a granular geometric filtering method. This method is based on parameters such as point cloud density and distance. By setting different filtering parameters and thresholds, operations such as point cloud screening and clustering can be implemented. For example, a coarse filter based on Euclidean distance or a Gaussian curvature filter based on a statistical model can be used. The acquired first point cloud information is input into the granular geometric filtering structure, an anomaly filtering algorithm is applied to remove abnormal point clouds, and a clustering algorithm is used to cluster the filtered point cloud information. The anomaly filtering algorithm can identify and remove point clouds that do not match the surrounding environment or the object shape, such as noisy point clouds and outlier point clouds. The clustering algorithm can divide the filtered point cloud information into different clusters, each representing an object or object part. After clustering, the largest cluster is selected as the point cloud information for the object to be grasped. This process typically calculates parameters such as the number of points and density of different clusters to determine the largest cluster, which is then transmitted as the second point cloud information to the robotic arm controller for operations such as precise positioning and posture adjustment. In summary, filtering the first point cloud information based on granular geometry filtering method can achieve a certain degree of automated processing, while improving the accuracy and efficiency of object grasping.

[0057] Furthermore, there are multiple granular geometric filtering structures, each with a different radius between the point cloud and its surrounding neighborhood. By setting multiple granular geometric filtering structures and limiting the radius between the point cloud and its surrounding neighborhood to different structures, the flexibility and accuracy of point cloud processing are improved to better adapt to the characteristics of different object shapes and sizes.

[0058] For example, for larger objects, a larger radius can be used for clustering to ensure that the point cloud information covers the entire object surface; for smaller objects, a smaller radius can be used to avoid mistaking multiple objects for the same object. Different coarse filters and clustering algorithms can also be selected and adjusted based on characteristics such as object shape and surface texture, thereby achieving more accurate point cloud processing and capture operations.

[0059] In one embodiment, it also includes: when there are multiple granular geometric filtering structures, determining the application order of the multiple granular geometric filtering structures; arranging the multiple granular geometric filtering structures according to the application order, wherein the neighborhood distance of the granular geometric filtering structure applied first is smaller than the neighborhood distance of the granular geometric filtering structure applied later.

[0060] Determining the order in which multiple granular geometric filtering structures are applied requires considering the size of the neighborhood distance. The neighborhood distance of the granular geometric filtering structure applied first should be smaller, while the neighborhood distance of the granular geometric filtering structure applied later should be larger. This ensures that the abnormal points in the closer neighborhood are removed first, and then the point clouds in the farther neighborhood are clustered, thereby obtaining a more accurate and stable second point cloud information. Furthermore, by first applying the granular geometric filtering structure with a smaller neighborhood distance, the abnormal point cloud in the first point cloud information can be removed more carefully, and the valid point cloud in the closer neighborhood can be retained. The granular geometric filtering structure with a larger neighborhood distance is then applied, which can cluster the point clouds in the farther neighborhood together to form larger clusters, and remove unnecessary small clusters, thereby obtaining the target point cloud.

[0061] In this order, filtering with smaller neighborhood distances first removes most outliers, and then filtering with larger neighborhood distances clusters the valid point clouds into target point clouds. This processing order maximizes the retention of target point clouds while removing unnecessary noise and outliers, improving the quality and accuracy of point cloud data.

[0062] Furthermore, it includes three different levels of anomaly filtering algorithms and clustering algorithms. By setting the radius distance between the point cloud and the surrounding neighborhood as the level division, points with farther neighbors are deleted. The neighborhood distances set in the first, second, and third levels of outlier filtering are gradually increased. This ensures that more obvious abnormal point clouds are effectively removed during the initial anomaly filtering, and gradually reduces the filtering effect as the filtering increases, ensuring that the geometric structure of the target point cloud itself can be effectively preserved. The filtered point cloud is clustered and the largest cluster is selected as the target point cloud, thereby further eliminating the outlier areas generated during the filtering process.

[0063] Based on any of the above embodiments, multiple grasping postures for grasping the object to be grasped are determined according to the second point cloud information, specifically including: segmenting the second point cloud information based on the geometric segmentation structure to determine the target point cloud; determining that the number of target point clouds is less than the stopping threshold corresponding to multiple grasping postures.

[0064] When determining multiple grasping postures, the second point cloud is first separated from the first point cloud to determine the target point cloud. Once the second point cloud is obtained, it must be separated from the first point cloud. This process typically uses segmentation and extraction techniques based on Euclidean distance or normal vectors to determine the target point cloud. Once the target point cloud is determined, the robotic arm controller can calculate multiple possible grasping postures. These postures are calculated based on parameters such as the position, posture, and size of the target point cloud to ensure the robotic arm can safely and effectively grasp the object.

[0065] The second point cloud information is segmented based on the geometric segmentation structure to determine the target point cloud, including: obtaining the geometric segmentation structure and the clustering structure; separating the second point cloud information according to the geometric segmentation structure to determine the separated point cloud information; clustering the separated point cloud information according to the clustering structure to determine the largest cluster as the target point cloud.

[0066] Determining the target point cloud is primarily accomplished through segmentation and clustering algorithms. Specifically, a geometric segmentation structure and a clustering structure are first obtained. The geometric segmentation structure can be a set of predefined geometric features or segmentation algorithms used to segment the point cloud; the clustering structure can be a set of predefined clustering algorithms or parameter settings used to cluster the point cloud. The selected geometric segmentation structure is then used to segment the second point cloud information, resulting in separate point cloud sets. The segmentation process can operate based on the geometric features of the point cloud (e.g., distance, normal vector, etc.) to divide the point cloud into different segmentation clusters. For each separate point cloud set, the selected clustering algorithm or parameter settings are applied to cluster the point cloud. The clustering algorithm can group the point clouds based on their location, density, features, etc., to form different clusters. Based on the clustering results, the cluster with the largest number of point clouds is determined as the target point cloud. By comparing the number of point clouds in each cluster, the cluster with the largest number of point clouds is selected as the point cloud set representing the target object.

[0067] Through the above steps, multiple point cloud sets can be separated from the second point cloud information, and the largest cluster among them can be determined as the target point cloud through clustering operation to facilitate subsequent grasping operations.

[0068] When calculating grasping postures, it is important to set a stopping threshold to prevent the calculation process from being too time-consuming. When the number of calculated grasping postures reaches the stopping threshold, the calculation can be stopped and the best grasping posture can be selected for subsequent robot arm control operations.

[0069] In summary, determining multiple grasping postures for grasping the object to be grasped based on the second point cloud information can provide an important reference basis for subsequent robotic arm control and object grasping operations.

[0070] Determining multiple grasping postures corresponding to the number of target point clouds being less than a stopping threshold includes: determining direction vectors of multiple target point clouds; determining a matrix corresponding to the direction vectors; determining a grasping position according to the center of mass position of each target point cloud; determining a grasping posture corresponding to the target point cloud according to the grasping position and the matrix; determining the number of target point clouds; iteratively determining the target point clouds and the grasping posture until the number of target point clouds is less than the stopping threshold, stopping the iteration, and determining multiple grasping postures.

[0071] For each target point cloud, calculate its normal vector or surface normal as the direction vector of the target point cloud. Calculate the rotation matrix for the direction vector of each target point cloud, which describes the orientation of the robotic arm during grasping. Calculate the center of mass position of each target point cloud, that is, the average position of all points in the point cloud, as the grasping position of the target point cloud. Combine the grasping position and the corresponding rotation matrix to determine the posture of the robotic arm during grasping, including position and orientation. Count the number of target point clouds and determine whether it is less than the stopping threshold. If the number of target point clouds is greater than the stopping threshold, continue to iterate for the next round of point cloud segmentation and grasping posture determination, repeat the above steps, if the number of target point clouds is less than the stopping threshold, stop the iteration and determine multiple grasping postures.

[0072] Through the above steps, the number of target point clouds and the corresponding grasping postures can be iteratively determined until the number of target point clouds is less than the stopping threshold, thereby obtaining multiple grasping postures that meet the requirements.

[0073] Determining the direction vector of the target point cloud includes: determining the direction vectors of multiple surface normals in the target point cloud; determining the average value of the direction vectors of the multiple surface normals corresponding to one target point cloud, and using the average value as the direction vector of the target point cloud.

[0074] For a target point cloud, calculate the direction vectors of multiple surface normals. Each point has a normal vector that represents the direction of the surface normal at that point. Collect multiple surface normal vectors from the target point cloud. Average the collected surface normal vectors and calculate their average value. Use the average normal vector as the direction vector of the target point cloud.

[0075] Through the above steps, the direction vectors of multiple surface normals in the target point cloud can be calculated, and their average value is used as the direction vector of the target point cloud. In this way, a direction vector representing the overall orientation of the target point cloud can be obtained.

[0076] Determining a matrix corresponding to the direction vector includes: determining a corresponding Euler angle according to the direction vector; and determining a matrix according to a plurality of Euler angles.

[0077] When determining the matrix, first calculate the corresponding Euler angles based on the given direction vector. Euler angles are a common way to describe rotations, typically expressed as three angles. Then, based on these Euler angles, the rotation matrix is ​​determined. The rotation matrix is ​​a 3×3 matrix used to represent rotational transformations in three-dimensional space. Based on the obtained Euler angles, calculations can be performed using the rotation matrix formula.

[0078] It's important to note that the specific method for converting Euler angles to rotation matrices depends on the coordinate system and rotation order used. Common rotation orders include ZYX, XYZ, and YXZ, while common coordinate systems include Euler coordinates and rotation matrix coordinates. Based on your specific needs, select the appropriate rotation order and coordinate system and use the corresponding conversion formulas for calculations.

[0079] Due to different conventions and representations, there may be many different ways to convert Euler angles to rotation matrices. Therefore, when implementing this, please refer to the relevant functions and conversion methods provided in the math library or documentation used.

[0080] In a specific embodiment, a general object grasping algorithm based on visual geometry is proposed by calculating the grasping pose in 3D space by geometrically structuring the point cloud within the target area, as shown in Figure 8. The RGB and depth images are fed into different structures to ultimately determine the pose of the current object in 3D space. An external 3D camera captures the current scene, obtaining RGB and depth images, denoted by S1 and S2, respectively. S1 and S2 have been registered.

[0081] Specifically, S1 and S2 are sent to model 1 to obtain the target point cloud P1. Model 1 can be a conventional 2D target detection and segmentation network, such as the target detection network model of the Yolo series, the semantic segmentation model of the FCN series, or the instance segmentation model of the MaskRcnn series. The depth information of the target area is obtained by placing the coordinates of the area identified by the detection and segmentation network on S1 on S2, and then the depth information of the area is converted into the target point cloud P1 using the intrinsic parameters of the 3D camera. Model 1 can also be a point cloud recognition and segmentation algorithm, such as a conventional 3D target detection algorithm, an unsupervised clustering algorithm, etc., which can separate the geometric point cloud of the object from the scene point cloud to obtain the target point cloud P1. Model 1 in this application does not limit the type of 2D / 3D target detection and segmentation model used, as long as it can separate the target point cloud.

[0082] The target point cloud P1 is input into the multi-level granularity geometric filtering structure to obtain the filtered target point cloud P2. The multi-level granularity geometric filtering structure includes three different levels of anomaly filtering algorithms and clustering algorithms. By setting the radius distance between the point cloud and the surrounding neighborhood as the level division, points in the farther neighborhood are deleted. The neighborhood distances set in the first, second, and third level outlier filtering are gradually increased, which ensures that the more obvious abnormal point clouds are effectively removed during the initial anomaly filtering, and the filtering effect is gradually reduced as the filtering increases, ensuring that the geometric structure of the target point cloud itself can be effectively retained. The filtered point cloud is clustered and the largest cluster is selected as the target point cloud P2 to further eliminate the outlier areas generated during the filtering process.

[0083] The target point cloud P2 is sent to the automatic refined geometric decomposition structure to obtain multiple grasping postures M1, M2...M for the current object. n Specifically, as shown in Figure 9, the target point cloud P2 is first fed into a geometric segmentation structure, which can be composed of segmentation algorithms such as plane segmentation, Delaunay triangulation segmentation, curvature segmentation, or a mixture of different algorithms. This can separate a portion of the point cloud P2 from the target point cloud, and then the clustering algorithm in the clustering structure is used to select the largest cluster as the candidate target point cloud P2 to be captured. 1 , thereby eliminating the interference of other clusters. Calculate the target point cloud The direction vectors of the m surface normals Then calculate the mean of the direction vectors of the m surface normals as the target point cloud Direction vector The direction vector Convert to rotation Euler angle [α * ,β * ,γ * ] indicates that it is convenient for subsequent operations. At this time, a matrix consisting of k randomly generated Euler angles is introduced:

[0084] Among them, each value in L1 is sampled from the uniform distribution U(-5°,5°). Similarly, for L2······L n , each value is sampled from the uniform distribution U(-5°,5°), and the target point cloud is calculated The center of mass of the object is taken as the grasping position [x1, y1, z1], and the grasping posture of the object is:

[0085] Each row represents a grasping posture of the point cloud in the region, and each region point cloud contains k grasping postures.

[0086] Then the target point cloud Separate the target point cloud P2 and obtain the calibrated point cloud Through the same process as above, the grasping posture is obtained By continuously iterating to obtain new grasping postures and target point clouds, until the separated target point cloud P2 n When the number of points is less than the set threshold, the iteration stops, and n groups of grasping postures {M1, M2, ..., M n The information is fed back to the robotic arm for planning and selecting a more favorable grasping posture to pick up the object.

[0087] According to the embodiments provided in this application, the 3D geometric information of an object's point cloud can be directly used to calculate the grasping posture, eliminating the need for additional grasping posture annotation, significantly saving both the manpower required for annotation and the material resources required for simulation machines. Furthermore, this technical solution utilizes logical deduction, without the use of high-performance computing resources such as GPUs. It is fast, has minimal memory overhead, and is easily ported to devices. It has broad applicability for common object grasp estimation and can be flexibly adapted to existing common recognition algorithms.

[0088] As shown in FIG6 , an embodiment of the present application provides an object grasping device 200 . The object grasping device 200 includes a scene acquisition module 202 , a point cloud acquisition module 204 , a filtering module 206 , a posture determination module 208 , and a grasping module 210 .

[0089] Among them, the scene acquisition module 202 is used to obtain the scene information of the environment in which the object to be grasped is located; the point cloud acquisition module 204 is used to determine the first point cloud information corresponding to the object to be grasped based on the scene information; the filtering module 206 is used to filter the first point cloud information and determine the filtered second point cloud information; the posture determination module 208 is used to determine multiple grasping postures for grasping the object to be grasped based on the second point cloud information; the grasping module 210 is used to determine the target posture among multiple grasping postures, and control the robotic arm to grasp the object to be grasped according to the target posture.

[0090] Before grasping an object, this embodiment requires obtaining scene information about the environment in which the object to be grasped resides. This information may include the object's position, posture, size, and the locations of surrounding objects. By processing the acquired scene information, a first point cloud corresponding to the object to be grasped can be obtained. It can be understood that a point cloud is a collection of three-dimensional coordinate points that can be used to represent information such as the object's shape and position. Based on the first point cloud information, filtering operations can be performed, such as removing point clouds containing irrelevant objects, to obtain a filtered second point cloud information. Based on the second point cloud information, multiple possible grasping postures can be determined. These grasping postures may include information such as the position and posture of the robot arm's end effector, which is used to control the robot arm for grasping. Ultimately, an optimal target posture is selected from the multiple possible grasping postures and used to control the robot arm for grasping. This process typically takes into account factors such as the object's position, posture, and relationship to surrounding objects. Once the target posture is determined, the robot arm can be moved to the appropriate position by controlling the movement of the end effector and adjusting parameters such as the posture to ultimately complete the grasping operation of the object to be grasped. Generally speaking, this object grasping method uses robot vision and intelligent control technology, combined with point cloud information processing and posture adjustment technologies, to achieve automated grasping of the object to be grasped.

[0091] The object grasping device 200 in the embodiment of the present application can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc. The non-mobile electronic device can be a server, a network attached storage 212 (NAS), a personal computer (PC), a television (TV), an ATM, or an kiosks, etc., which are not specifically limited in the embodiment of the present application.

[0092] The object grasping device 200 in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0093] The object grasping device 200 provided in the embodiment of the present application can implement each process implemented in the method embodiments of Figures 1 to 5. To avoid repetition, they are not described here.

[0094] Optionally, as shown in Figure 7, an embodiment of the present application also provides a robot 100, including a processor 214, a memory 212, and a program or instruction stored in the memory 212 and executable on the processor 214. When the program or instruction is executed by the processor 214, each process of the embodiment of the above-mentioned object grasping method is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0095] It should be noted that the robot in the embodiment of the present application includes electronic equipment and non-electronic equipment.

[0096] Among them, the processor 214 is used to obtain scene information of the environment in which the object to be grasped is located; determine the first point cloud information corresponding to the object to be grasped based on the scene information; filter the first point cloud information to determine the filtered second point cloud information; determine multiple grasping postures for grasping the object to be grasped based on the second point cloud information; determine the target posture among the multiple grasping postures, and control the robotic arm to grasp the object to be grasped according to the target posture.

[0097] With the above solution, before grasping an object, it is necessary to first obtain scene information about the environment in which the object to be grasped resides. This information may include the object's position, posture, size, and the positions of surrounding objects. By processing the acquired scene information, a first point cloud corresponding to the object to be grasped can be obtained. It can be understood that a point cloud is a collection of three-dimensional coordinate points that can be used to represent information such as the object's shape and position. Based on the first point cloud information, some filtering operations can be performed, such as removing point clouds of irrelevant objects, to obtain a filtered second point cloud information. Based on the second point cloud information, multiple possible grasping postures can be determined. These grasping postures may include information such as the position and posture of the robot arm's end effector, which is used to control the robot arm for grasping. Finally, an optimal target posture is selected from the multiple possible grasping postures and used to control the robot arm for grasping. This process typically takes into account factors such as the object's position, posture, and relationship to surrounding objects. Once the target posture is determined, the robot arm can be moved to the appropriate position by controlling the movement of the end effector and adjusting parameters such as the posture to ultimately complete the grasping operation. Generally speaking, this object grasping method uses robot vision and intelligent control technology, combined with point cloud information processing and posture adjustment technologies, to achieve automated grasping of the object to be grasped.

[0098] Optionally, the processor 214 is further configured to implement the following steps: obtaining an intrinsic parameter matrix of the three-dimensional camera; and determining first point cloud information according to the intrinsic parameter matrix, image information, and depth information.

[0099] Optionally, the processor 214 is also used to obtain at least one granular geometric filtering structure, each granular geometric filtering structure includes an anomaly filtering algorithm and a clustering algorithm; remove abnormal point clouds in the first point cloud information according to the anomaly filtering algorithm, and cluster the filtered first point cloud information according to the clustering algorithm; determine the point cloud information of the largest cluster after clustering as the second point cloud information.

[0100] Optionally, the processor 214 is further configured to separate the second point cloud information from the first point cloud information to determine a target point cloud; and determine that the number of target point clouds is less than a plurality of grasping postures corresponding to a stop threshold.

[0101] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned object grasping method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0102] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a readable storage medium such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0103] The methods may be implemented in a variety of different ways depending on the specific features and / or example applications. For example, the methods may be implemented through a combination of hardware, firmware, and / or software. For example, in a hardware implementation, the processor may be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, electronic devices, other device units for performing the above functions, and / or combinations thereof.

[0104] A computer-readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. A computer-readable storage medium can be, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: a portable computer floppy disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory card, a floppy disk, an encoding mechanical device (such as a punched card or a groove with a raised structure on which instructions are recorded), and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be understood as a transmission signal per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium, or electrical signals transmitted through wires.

[0105] An embodiment of the present application further provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned object grasping method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0106] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0107] An embodiment of the present application further provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned object grasping methods when executed by a processor.

[0108] According to the embodiments of the object grasping method, object grasping device, robot, readable storage medium and chip of the present application, when performing posture calculation, the 3D geometric information of the object point cloud is directly used to implement it, and there is no need to mark the grasping posture of the object in 3D space, which greatly saves manpower for marking and material resources of simulation machines.

[0109] It should be noted that the solution designed in this application can be applied to the deployment and use of various product platforms, such as intelligent robots, mobile phones, surveillance cameras, smart appliances, etc. The technical solution provided in this application can be applied to various sensors as input, such as RGB-D cameras, RGB cameras + lidar, RGB cameras + millimeter-wave radars, RGB cameras + infrared cameras, and other components that can capture RGB images and perform ranging imaging.

[0110] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0111] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0112] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A method for grasping an object, wherein: include: Obtain scene information of the environment where the object to be grasped is located; Determine first point cloud information corresponding to the object to be grasped according to the scene information; Determining a plurality of grasping postures for grasping the object to be grasped according to the first point cloud information; A target posture is determined among the multiple grasping postures, and the robot arm is controlled to grasp the object to be grasped according to the target posture.

2. The object grasping method according to claim 1, wherein: The step of obtaining scene information of the environment in which the object to be grasped is located includes: The scene information of the current environment is acquired through a three-dimensional camera, wherein the scene information includes image information and depth information of the scene, and the image information and the depth information are information after registration.

3. The object grasping method according to claim 2, wherein: The determining, according to the scene information, first point cloud information corresponding to the object to be grasped includes: The first point cloud information is determined according to the intrinsic parameter matrix of the three-dimensional camera, the image information and the depth information.

4. The object grasping method according to claim 3, wherein: Determining the first point cloud information according to the intrinsic parameter matrix of the three-dimensional camera, the image information and the depth information includes: Obtain target detection network model and segmentation network model; Identify the image information by using the target detection network model and the segmentation network model to determine coordinate information of the first area; determining depth information corresponding to the first area according to the coordinate information of the first area; The depth information of the first area is converted into first point cloud information according to the intrinsic parameter matrix.

5. The object grasping method according to any one of claims 2 to 4, wherein: The method further includes: filtering the first point cloud information to determine filtered second point cloud information; The second point cloud information after determining the filtering includes: Acquire at least one granular geometric filtering structure, each of which includes an anomaly filtering algorithm and a clustering algorithm; removing abnormal point clouds in the first point cloud information according to the abnormal filtering algorithm, and clustering the filtered first point cloud information according to the clustering algorithm; The point cloud information of the largest cluster after clustering is determined as the second point cloud information.

6. The object grasping method according to claim 5, wherein: There are multiple granular geometric filtering structures, and the point cloud in each granular geometric filtering structure has a different radius distance from the surrounding neighborhood.

7. The object grasping method according to claim 5 or 6, wherein: Also includes: In case that there are multiple granular geometric filtering structures, determining an application order of the multiple granular geometric filtering structures; The plurality of granular geometric filtering structures are arranged according to the application order, wherein the neighborhood distance of the granular geometric filtering structure applied first is smaller than the neighborhood distance of the granular geometric filtering structure applied later.

8. The object grasping method according to claim 5, wherein: The determining, according to the first point cloud information, a plurality of grasping postures for grasping the object to be grasped specifically includes: Segmenting the second point cloud information based on the geometric segmentation structure to determine the target point cloud; It is determined that the number of the target point cloud is less than a plurality of grasping postures corresponding to a stop threshold.

9. The object grasping method according to claim 8, wherein: The step of segmenting the second point cloud information based on the geometric segmentation structure to determine the target point cloud includes: Obtain geometric segmentation structure and clustering structure; Separating the second point cloud information according to the geometric segmentation structure to determine separated point cloud information; The separated point cloud information is clustered according to the clustering structure, and the largest cluster is determined as the target point cloud.

10. The object grasping method according to claim 9, wherein: The step of determining that the number of the target point cloud is less than a plurality of grasping postures corresponding to a stop threshold comprises: Determining direction vectors of a plurality of said target point clouds; determining a matrix corresponding to the direction vector; Determining a grasping position according to the centroid position of each target point cloud; Determining a grasping posture corresponding to the target point cloud according to the grasping position and the matrix; Determining the number of the target point cloud; The target point cloud and the grasping posture are determined iteratively until the number of the target point cloud is less than a stopping threshold, then the iteration is stopped and a plurality of the grasping postures are determined.

11. The object grasping method according to claim 10, wherein: The determining of the direction vectors of the plurality of target point clouds comprises: Determining direction vectors of a plurality of surface normals in the target point cloud; An average value of direction vectors corresponding to a plurality of surface normals of the target point cloud is determined, and the average value is used as the direction vector of the target point cloud.

12. The object grasping method according to claim 10, wherein: The determining of a matrix corresponding to the direction vector comprises: Determine the corresponding Euler angle according to the direction vector; A matrix is ​​determined according to the plurality of Euler angles.

13. A robot, wherein: The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the object grasping method as claimed in any one of claims 1 to 12.

14. An object grasping device, wherein: include: A scene acquisition module is used to obtain scene information of the environment where the object to be grasped is located; A point cloud acquisition module, used to determine first point cloud information corresponding to the object to be grasped according to the scene information; a posture determination module, used to determine a plurality of grasping postures for grasping the object to be grasped according to the first point cloud information; The grasping module is used to determine a target posture among the multiple grasping postures, and control the robot arm to grasp the object to be grasped according to the target posture.

15. A readable storage medium, wherein: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the object grasping method according to any one of claims 1 to 12 are implemented.

16. A chip, wherein: The chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the steps of the object grasping method according to any one of claims 1 to 12.

17. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the steps of the object grasping method according to any one of claims 1 to 12 are implemented.

Images