A robot end gripping pose generation method combining vision and a small amount of teaching

Abstract

The present application belongs to the technical field of robot grabbing, and discloses a robot end grabbing pose generation method combining vision and a small amount of teaching, comprising: obtaining an original point cloud image through a depth camera, segmenting each object point cloud through background subtraction and connected domain corrosion, and constructing an object coordinate system by principal component analysis; obtaining a transformation matrix of the camera and the robot base through hand-eye calibration, and obtaining a teaching grabbing pose of the object relative to the robot end by manual teaching; for a rotating body object, generating a plurality of candidate poses by discretely rotating the teaching pose around the rotation axis; for a non-rotating body object, generating a candidate pose on the opposite side by mirror transformation of the teaching pose; and finally screening feasible poses by inverse kinematics and selecting the optimal one to perform grabbing. The present application can expand the feasible grabbing space of the robot arm with only a small amount of teaching, significantly improve the grabbing success rate, and is suitable for structured industrial scenes with high degree of freedom and dexterous hands.

Description

Technical Field

[0001] This invention belongs to the field of robot operation technology, and more specifically, relates to a method for generating robot end-effector grasping pose by combining vision and minimal teaching. Background Technology

[0002] In existing industrial robot grasping applications, the methods for determining the grasping pose of the end effector mainly include the following two categories.

[0003] The first type of method uses a conveyor to physically limit the object being operated on, ensuring that the object always reaches the same fixed position. The end effector of the robotic arm then repeatedly moves to the preset fixed pose to complete the operation. This method relies on external mechanical constraints and lacks the ability to adapt to changes in the object's pose.

[0004] The second type of method uses vision and other sensing devices to identify the pose of specific types of objects. The operator determines the grasping pose by manually adjusting the relative pose offset between the robot arm's end effector and the object multiple times. However, because the SE(3) space in which the robot pose is located has a nonlinear manifold structure, it is difficult to express and adjust intuitively, and the accuracy of manually setting the offset is low. This method is usually only suitable for low-degree-of-freedom end effectors such as grippers to perform low-precision grasping tasks, and it is difficult to adapt to the high-precision operation required by high-degree-of-freedom end effectors such as dexterous hands.

[0005] On the other hand, learning-based grasping pose generation methods have been proposed in recent years. These methods collect a large amount of grasping data and train it over a long period to map perceptual information about complex objects to reasonable grasping poses suitable for specific end effectors. While these methods possess a certain generalization ability and can generate feasible grasping poses for unseen objects, they suffer from the following drawbacks: grasping success rates are mostly below 70%; data acquisition and annotation costs are high; some data needs to be reconstructed and the network fine-tuned after changing the end effector; computation time is long and real-time performance is poor; high requirements for device computing power are imposed; and the generative grasping results are uncertain, making it difficult to meet the stability and reliability requirements of structured, repetitive industrial scenarios.

[0006] In recent years, with embodied intelligence being established as a strategic development direction in the field of robotics, higher demands have been placed on humanoid robot manipulation tasks. It is foreseeable that dexterous hands with higher degrees of freedom will gradually replace traditional grippers and become the mainstream end effector. The significantly increased degrees of freedom of dexterous hands lead to a sharp rise in the spatial complexity of grasping planning. The original method of manually adjusting and setting biases is no longer applicable because dexterous hands employ a multi-point contact grasping strategy, requiring much higher precision in grasping pose than grippers, thus significantly reducing the practically feasible end-effector grasping space. Under these conditions, if the robotic arm still uses a manually set single grasping pose, it often cannot obtain a kinematically feasible solution, leading to grasping task failure.

[0007] In summary, existing technologies have significant shortcomings in handling high-precision grasping pose planning for high-degree-of-freedom dexterous hands. There is an urgent need for a robot end effector grasping pose generation method that combines vision and minimal teaching, which can adapt to the characteristics of dexterous hands, has high precision, good real-time performance, and is applicable to structured industrial scenarios. Summary of the Invention

[0008] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a method for generating robot end-effector grasping poses by combining vision and a small amount of teaching. The purpose is to enrich the feasible grasping postures in robot operation tasks, thereby solving the technical problem that the few existing grasping postures exceed the operating space of commonly used six-degree-of-freedom robotic arms, resulting in grasping task failure.

[0009] To achieve the above objectives, according to one aspect of the present invention, a method for generating robot end-effector grasping pose by combining vision and minimal teaching is provided, comprising the following steps: S1. Take an original point cloud image containing the grasped object and its surrounding environment using a depth camera, perform image processing on the original point cloud image to obtain a segmented point cloud of the grasped object, and construct an object coordinate system. S2. Obtain the homogeneous transformation matrix between the camera coordinate system and the robot base coordinate system through hand-eye calibration, and based on the homogeneous transformation matrix, use the teaching method to obtain the homogeneous transformation representation of the object coordinate system in the robot end-effector coordinate system; S3. Based on the type of object to be grasped, and using the homogeneous transformation representation of the object's coordinate system in the robot's end-effector coordinate system, different pose generation methods are employed to generate multiple candidate grasping poses. S4. Select the pose with feasible robot kinematics solution from multiple candidate grasping poses, and select the pose with the minimum loss as the output optimal robot end-effector grasping pose according to the preset loss function.

[0010] Preferably, step S1 includes: S11. Add restrictions to the original point cloud coordinate range to retain the point cloud information of the grabbed object and the area of ​​the desktop it is located in; S12. Before placing the object to be grasped, take a picture of the original point cloud containing the desktop and the surrounding environment. After filtering out the messy point cloud through the aforementioned constraints, obtain the desktop point cloud set. S13. After placing the object to be grasped, an original point cloud image containing the object to be grasped, the desktop and the surrounding environment is captured. After filtering out the cluttered point cloud by the aforementioned limiting conditions, the point cloud sets of the desktop and the object to be grasped are obtained. Then, the point cloud set of the object to be grasped is subtracted from the point cloud set of the desktop to obtain the original point cloud set of the object to be grasped. S14. Project the original point cloud of the captured object onto the uv two-dimensional plane of the camera coordinate system, perform connected component erosion operation, and then remap the remaining point cloud back to the original space to obtain the segmented multi-object point cloud. S15. Constructing the object coordinate system: For each object's point cloud set, take the average value of all point cloud coordinates as the origin, uniformly set the object's z-axis to be perpendicular to the table, and use the first and second principal axes obtained by principal component analysis after projecting the point cloud onto the table plane as the object's x-axis and y-axis, respectively, thereby obtaining the object's pose in the camera coordinate system.

[0011] Preferably, the specific process of hand-eye calibration in step S2 is as follows: S21. Fix the chessboard to the end of the robotic arm, and adjust the pose of the end of the robotic arm so that the chessboard is completely in the camera's field of view. Record the current image and the pose of the end of the robotic arm. S22. Repeat step S21 until data from N different poses are collected; S23. Based on the collected data, transform the hand-eye calibration problem into... AX=YB The model employs a geometric stochastic global optimization method to optimize the objective function, and calculates the homogeneous transformation matrix of the chessboard coordinate system in the robot's base coordinate system. Its expression is as follows:

[0012] in, A、 This represents the pose of the robotic arm's end effector within the base system. X、 The pose representation of the chessboard grid under the end effector of the robotic arm. Y、 This represents the pose of the camera system within the robotic arm's base system. B、 The pose representation of the chessboard grid under the camera system; Substituting the multiple sets of pose data obtained in S22 into the objective function yields:

[0013] in, Denotes the Frobenius norm. For the first i The pose representation of the robotic arm's end effector in the base coordinate system during the next measurement. For the first i The pose representation of the checkerboard grid under the end effector of the robotic arm during the measurement, SE(3) is a special Euclidean group used to describe the six degrees of freedom motion of a rigid body in three-dimensional space.

[0014] Preferably, the teaching method in step S2 is as follows: S24. Within the depth camera's sensing range, manually drag the robotic arm to the object to be grasped, and record the pose of the robotic arm's end effector relative to the base coordinate system at this moment. ; S25. Move the robotic arm out of the sensing range, and obtain the pose of the grasped object relative to the camera coordinate system at the current moment through step S1. ; S26. Substitute the poses obtained in steps S23, S24, and S25 into the following formula to obtain the homogeneous transformation representation of the grasped object coordinate system in the robot's end effector coordinate system. :

[0015] in, This represents the pose of the robotic arm's end effector relative to the base coordinate system. This represents the pose of the camera system within the robotic arm's base system. To capture the pose of an object relative to the camera coordinate system at the current moment; And save the homogeneous transformation representation of the calculated object coordinate system in the robot's end effector coordinate system. This serves as a teaching grasping pose for the object.

[0016] Preferably, in step S3, when the grasped object is a rotating body, a 5D pose generation method is used, specifically including: S31. Calculate the pose of the robotic arm's end effector relative to the base coordinate system under the guidance of the current teaching information. The rotation axis is then extracted as the z-axis of the object's coordinate system; the specific calculation formula is as follows: ; S32. Rotate the initial grasping pose around the rotation axis by discrete angles. θ Multiple candidate grasping poses are generated, and the rotated grasping poses are obtained. Calculated using the following formula:

[0017]

[0018]

[0019] in, This represents the rotational attitude of the robot's end effector within the base system. Let this be the displacement of the robot's end effector within the base system. This represents the pose of the robot's end effector within the pedestal system. For the axis of rotation about the base coordinate system Rotation The rotation representation obtained from the angle, This represents the rotational attitude of the newly generated candidate robot end effector in the base coordinate system. Let this be the displacement representation of the newly generated robot end effector in the base system. This represents the pose of the newly generated robot end effector in the pedestal system.

[0020] Preferably, in step S3, when the object to be grasped is a non-rotating body, a 6D pose generation method is used, specifically including: The pose of the grasped object relative to the end effector of the robotic arm obtained through teaching. Perform the transformation to generate the corresponding reverse side grasping pose. Reverse its position relative to the x-axis and y-axis of its orientation, using the following transformation formula: , Then, the transformed pose is converted to the grasping pose of the robotic arm's end effector in the base coordinate system. The specific formula is as follows:

[0021] Preferably, step S4 specifically includes: inputting all generated candidate grasping poses into the inverse kinematics solver of the robotic arm for feasibility verification, retaining poses with kinematically feasible solutions, and then calculating the weighted loss between each feasible pose and the anthropomorphic initial arm joint angle according to a preset loss function, selecting the pose with the minimum loss as the optimal grasping pose. The loss function formula is as follows: as follows:

[0022] in, The initial arm posture designed for humanoid design. The current joint angle of the robotic arm. The weights are assigned to each joint angle.

[0023] According to another aspect of the present invention, a robot end effector grasping pose generation system combining vision and minimal teaching is provided, for implementing the above-described robot end effector grasping pose generation method combining vision and minimal teaching, comprising: The visual perception module is used to capture raw point cloud images containing the grasping object and its surrounding environment using a depth camera, and to perform point cloud segmentation and coordinate system construction on the raw point cloud images to obtain the pose of the object in the camera coordinate system. The teaching module is used to obtain the homogeneous transformation matrix between the camera coordinate system and the robot base coordinate system through hand-eye calibration, and to obtain the homogeneous transformation representation of the object coordinate system in the robot end-effector coordinate system through manual teaching. The pose generation module is used to generate multiple candidate grasping poses based on the homogeneous transformation representation obtained by the teaching module, according to the type of the grasped object, and using the corresponding pose generation method to generate multiple candidate grasping poses. The grasping control module is used to select the optimal grasping pose from the multiple candidate grasping poses and control the robotic arm to grasp the object to be grasped.

[0024] According to another aspect of the present invention, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described method for generating a robot end effector grasping pose by combining vision and minimal teaching.

[0025] According to another aspect of the present invention, a computer-readable storage medium is provided thereon storing a computer program that, when executed by a processor, implements the above-described method for generating a robot end effector grasping pose by combining vision and minimal teaching.

[0026] In summary, compared with the prior art, the robot end-effector grasping pose generation method combining vision and minimal teaching provided by the present invention has the following advantages: 1. Effectively Expanding the Feasible Grasping Space: The grasping pose generation method proposed in this invention can generate multiple candidate poses for both rotating and non-rotating bodies. These candidate poses are all obtained by spatial rotation transformation based on the feasible poses obtained from actual teaching, which greatly increases the probability that the grasping pose executed under the teaching is within the feasible solution space of the robot arm's kinematics. This fundamentally solves the problem that the grasping poses generated by existing methods have no inverse solution, and also solves the problem that a small number of grasping poses exceed the commonly used six-degree-of-freedom robot arm operation space, causing grasping tasks to fail. At the same time, this method can also be used to solve the problem that the deep learning output results exceed the feasible solution space of the robot arm's kinematics, greatly improving the grasping success rate and enriching the feasible grasping poses in robot operation tasks.

[0027] 2. Fast and robust visual perception: The multi-object segmentation and coordinate system establishment method based on cluttered point cloud background filtering and connected component erosion proposed in this invention can quickly filter out cluttered point clouds, segment objects and establish object coordinate systems in complex environments at millisecond speeds. It has good robustness to the environment and extremely fast response performance, and solves the depth jump problem caused by the depth camera mechanism.

[0028] 3. Simple teaching and highly scalable: The teaching method proposed in this invention allows users to quickly build a grasping pose library according to the application scenario; when faced with a new object, the grasping library can be expanded by simply teaching it again, without having to collect a large amount of data and retrain the network as required by deep learning methods, which significantly reduces the cognitive burden and deployment cost for users.

[0029] 4. Solved the problem of end effector coordinate system calibration: This invention indirectly obtains the precise pose of the object relative to the end of the robotic arm through a teaching method, avoiding the difficulty of directly calibrating the complex transformation relationship between the end effector coordinate system and the end of the robotic arm coordinate system, and providing a practical and feasible solution for vision-guided grasping tasks. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the teaching method in an embodiment of the present invention; Figure 2 This is a schematic diagram of a 5D pose generation method for a rotating object in an embodiment of the present invention; Figure 3 This is a schematic diagram of a 6D pose generation method for non-rotating objects in an embodiment of the present invention; Figure 4 This is a flowchart of a robot end-effector grasping pose generation method that combines vision and minimal teaching in an embodiment of the present invention; Figure 5 This is a schematic diagram of the asymmetric chessboard used in the hand-eye calibration of this invention; Figure 6 This is a physical image of a grasping experiment conducted using the UR5 robotic arm in an embodiment of the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0032] This invention uses the UR5 collaborative robotic arm as the research object, equipped with an ORBBEC femto bolt depth camera as a vision sensor, and a five-fingered underactuated dexterous hand as the end effector. The grasping pose generation method of this invention can operate independently or as part of a complex robotic operation task.

[0033] Please see Figure 4 This invention provides a method for generating robot end-effector grasping poses that combines vision and minimal teaching. Based on the ORBBEC femto bolt depth camera and point cloud processing method, object segmentation and localization are achieved. A robotics-based teaching method is used to construct a feasible grasping pose library for multiple objects. A pose generation method based on geometric transformations expands upon a single feasible solution, significantly improving the grasping success rate. Given a scene containing objects and a UR5 robotic arm, the algorithm takes the visual image information of the scene as input and, based on the grasping pose library and pose generation method obtained through teaching, calculates an optimal feasible grasping pose, enabling the UR5 robotic arm to stably grasp objects in this pose. The grasping algorithm can run independently or as part of a complex task. This method for generating robot end-effector grasping poses that combines vision and minimal teaching specifically includes the following steps: S1. Take an original point cloud image containing the grasped object and its surrounding environment using a depth camera, perform image processing on the original point cloud image to obtain a segmented point cloud of the grasped object, and construct an object coordinate system. S2. Obtain the homogeneous transformation matrix between the camera coordinate system and the robot base coordinate system through hand-eye calibration, and based on the homogeneous transformation matrix, use the teaching method to obtain the homogeneous transformation representation of the object coordinate system in the robot end-effector coordinate system; S3. Based on the type of object to be grasped, and using the homogeneous transformation representation of the object's coordinate system in the robot's end-effector coordinate system, different pose generation methods are employed to generate multiple candidate grasping poses. S4. Select the pose with feasible robot kinematics solution from multiple candidate grasping poses, and select the pose with the minimum loss according to the preset loss function as the output optimal robot end-effector grasping pose, which is used as the actual grasping pose to control the robotic arm to grasp the object to be grasped.

[0034] Specifically, step S1 includes: S11. Add restrictions to the original point cloud coordinate range to retain the point cloud information of the grabbed object and the area of ​​the desktop it is located in; S12. Before placing the object to be grasped, take a picture of the original point cloud containing the desktop and the surrounding environment. After filtering out the messy point cloud through the aforementioned constraints, obtain the desktop point cloud set. S13. After placing the object to be grasped, an original point cloud image containing the object to be grasped, the desktop and the surrounding environment is captured. After filtering out the cluttered point cloud by the aforementioned limiting conditions, the point cloud sets of the desktop and the object to be grasped are obtained. Then, the point cloud set of the object to be grasped is subtracted from the point cloud set of the desktop to obtain the original point cloud set of the object to be grasped. S14. Project the original point cloud of the captured object onto the uv two-dimensional plane of the camera coordinate system, perform connected component erosion operation, and then remap the remaining point cloud back to the original space to obtain the segmented multi-object point cloud. S15. Constructing the object coordinate system: For each object's point cloud set, take the average value of all point cloud coordinates as the origin, uniformly set the object's z-axis to be perpendicular to the table, and use the first and second principal axes obtained by principal component analysis after projecting the point cloud onto the table plane as the object's x-axis and y-axis, respectively, thereby obtaining the object's pose in the camera coordinate system.

[0035] Further, please refer to Figure 5 The vision system camera is fixed to the robotic arm base, and the checkerboard pattern is fixed to the end effector of the robotic arm, thus fixing the coordinate relationships between the camera and the robotic arm base, and between the checkerboard pattern and the end effector. Hand-eye calibration is performed using the black and white checkerboard pattern, as shown in the image. Figure 5 As shown, the printed chessboard grid is set to a length of 1.504cm. Note that it should be exported at the original size during printing, and an asymmetrical chessboard grid is generally used to easily distinguish the origin position of the chessboard grid. The entire chessboard grid is placed completely in the camera's observation frame by adjusting the robot's angle to achieve stable chessboard grid recognition.

[0036] By repeatedly adjusting the pose of the robotic arm's end effector, capturing and recording the current image and the end effector's posture, the data was imported into MATLAB and the camera calibrator application was used to obtain the extrinsic parameters for camera calibration. Currently, the representation of the robotic arm's end effector under the base and the representation of the checkerboard coordinate system under the camera coordinate system are known. However, the representations of the camera coordinate system under the base coordinate system and the checkerboard coordinate system under the end effector coordinate system are unknown. Therefore, there are two methods to represent the checkerboard in the base coordinate system:

[0037]

[0038] in, This represents the pose of the robotic arm's end effector within the base system. The pose representation of the chessboard grid under the end effector of the robotic arm. This represents the pose of the camera system within the robotic arm's base system. The pose representation of the chessboard grid under the camera system.

[0039] The specific process of hand-eye alignment in step S2 is as follows: S21. Fix the chessboard to the end of the robotic arm, and adjust the pose of the end of the robotic arm so that the chessboard is completely in the camera's field of view. Record the current image and the pose of the end of the robotic arm. S22. Repeat step S21 until data from N different poses are collected; S23. Based on the collected data, transform the hand-eye calibration problem into... AX=YB The model employs a geometric stochastic global optimization method to optimize the objective function, and calculates the homogeneous transformation matrix of the chessboard coordinate system in the robot's base coordinate system. Its expression is as follows:

[0040] in, This represents the pose of the robotic arm's end effector within the base system. A , The pose representation of the chessboard grid under the end effector of the robotic arm, i.e. X , The pose representation of the camera system within the robotic arm base system, i.e. Y , The pose representation of the checkerboard grid under the camera system, i.e. B ; Substituting the multiple sets of pose data obtained in S22 into the objective function yields:

[0041] in, Denotes the Frobenius norm. for That is, the first i The pose representation of the robotic arm's end effector in the base coordinate system during the next measurement. for That is, the first i The pose representation of the checkerboard pattern under the end effector of the robotic arm during this measurement is shown above. These are data corresponding to multiple sets of different poses collected by S21. for , for All are fixed values ​​to be solved. SE(3) is a special Euclidean group used to describe the six degrees of freedom motion of a rigid body in three-dimensional space.

[0042] Further, please refer to Figure 1 This is a schematic diagram of the teaching method of the present invention, illustrating the process of manually dragging a robotic arm to record the end-effector pose, combining this with visual detection of the object's pose, and then calculating the object's pose relative to the end-effector. The process. The teaching method in step S2 is as follows: S24. Within the depth camera's sensing range, manually drag the robotic arm to the object to be grasped, and record the pose of the robotic arm's end effector relative to the base coordinate system at this moment. ; S25. Move the robotic arm out of the sensing range, and obtain the pose of the grasped object relative to the camera coordinate system at the current moment through step S1. ; S26. Substitute the poses obtained in steps S23, S24, and S25 into the following formula to obtain the homogeneous transformation representation of the grasped object coordinate system in the robot's end effector coordinate system. :

[0043] in, This represents the pose of the robotic arm's end effector relative to the base coordinate system. This represents the pose of the camera system within the robotic arm's base system. To capture the pose of an object relative to the camera coordinate system at the current moment; And save the homogeneous transformation representation of the calculated object coordinate system in the robot's end effector coordinate system. This serves as a teaching grasping pose for the object.

[0044] Further, please refer to Figure 2 This invention provides a schematic diagram of a 5D pose generation method for rotating objects, illustrating the principle of generating multiple candidate poses by discretely rotating the taught grasping pose around the object's z-axis (rotation axis). In step S3, when the grasped object is a rotating body, the 5D pose generation method is employed, specifically including: S31. Calculate the pose of the robotic arm's end effector relative to the base coordinate system under the guidance of the current teaching information. The rotation axis is then extracted as the z-axis of the object's coordinate system; the specific calculation formula is as follows: ; S32. Rotate the initial grasping pose around the rotation axis by discrete angles. θ Multiple candidate grasping poses are generated, and the rotated grasping poses are obtained. Calculated using the following formula:

[0045]

[0046]

[0047] in, This represents the rotational attitude of the robot's end effector within the base system. Let this be the displacement of the robot's end effector within the base system. This represents the pose of the robot's end effector within the pedestal system. For the axis of rotation about the base coordinate system Rotation The rotation representation obtained from the angle, This represents the rotational attitude of the newly generated candidate robot end effector in the base coordinate system. Let this be the displacement representation of the newly generated robot end effector in the base system. This represents the pose of the newly generated robot end effector in the pedestal system.

[0048] Further, please refer to Figure 3 This invention provides a schematic diagram of a 6D pose generation method for non-rotating objects, illustrating the principle of generating a reverse-side grasping pose through mirror transformation. In step S3, when the grasped object is a non-rotating body, the 6D pose generation method is employed, specifically including: The pose of the grasped object relative to the end effector of the robotic arm obtained through teaching. Perform the transformation to generate the corresponding reverse side grasping pose. Reverse its position relative to the x-axis and y-axis of its orientation, using the following transformation formula:

[0049] Then, the transformed pose is converted to the grasping pose of the robotic arm's end effector in the base coordinate system. The specific formula is as follows: .

[0050] Further, please refer to Figure 6 This is a physical image of the UR5 robotic arm used in a grasping experiment in this embodiment of the invention. Step S4 includes: inputting all generated candidate grasping poses into the inverse kinematics solver of the robotic arm for feasibility verification, retaining poses with kinematically feasible solutions, and then calculating the weighted loss between each feasible pose and the anthropomorphic initial arm joint angle according to a preset loss function. The pose with the minimum loss is selected as the optimal grasping pose. The specific loss function... The formula is as follows:

[0051] in, , , The initial arm posture designed for humanoid design. The current joint angle of the robotic arm. The weights are assigned to each joint angle.

[0052] Specifically, all generated grasping poses are input into the inverse kinematics solver of the robotic arm to check if they are within the reachable space of the robotic arm. Then, the most suitable pose is selected from the candidate poses that meet the conditions according to custom criteria for grasping. For example, if an initial anthropomorphic arm shape is set, all feasible poses are subtracted from the joint angles of the initial anthropomorphic arm shape and weighted. The loss weight can be set higher for the upper arm and lower for the forearm. The pose with the smallest loss is selected as the optimal grasping pose to complete the grasping task.

[0053] The point cloud filtering, segmentation, and object coordinate system establishment method proposed in this invention solves the depth jump problem caused by the depth camera mechanism. It can quickly and robustly filter out cluttered point clouds, segment each task object, and establish an object coordinate system in complex environments. Furthermore, the teaching and pose generation method of this invention is not only applicable to two-finger grippers but also fully compatible with dexterous hands with higher degrees of freedom. For dexterous hands, since the grasping strategy involves multi-point contact, the pose accuracy requirements are higher. However, this method, through teaching-based candidate pose generation, ensures that all candidate poses have kinematically feasible solutions and that the accuracy comes from actual teaching, thus still working effectively. When changing the end effector, only one re-teaching is required, without modifying the algorithm framework, demonstrating good versatility and transferability.

[0054] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating robot end-effector grasping pose by combining vision and minimal teaching, characterized in that: Includes the following steps: S1. Take an original point cloud image containing the grasped object and its surrounding environment using a depth camera, perform image processing on the original point cloud image to obtain a segmented point cloud of the grasped object, and construct an object coordinate system. S2. Obtain the homogeneous transformation matrix between the camera coordinate system and the robot base coordinate system through hand-eye calibration, and based on the homogeneous transformation matrix, use the teaching method to obtain the homogeneous transformation representation of the object coordinate system in the robot end-effector coordinate system; S3. Based on the type of object to be grasped, and using the homogeneous transformation representation of the object's coordinate system in the robot's end-effector coordinate system, different pose generation methods are employed to generate multiple candidate grasping poses. S4. Select the pose with feasible robot kinematics solution from multiple candidate grasping poses, and select the pose with the minimum loss as the output optimal robot end-effector grasping pose according to the preset loss function.

2. The method for generating robot end-effector grasping pose by combining vision and minimal teaching as described in claim 1, characterized in that: Step S1 includes: S11. Add restrictions to the original point cloud coordinate range to retain the point cloud information of the grabbed object and the area of ​​the desktop it is located in; S12. Before placing the object to be grasped, take a picture of the original point cloud containing the desktop and the surrounding environment. After filtering out the messy point cloud through the aforementioned constraints, obtain the desktop point cloud set. S13. After placing the object to be grasped, an original point cloud image containing the object to be grasped, the desktop and the surrounding environment is captured. After filtering out the cluttered point cloud by the aforementioned limiting conditions, the point cloud sets of the desktop and the object to be grasped are obtained. Then, the point cloud set of the object to be grasped is subtracted from the point cloud set of the desktop to obtain the original point cloud set of the object to be grasped. S14. Project the original point cloud of the captured object onto the uv two-dimensional plane of the camera coordinate system, perform connected component erosion operation, and then remap the remaining point cloud back to the original space to obtain the segmented multi-object point cloud. S15. Constructing the object coordinate system: For each object's point cloud set, take the average value of all point cloud coordinates as the origin, uniformly set the object's z-axis to be perpendicular to the table, and use the first and second principal axes obtained by principal component analysis after projecting the point cloud onto the table plane as the object's x-axis and y-axis, respectively, thereby obtaining the object's pose in the camera coordinate system.

3. The method for generating robot end-effector grasping pose by combining vision and minimal teaching as described in claim 2, characterized in that: The specific process of hand-eye alignment in step S2 is as follows: S21. Fix the chessboard to the end of the robotic arm, and adjust the pose of the end of the robotic arm so that the chessboard is completely in the camera's field of view. Record the current image and the pose of the end of the robotic arm. S22. Repeat step S21 until data from N different poses are collected; S23. Based on the collected data, transform the hand-eye calibration problem into... AX=YB The model employs a geometric stochastic global optimization method to optimize the objective function, and calculates the homogeneous transformation matrix of the chessboard coordinate system in the robot's base coordinate system. Its expression is as follows: in, A、 This represents the pose of the robotic arm's end effector within the base system. X、 The pose representation of the chessboard grid under the end effector of the robotic arm. Y、 This represents the pose of the camera system within the robotic arm's base system. B、 The pose representation of the chessboard grid under the camera system; Substituting the multiple sets of pose data obtained in S22 into the objective function yields: in, Denotes the Frobenius norm. For the first i The pose representation of the robotic arm's end effector in the base coordinate system during the next measurement. For the first i The pose representation of the checkerboard grid under the end effector of the robotic arm during the measurement, SE(3) is a special Euclidean group used to describe the six degrees of freedom motion of a rigid body in three-dimensional space.

4. The method for generating robot end-effector grasping pose by combining vision and minimal teaching as described in claim 3, characterized in that: The teaching method in step S2 is as follows: S24. Within the depth camera's sensing range, manually drag the robotic arm to the object to be grasped, and record the pose of the robotic arm's end effector relative to the base coordinate system at this moment. ; S25. Move the robotic arm out of the sensing range, and obtain the pose of the grasped object relative to the camera coordinate system at the current moment through step S1. ; S26. Substitute the poses obtained in steps S23, S24, and S25 into the following formula to obtain the homogeneous transformation representation of the grasped object coordinate system in the robot's end effector coordinate system. : in, This represents the pose of the robotic arm's end effector relative to the base coordinate system. This represents the pose of the camera system within the robotic arm's base system. To capture the pose of an object relative to the camera coordinate system at the current moment; And save the homogeneous transformation representation of the calculated object coordinate system in the robot's end effector coordinate system. This serves as a teaching grasping pose for the object.

5. The method for generating robot end-effector grasping pose by combining vision and minimal teaching as described in claim 4, characterized in that: In step S3, when the grasped object is a rotating body, a 5D pose generation method is used, specifically including: S31. Calculate the pose of the robotic arm's end effector relative to the base coordinate system under the guidance of the current teaching information. The rotation axis is then extracted as the z-axis of the object's coordinate system; the specific calculation formula is as follows: ； S32. Rotate the initial grasping pose around the rotation axis by discrete angles. θ Multiple candidate grasping poses are generated, and the rotated grasping poses are obtained. Calculated using the following formula: in, This represents the rotational attitude of the robot's end effector within the base system. Let this be the displacement of the robot's end effector within the base system. This represents the pose of the robot's end effector within the pedestal system. For the axis of rotation about the base coordinate system Rotation The rotation representation obtained from the angle, This represents the rotational attitude of the newly generated candidate robot end effector in the base coordinate system. Let this be the displacement representation of the newly generated robot end effector in the base system. This represents the pose of the newly generated robot end effector in the pedestal system.

6. The method for generating robot end-effector grasping pose by combining vision and minimal teaching as described in claim 4, characterized in that: In step S3, when the object being grasped is a non-rotating body, a 6D pose generation method is used, specifically including: The pose of the grasped object relative to the end effector of the robotic arm obtained through teaching. Perform the transformation to generate the corresponding reverse side grasping pose. Reverse its position relative to the x-axis and y-axis of its orientation, using the following transformation formula: Then, the transformed pose is converted to the grasping pose of the robotic arm's end effector in the base coordinate system. The specific formula is as follows: 。 7. The method for generating robot end-effector grasping pose by combining vision and minimal teaching as described in claim 1, characterized in that: Step S4 specifically includes: inputting all generated candidate grasping poses into the inverse kinematics solver of the robotic arm for feasibility verification, retaining poses with kinematically feasible solutions, and then calculating the weighted loss between each feasible pose and the anthropomorphic initial arm joint angle according to a preset loss function. The pose with the minimum loss is selected as the optimal grasping pose. The formula is as follows: in, The initial arm posture designed for humanoid design. The current joint angle of the robotic arm. The weights are assigned to each joint angle.

8. A robot end-effector grasping pose generation system combining vision and minimal teaching, used to implement the robot end-effector grasping pose generation method combining vision and minimal teaching as described in any one of 1 to 7, characterized in that, include: The visual perception module is used to capture raw point cloud images containing the grasping object and its surrounding environment using a depth camera, and to perform point cloud segmentation and coordinate system construction on the raw point cloud images to obtain the pose of the object in the camera coordinate system. The teaching module is used to obtain the homogeneous transformation matrix between the camera coordinate system and the robot base coordinate system through hand-eye calibration, and to obtain the homogeneous transformation representation of the object coordinate system in the robot end-effector coordinate system through manual teaching. The pose generation module is used to generate multiple candidate grasping poses based on the homogeneous transformation representation obtained by the teaching module, according to the type of the grasped object, and using the corresponding pose generation method to generate multiple candidate grasping poses. The grasping control module is used to select the optimal grasping pose from the multiple candidate grasping poses and control the robotic arm to grasp the object to be grasped.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the robot end-effector grasping pose generation method that combines vision and minimal teaching as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the robot end-effector grasping pose generation method that combines vision and minimal teaching, as described in any one of claims 1 to 7.

Images