An intelligent grasping method for a humanoid two-armed robot

Abstract

The present invention belongs to the technical field of humanoid dual-arm robots, and specifically relates to an intelligent grasping method for humanoid dual-arm robots, including: obtaining the RGB image and depth image of a target object through a global camera and performing alignment processing, using a rough positioning model to detect the target object, and determining the pre-grasping positions of the dual arms; obtaining the global point cloud data in the working space in real time through the global camera, and adopting an improved artificial potential field algorithm for path planning to enable the dual arms to reach the pre-grasping positions respectively; obtaining the RGB image and depth image of the target object through the cameras at the ends of the dual arms and performing alignment processing, using a fine positioning model to detect the RGB image of the target camera, and using a pre-trained 6D grasping pose prediction model to find a feasible target grasping pose; controlling the dual arms to execute the grasping action according to the target grasping pose and placing it at a preset position. The present invention can adapt to various complex scenarios and achieve diversified grasping.

Description

Technical Field

[0001] The present invention belongs to the technical field of humanoid dual-arm robots, and particularly relates to an intelligent grasping method for a humanoid dual-arm robot. Background Art

[0002] In the field of modern automation and robotics, the demand for intelligent grasping technology is increasing day by day, especially in application scenarios of object sorting in complex environments. However, existing single-arm robots often face problems such as low operation efficiency, insufficient flexibility, and high collision risks when dealing with multi-object grasping tasks. These technical bottlenecks make it particularly difficult to achieve efficient and accurate object grasping in a dynamic environment. The introduction of a dual-manipulator system can significantly improve the flexibility and efficiency of grasping tasks. The two arms can work cooperatively to grasp objects simultaneously, reducing the operation time and showing stronger adaptability when dealing with complex object layouts. However, the motion coordination of the two arms also brings additional challenges. Especially in a dynamic environment, how to effectively avoid collisions between the two arms and interference with other dynamic obstacles in the environment has become a key issue for intelligent grasping systems. Summary of the Invention

[0003] Object of the Invention: The object of the present invention is to provide an intelligent grasping method for a humanoid dual-arm robot in view of the deficiencies of the prior art. Through a phased rough positioning and fine positioning method, the system can adapt to various complex scenarios and achieve diversified grasping.

[0004] Technical Solution: The intelligent grasping method for the humanoid dual-arm robot described in the present invention includes the following steps:

[0005] S1. Target rough positioning stage: Obtain the RGB image and depth image of the target object through the global camera and perform alignment processing; use the rough positioning model to detect the RGB image of the target object, extract the rough positioning information of the target object, calculate the point cloud data of the rough positioning information based on the alignment processing, calculate the point cloud centroid and the centroid normal vector of the target object based on the point cloud data and convert them to the robot world coordinate system, and determine the pre-grasping positions of the two arms;

[0006] S2. Dual-arm motion stage: Construct the kinematic model of the two arms and perform envelope on the arm joints; obtain the global point cloud data in the working space in real time through the global camera, preprocess the global point cloud data, and retain the obstacle point cloud; combine the kinematic model and coordinate transformation to calculate the distance between the arm joints and the obstacle point cloud in real time; calculate the repulsive force field and the gravitational force field in the joint space, and adopt an improved artificial potential field algorithm for path planning to make the two arms reach the pre-grasping positions respectively;

[0007] S3. Target fine positioning stage: After the two arms reach the pre-grasping position, the RGB image and depth image of the target object are obtained by the cameras at the ends of the two arms and aligned. The fine positioning model is used to detect the RGB image of the target object, extract the fine positioning information of the target object, and calculate the point cloud data of the fine positioning information based on the alignment process. The RGB image and depth image processed by the fine positioning model are input into the pre-trained 6D grasping pose prediction model to generate an unordered sequence of grasping poses. The grasping pose with the highest score is selected, and inverse kinematics solution is performed on the selected grasping pose. If there is no solution, the pose with the second highest score is selected until a feasible target grasping pose is found.

[0008] S4. Object placement stage: Control the two arms to perform the grasping action according to the target grasping pose and place it at the preset position; Determine whether there are ungrasped target objects. If so, repeat S1 - S3 until all grasping tasks are completed.

[0009] To further improve the above technical solution, the following steps are used in the target rough positioning stage to obtain the pre-grasping position of the target object:

[0010] Obtain the RGB image and depth image of the target object through the global camera, align the RGB image and depth image to obtain the depth value corresponding to each RGB pixel point in the RGB image, and generate point cloud data;

[0011] Use the rough positioning model to perform target detection on the RGB image to obtain the rough prediction box of the target object;

[0012] Extract the point cloud data of the corresponding area in the depth image according to the rough prediction box;

[0013] Perform centering processing on the point cloud data to obtain the centroid and normal vector of the point cloud data;

[0014] Convert the normal vector of the point cloud and the center of the rough prediction box to the robot world coordinate system, and determine the pre-grasping position of the target object by offsetting a preset distance along the direction of the normal vector of the point cloud;

[0015] Obtain the joint angles corresponding to the pre-grasping position through the inverse kinematics calculation of the robotic arm.

[0016] Further, the rough prediction box of the th target object is expressed as: , where and represent the upper left coordinate and lower right coordinate of the rough prediction box respectively;

[0017] The point cloud data of the area corresponding to the rough prediction box in the depth image is expressed as: ;

[0018] Center the point cloud data to obtain the corresponding centroid of the point cloud and the centered point cloud data , expressed as: , , calculate the covariance matrix of the centered point cloud data , and the calculation formula is: , where is the transpose of the centered point cloud data, solve the covariance matrix for its eigenvalues and eigenvectors , and the calculation formula is: , assuming that the eigenvector corresponding to the largest eigenvalue is , then this eigenvector is the normal vector of the point cloud data ; ;

[0019] Convert the point cloud normal vector and the center of the rough prediction box from the camera coordinate system to the robot world coordinate system, and the conversion formula is: , , where represents the rotation transformation matrix from the coordinate system to the coordinate system, and represents the offset vector from the

[0020] Set the rough positioning result as the pre-grasping position of the robotic arm , and the pre-grasping position is obtained by moving the center of the prediction box along the point cloud normal vector by 40 cm, and the calculation formula is as follows: ;

[0021] After obtaining the pre-grasping position , use the inverse kinematics of the robotic arm to calculate the corresponding joint angles:

[0022] .

[0023] Furthermore, the dual-arm motion stage includes:

[0024] Based on the dual-arm kinematic model, simplify the robotic arm structure, determine the joint points of the robotic arm, set virtual joint points at the center of the connecting rods between adjacent joints of the robotic arm, and use spherical bounding boxes to envelope at each joint point to obtain a dual-arm spherical envelope body;

[0025] Obtain global point cloud data in real time through a global camera, and process the global point cloud data, including: using pass-through filtering to limit the point cloud data within the workspace of the dual-arm robot; using voxel downsampling to reduce the density of the point cloud data; removing the point cloud data of the manipulator body through conditional filtering, and only retaining the point cloud of obstacles within the workspace;

[0026] According to the dual-arm spherical envelope and the manipulator kinematic model, calculate the position of each joint point of the dual-arm in the world coordinate system, and through the positions of the joint points, calculate the distances between the joint points of the dual-arm and the distances between the obstacle point cloud and the joint points of the dual-arm;

[0027] Establish a repulsive force field based on the distances between the obstacle point cloud and the joint points of the dual-arm, use the Jacobian matrix 𝐽 to map the repulsive force from the Cartesian space to the joint space, calculate the gravitational force field according to the difference between the current joint angles and the target joint angles of the dual-arm, and synthesize the gravitational force field and the repulsive force field to obtain the total force field acting on the dual-arm;

[0028] Iteratively update the joint angles of the dual-arm through the total force field to obtain the motion path of the dual-arm.

[0029] Furthermore, for each point in the point cloud data , the pass-through filtering is expressed as:

[0030]

[0031] where and are the ranges set by the filter;

[0032] Removing the point cloud data of the manipulator body through conditional filtering includes: taking the position of the th joint of the manipulator as the center, setting a distance threshold, and filtering the points whose distance from the reference point is less than the threshold: .

[0033] Furthermore, calculate the distances between the joint points of the dual-arm through the following formula: ,

[0034] where are the position coordinates of the th joint of the left arm and the th joint of the right arm in the world coordinate system respectively;

[0035] Calculate the distances between each obstacle point cloud and the joint points of the dual-arm through the following formula:

[0036]

[0037] where respectively represent the The position coordinates of an obstacle point cloud and the joint points of the two arms, taking values of L or R, representing the left arm and the right arm respectively;

[0038] According to the calculated distance, a repulsive force field is established, and the calculation formula of the repulsive force field is as follows:

[0039]

[0040]

[0041] Among them, is the repulsive force coefficient, is the set safety threshold;

[0042] The total repulsive force field The calculation formula is: ;

[0043] The repulsive force function can be obtained by derivation as: ;

[0044] Calculate the Jacobian matrix of the robotic arm Map the repulsive force to the joint space: ;

[0045] Calculate the gravitational field, and the calculation formula is as follows: ,

[0046] Among them, and are the current joint angle and the target joint angle of the robotic arm respectively;

[0047] Update the joint angle of the robotic arm;

[0048] Among them, is the iteration step size.

[0049] Beneficial effects: Compared with the prior art, the advantages of the present invention are as follows: Through the phased rough positioning and fine positioning methods, the system can adapt to various complex scenarios and achieve diversified grasping. In the rough positioning stage, through point cloud data and feature extraction (such as the centroid and normal vector of the point cloud), the precise positioning of the target object in three-dimensional space is realized, and this method provides a reliable initial position for subsequent precise grasping; in the fine positioning stage, based on the selected grasping posture, the corresponding robotic arm joint angles are calculated through inverse kinematics and solved to ensure that the robotic arm can accurately execute the grasping task.

[0050] The improved artificial potential field method effectively avoids collisions between the two arms and conflicts with dynamic obstacles in the environment, ensuring the safety of operations; the gravitational field guides the robotic arm to the target position, while the repulsive field ensures collision avoidance during movement. The design of the repulsive field effectively avoids collisions between joints and conflicts with obstacles by setting a safety threshold and using a smooth exponential function; the use of the Jacobian matrix ensures the effective conversion of the control force in the joint space, and the iterative update of the joint angles guarantees the stability and target orientation of the movement.

[0051] The dual-arm system can perform grasping tasks simultaneously, significantly shortening the working time and improving operation efficiency. It can be applied to the control of dual-arm robots in complex environments, especially suitable for scenarios that require simultaneous obstacle avoidance and target positioning. Description of the Drawings

[0052] Figure 1 is a schematic diagram of the coordinate system of the dual-arm robot in the present invention;

[0053] Figure 2 is a schematic diagram of the target rough positioning process in the present invention;

[0054] Figure 3 is a schematic diagram of the result of point cloud preprocessing in the present invention;

[0055] Figure 4 is a specific flowchart of the dual-arm movement stage in the present invention;

[0056] Figure 5 is a schematic diagram of obtaining the refined prediction box in the target fine positioning stage of the present invention;

[0057] Figure 6 is a schematic diagram after processing the refined prediction box in the target fine positioning stage of the present invention;

[0058] Figure 7 is a schematic diagram of the target grasping posture in the present invention;

[0059] Figure 8 is a specific flowchart of the target fine positioning stage in the present invention;

[0060] Figure 9 is the overall flowchart of the present invention. Detailed Description of the Invention

[0061] The technical solution of the present invention will be described in detail below with reference to the drawings, but the protection scope of the present invention is not limited to the described embodiments.

[0062] Embodiment 1: The double-arm intelligent grasping operation process of the present invention is divided into a target rough positioning stage, a target fine positioning stage, and a double-arm movement stage. In the target rough positioning stage, the global camera information is used to provide the position information for the manipulator to perform target fine positioning; in the target fine positioning stage, the 6D grasping pose estimation algorithm is used to obtain the grasping poses of the two arms respectively; in the double-arm movement stage, the improved artificial potential field method is used for planning and control to solve the problems of dynamic obstacle avoidance during the movement of the double-arm robot and the collision between the two arms. The specific process of the double-arm intelligent grasping operation process is as Figure 9 shown.

[0063] Step 1: Establish the mutual conversion of each coordinate system on the double-arm robot;

[0064] The coordinate system of the double-arm robot of the present invention includes the left arm, the right arm, the global camera, the left-arm end camera, and the right-arm end camera, as Figure 1 shown. Among them, is the world coordinate system, is the left-arm base coordinate system, is the right-arm base coordinate system, is the left-arm end coordinate system, is the right-arm base coordinate system, is the global camera coordinate system, is the left-arm end camera coordinate system, is the right-arm end camera coordinate system. Taking the rotation from the left-arm end coordinate system to the world coordinate system as an example:

[0065]

[0066] In the above formula, represents the rotation transformation matrix from the coordinate system to the coordinate system, represents the offset vector from the

[0067] Step 2: Set the joint angles when the two arms start to work and are placed , , , .

[0068] Step 3: Use the equipped global depth camera to obtain RGB image information and depth information respectively, and use the interface provided by the camera to align the RGB image and the depth image to realize the mutual conversion between the RGB image pixel coordinates and the depth point cloud position information.

[0069] Step 4: The robotic arm enters the target rough positioning stage. Using the rough positioning model (the YOLOv8 model is adopted for the rough positioning model) and point cloud computing, obtain the rough positioning results of all objects to be grasped, that is, the pre-grasping positions of the two arms. The target rough positioning process is as Figure 2 shown.

[0070] The specific steps are as follows:

[0071] 1. Use the trained YOLOv8 model to detect the input image (RGB image) to obtain the prediction box of the

[0072]

[0073] th object to be grasped, where and

[0074] represent the upper left and lower right coordinates of the prediction box respectively;

[0075]

[0076] 2. Use image alignment to extract the point cloud data in the depth image that aligns with the prediction box. Among them, each point cloud data is represented as a three-dimensional coordinate

[0077] with the camera coordinate system as the origin; 3. Centralize the point cloud data and calculate the centroid of the point cloud corresponding within the prediction box and the centralized point cloud data

[0078]

[0079]

[0080] The calculation formulas are as follows: 4. Calculate the covariance matrix

[0081]

[0082] of the centralized point cloud data. The calculation formula is as follows: where

[0083] is the transpose of the centralized point cloud data; 5. Solve the eigenvalues and eigenvectors

[0084]

[0085] of the covariance matrix. The calculation formulas are as follows: Calculate. Assume that the eigenvector corresponding to the largest eigenvalue is , then this eigenvector is the normal vector of the point cloud data ;

[0086] 7. Point cloud normal vector and the center of the prediction box are transformed from the camera coordinate system to the dual-arm robot world coordinate system. The transformation formula is:

[0087]

[0088]

[0089] 8. Determine the pre-grasping position and set the rough positioning result as the target position of the robotic arm , at this position there is the center of the prediction box Moving 40 cm along the normal vector of the point cloud in this area direction is obtained. The calculation formula is as follows:

[0090]

[0091] 9. After obtaining the target position, use the inverse kinematics of the robotic arm to calculate the corresponding joint angles:

[0092]

[0093] Step Five: The dual arms enter the motion stage, and use the improved artificial potential field method to control the dual arms to reach their respective pre-grasping positions respectively.

[0094] The specific process of the dual-arm motion stage is as Figure 4 shown. The steps are as follows:

[0095] 1. Based on the dual-arm kinematic model, simplify the robotic arm model, determine the joint points of the robotic arm, set virtual joint points at the center of the connecting rods between adjacent joints of the robotic arm, and use spherical bounding boxes to envelope at each joint point;

[0096] 2. Use the global depth camera to continuously obtain all the point cloud data in the camera. Since the amount of the original point cloud data is large, using the original point cloud data for calculation cannot meet the real-time requirements of the dual-arm robot. Therefore, use the PCL library to process the point cloud to reduce the amount of calculation.

[0097] a. Pass-through filtering. Set the filtering range. For each point in the point cloud, the pass-through filtering condition can be expressed as:

[0098]

[0099] Among them, and Set the range for the filter and perform the same processing for other dimensions to filter the point cloud data outside the working space of the dual-arm robot.

[0100] b. Voxel downsampling filtering: Divide the point cloud data after pass-through filtering into uniform voxel grids to reduce the density of the point cloud. While retaining the shape of the point cloud, it effectively reduces the amount of point cloud data significantly and improves the calculation speed.

[0101] c. Point cloud filtering of the manipulator body: During the movement of the dual arms, the global camera will collect the point cloud of the manipulator body. It is necessary to further filter the point cloud of the manipulator body and only retain the point cloud data of the obstacles entering the working space of the dual-arm robot. Through the kinematic model and coordinate transformation, calculate the positions of each joint of each manipulator in the camera coordinate system. Take the left-arm joint as an example.

[0102] The calculation formula is as follows:

[0103] ,

[0104] Among them, is the transformation matrix of the th joint of the left arm in the left-arm base coordinate system, and is the position of the th joint of the left arm in the camera coordinate system.

[0105] Use conditional filtering. Taking the position of the th joint of the manipulator as the center, set a distance threshold to filter the points whose distance from the reference point is less than this threshold.

[0106]

[0107] After point cloud processing, the global camera only retains the obstacle information existing in the working space of the dual-arm robot. As Figure 3 shown.

[0108] 3. Through the dual-arm spherical envelope and the kinematic model, calculate the positions of all joints of the dual arms in the corresponding base coordinate systems, and then use coordinate transformation to convert them into the world coordinate system. Calculate the distances of each joint of the dual arms in the world coordinate system. The calculation method is as follows: ,

[0109] Among them, are the position coordinates of the th joint of the left arm and the th joint of the right arm in the world coordinate system respectively.

[0110] At the same time, calculate the distances between each point cloud in the world coordinate system and all joints of the dual manipulators

[0111] ,

[0112] Among them, respectively represent the position coordinates of the th point cloud in the camera and the th joint of the robotic arm in the world coordinate system.

[0113] 4. The traditional artificial potential field algorithm is based on the Cartesian coordinate system and requires inverse solution conversion to the joint space, which has a long calculation time, low efficiency, and is prone to falling into local minima. To improve the above problems, an artificial potential field algorithm based on the joint space is proposed: According to the calculated distance above, a repulsive force field is established, and the calculation formula of the repulsive force field is as follows:

[0114] ,

[0115] ,

[0116] Among them, is the repulsive force coefficient;

[0117] Total repulsive force field The calculation formula is: ;

[0118] Taking the derivative, the repulsive force function can be obtained as: ;

[0119] Since is the repulsive force of each joint of the robotic arm in the Cartesian space, the Jacobian matrix of the robotic arm needs to be calculated to map the repulsive force to the joint space: .

[0120] 5. Calculate the gravitational field, and the calculation formula is as follows: ,

[0121] Among them, and respectively represent the current joint angle and the target joint angle of the robotic arm;

[0122] 6. Update the joint angles of the robotic arm;

[0123] Among them, is the iteration step size;

[0124] 7. Repeat steps 3 to 6 until the robotic arm reaches the target joint angle.

[0125] Step Six: In the target fine-positioning stage, the two arms respectively use the 6D grasping pose prediction algorithm to obtain the precise grasping poses of the two arms at the pre-grasping pose; The specific process of the target fine-positioning stage is as Figure 8 shown, and the steps are as follows:

[0126] 1. After the robotic arm reaches the pre-grasping position, call the configured end camera to obtain RGB information and depth information in real time;

[0127] 2. Call the trained fine positioning model to obtain a more accurate prediction box for the object to be grasped. Coarse positioning is processed based on the images collected by the global camera. The effect of coarse positioning is as Figure 5 shown. It can only frame the approximate edge range. Fine positioning can directly segment the operation object and generate a recommended grasping posture, as Figure 7 shown. The fine positioning model uses the YOLOv8 model to process the RGB image and depth image of the end camera, performs dilation processing on the prediction box, and only retains the data within the dilated prediction box. Dilation generally refers to the edge expansion and sparse processing of the image. The dilation described in the present invention is also the edge prediction of the image processed by YOLOv8, which is the data after prediction, reducing the interference of irrelevant information and the computational amount of the posture prediction model, as Figure 6 shown;

[0128] 3. Input the processed RGB image and depth image into the pre-trained grasping prediction model GraspNet model (GraspNet is an existing model, an efficient convolutional neural network for real-time detection of grasping by low-power devices. This model can generate good grasping postures through training with a small number of samples), generate a large number of unordered jaw 6D grasping posture sequences, sort them according to the grasping posture scores, and take the grasping posture with the highest score as the target grasping posture, as Figure 7 shown;

[0129] 4. Perform inverse kinematics on the end grasping posture. If there is no inverse kinematics solution, it means that the robotic arm cannot reach the target end posture. Then take the grasping posture with the second highest score as the target grasping posture until there is an inverse kinematics solution for the target grasping posture or all the grasping postures in the posture sequence are traversed.

[0130] Although the YOLOv8 model is used in both the coarse positioning stage and the fine positioning stage, their application focuses are different:

[0131] Coarse positioning stage: mainly use YOLOv8 for fast object detection, obtain the approximate position of the object, and calculate the preliminary grasping position through point cloud data. At this time, the YOLOv8 model outputs a rough prediction box with relatively low positioning accuracy. It is achieved by configuring the YOLOv8 model as follows: use a smaller input size, larger Anchor Boxes, and set looser NMS and confidence thresholds in the coarse positioning stage.

[0132] Fine positioning stage: mainly use YOLOv8 to detect the target object more precisely, and cooperate with the depth image and the 6D grasping pose prediction model to generate accurate grasping poses. At this time, YOLOv8 is not only used to detect the object position, but also helps to calculate and optimize the grasping posture, with higher precision requirements. It is achieved by configuring the YOLOv8 model as follows: use a larger input size in the fine positioning stage, use more refined Anchor Boxes, and set a stricter threshold in the fine positioning stage to ensure precise positioning.

[0133] Step 7: The two arms move to the target grasping pose, and control the gripper to complete the object grasping.

[0134] Step 8: The two arms enter the movement stage to reach the placement pose and complete the object placement.

[0135] Step 9: Determine whether there are still objects to be grasped. If so, repeat Steps 3 to 8; otherwise, end the two-arm intelligent grasping task.

[0136] As described above, although the present invention has been shown and described with reference to specific preferred embodiments, it should not be construed as a limitation of the present invention itself. Various changes in form and detail may be made without departing from the spirit and scope of the present invention defined by the appended claims.

Claims

1. An intelligent grasping method for a humanoid dual-arm robot, characterized in that: The following steps are involved: S1, target coarse positioning stage: obtain the RGB image and depth image of the target object through the global camera and align them; Use the coarse positioning model to detect the RGB image of the target object, extract the coarse positioning information of the target object, calculate the point cloud data of the coarse positioning information based on the alignment process, calculate the point cloud centroid and point normal vector of the target object based on the point cloud data and convert them to the robot world coordinate system, and determine the pre-grasping position of the dual arms; S2, dual-arm motion stage: Simplify the structure of the robotic arm based on the dual-arm kinematic model, determine the joint points of the robotic arm, set virtual joint points at the center of the connecting rod between adjacent joints of the robotic arm, use a spherical bounding box to envelop each joint point, and obtain a spherical envelope of the dual arms; obtain the global point cloud data in the workspace in real time through the global camera, pre-process the global point cloud data, and retain the obstacle point cloud; calculate the position of each joint point of the dual arms in the world coordinate system according to the dual-arm spherical envelope and the robotic arm kinematic model, and calculate the distance between the joint points of the dual arms by the following formula: ， in, Left arm joints and right arm The position coordinates of the joint points in the world coordinate system The distance between each obstacle point cloud and each joint point of the arms is calculated using the following formula: ， in, They represent the obstacle point cloud and the two arms The position coordinates of the joint points, The value is L or R, representing the left arm and the right arm respectively; The repulsive field and the gravitational field are calculated in the joint space, and the improved artificial potential field algorithm is used for path planning so that the two arms can reach the pre-grasping positions respectively; The calculation formula of the repulsive field is as follows: in, is the repulsion coefficient, The safety threshold is set; Total repulsive field The calculation formula is: ; The derivation of the repulsion function is: ; Calculate the Jacobian matrix of the robot Map the repulsive force to the joint space: ; The calculation formula of the gravitational field is as follows: , in, and They are the current joint angle and target joint angle of the robot arm respectively; Update the robot arm joint angles; in, is the iteration step length; S3, target precise positioning stage: when the two arms reach the pre-grasping position, the RGB image and depth image of the target object are obtained through the cameras at the end of the two arms and aligned, the RGB image of the target object is detected using the precise positioning model, the precise positioning information of the target object is extracted, and the point cloud data of the precise positioning information is calculated based on the alignment process; the RGB image and depth image processed by the precise positioning model are input into the pre-trained 6D grasping posture prediction model to generate an unordered grasping posture sequence, the grasping posture with the highest score is selected, and the selected grasping posture is inversely kinematically solved. If there is no solution, the posture with the second highest score is selected until a feasible target grasping posture is found; S4, object placement stage: control the arms to perform the grasping action according to the target grasping posture and place it at the preset position; determine whether there is an ungrasped target object. If so, repeat S1-S3 until all grasping tasks are completed.

2. The intelligent grasping method of the humanoid dual-arm robot according to claim 1, characterized in that: The target rough positioning stage obtains the pre-grasp position of the target object through the following steps: The RGB image and depth image of the target object are acquired through the global camera, and the RGB image and the depth image are aligned to obtain the depth value corresponding to each RGB pixel in the RGB image and generate point cloud data; Use the coarse positioning model to detect targets in RGB images and obtain the coarse prediction box of the target object; Extracting point cloud data of the corresponding area in the depth image according to the coarse prediction frame; Perform central processing on the point cloud data to obtain the point cloud centroid and point cloud normal vector of the point cloud data; The point cloud normal vector and the center of the rough prediction box are converted to the robot world coordinate system, and the pre-grasping position of the target object is determined by offsetting a preset distance along the direction of the point cloud normal vector; The joint angles corresponding to the pre-grasping position are obtained through inverse kinematics calculation of the robot arm.

3. The intelligent grasping method of the humanoid dual-arm robot according to claim 2, characterized in that: No. The rough prediction box of a target object is expressed as: ,in, and Respectively represent the upper left corner coordinates and lower right corner coordinates of the rough prediction box; The point cloud data of the area corresponding to the coarse prediction box in the depth image It is expressed as: ; Centralize the point cloud data to obtain the corresponding point cloud centroid and centralized point cloud data , expressed as: , , calculate the centralized point cloud data The covariance matrix of , the calculation formula is: ,in, is the transpose of the centralized point cloud data, solving the covariance matrix The eigenvalue of and the eigenvector , the calculation formula is: , assuming that the eigenvector corresponding to the largest eigenvalue is , then the feature vector is the normal vector of the point cloud data ; The point cloud normal vector and the center of the rough prediction box The conversion formula from the camera coordinate system to the robot world coordinate system is: , ,in, express Coordinate system to The rotation transformation matrix of the coordinate system, express Coordinate system to The offset vector of the coordinate system; Set the rough positioning result as the pre-grasping position of the robot arm , the pre-grab position is through the prediction box center Along the point cloud normal vector The calculation formula is as follows: ; Get the pre-fetch position Finally, the corresponding joint angles are calculated using the inverse kinematics of the robotic arm: 。 4. The intelligent grasping method of a humanoid dual-arm robot according to claim 1, characterized in that: The two-arm movement phase includes: Based on the kinematic model of the two arms, the structure of the robot arm is simplified, the joint points of the robot arm are determined, virtual joint points are set at the center of the connecting rod between the adjacent joints of the robot arm, and a spherical bounding box is used to envelop each joint point to obtain a spherical envelope of the two arms; The global point cloud data is acquired in real time through the global camera, and the global point cloud data is processed, including: using straight-through filtering to limit the point cloud data to the workspace of the dual-arm robot; using voxel downsampling to reduce the density of the point cloud data; removing the point cloud data of the robot body through conditional filtering, and only retaining the obstacle point cloud in the workspace; According to the spherical envelope of the arms and the kinematic model of the robotic arm, the position of each joint point of the arms in the world coordinate system is calculated. Through the position of the joint points, the distance between the joint points of the arms and the distance between the obstacle point cloud and the joint points of the arms are calculated; A repulsive field is established based on the distance between the obstacle point cloud and each joint point of the two arms. The Jacobian matrix 𝐽 is used to map the repulsive force from the Cartesian space to the joint space. The gravitational field is calculated based on the difference between the current joint angle of the two arms and the target joint angle. The gravitational field and the repulsive field are synthesized to obtain the total force field acting on the two arms. The joint angles of both arms are updated iteratively through the total force field to obtain the motion paths of both arms.

5. The intelligent grasping method of the humanoid dual-arm robot according to claim 4, characterized in that: For each point in the point cloud data , the straight-through filtering is expressed as: in, and The range set for the filter; The point cloud data of the robot body is removed by conditional filtering, including: The joint position is taken as the center, and the distance threshold is set to filter out points whose distance from the reference point is less than the threshold: .

Images