An adaptive gripping device for humanoid robots based on visual feedback

Abstract

The invention relates to the technical field of industrial robots, and provides a humanoid robot self-adaptive clamping jaw grabbing device based on visual feedback, the grabbing device comprises a visual feedback module, a clamping jaw control module and a mechanical arm provided with a clamping jaw, and the mechanical arm comprises a plurality of joints; the visual feedback module is used for carrying out fine-grained object classification on the environment information through an improved YOLOv7 model and sending the shape type and the three-dimensional pose of an object to be grabbed to the clamping jaw control module; the clamping jaw control module is used for receiving the shape type and the three-dimensional pose of the to-be-grabbed object sent by the visual feedback module, controlling the mechanical arm to move by adopting a low space occupation configuration strategy, and generating a collision-free track of the clamping jaw from the initial position to the to-be-grabbed object; and when the clamping jaw approaches the object to be grabbed, the clamping jaw is controlled to move within the speed threshold range, and when the clamping jaw makes contact with the object to be grabbed, the clamping force of the clamping jaw is restrained based on the clamping force of the clamping jaw, the shape type of the object and the three-dimensional pose.

Description

(19) State Intellectual Property Office (12) Invention Patent Application (10) Application Publication Number (43) Application Publication Date (21) Application Number 202511246263.5 (22) Application Date 2025.09.02 (71) Applicant Shanghai Tiantai Intelligent Robot Co., Ltd. Address Room 1_203, No. 337 Shahe Road, Jiangqiao Town, Jiading District, Shanghai 201800, China (72) Inventors Lv Gang, He Zhixiong, Liu Ming (74) Patent Agency Foshan Hecai Intellectual Property Agency Co., Ltd. 44379 Patent Attorney Liang Mingxi (51) Int.Cl. B25J 9 / 16 (2006.01) B25J 11 / 00 (2006.01) (54) Invention Title: An Adaptive Gripper Grasping Device for Humanoid Robots Based on Visual Feedback (57) Abstract: This invention relates to the field of industrial robot technology and proposes an adaptive gripper grasping device for humanoid robots based on visual feedback. The grasping device includes a visual feedback module, a gripper control module, and a robotic arm equipped with grippers. The robotic arm includes multiple joints. The visual feedback module is used to perform fine-grained object classification of environmental information using an improved YOLOv7 model and sends the shape type and three-dimensional pose of the object to be grasped to the gripper control module. The gripper control module is used to receive the shape type and three-dimensional pose of the object to be grasped sent by the visual feedback module, control the robotic arm to move using a low-space-occupancy configuration strategy, and generate a collision-free trajectory of the gripper from its initial position to the object to be grasped. When the gripper approaches the object to be grasped, the gripper is controlled to move within a speed threshold range. When the gripper contacts the object to be grasped, the gripping force of the gripper is constrained based on the gripping force of the gripper, the shape type of the object, and the three-dimensional pose. Claims 3 pages, Description 14 pages, Drawings 1 page, CN 120962662 A 2025.11.18 CN 1 20 96 26 62 A 1. A humanoid robot adaptive gripper grasping device based on visual feedback, characterized in that the grasping device includes a visual feedback module, a gripper control module, and a robotic arm equipped with grippers, the robotic arm including multiple joints; the visual feedback module is used to perform fine-grained object classification of environmental information using an improved YOLOv7 model, identify the shape type and three-dimensional pose of each object, select the object to be grasped, and send the shape type and three-dimensional pose of the object to be grasped to the gripper control module, the improvement including adding a fine-grained classifier for classifying cylinders, cuboids, and irregular objects to the YOLOv7 model; the gripper control module is used to receive the shape type and three-dimensional pose of the object to be grasped sent by the visual feedback module, and, when space is limited, control the robotic arm to adopt a low-space-occupancy configuration based on an improved random sampling path planning algorithm.The strategy involves generating a collision-free trajectory for the gripper from its initial position to the object to be grasped. This low-space-occupancy configuration strategy includes coordinated joint bending to form a spatial compression posture, and real-time calculation of the minimum swing parameters for each joint's motion sequence. These swing parameters include joint angle constraints and motion direction priorities. When the gripper approaches the object, it is controlled to move within a preset speed threshold range. When the gripper contacts the object, its gripping force is constrained based on the gripper's clamping force, the object's shape, and its three-dimensional pose. Specifically, when generating the collision-free trajectory for the object, the minimum Euclidean distance between the robotic arm and obstacles is calculated in real-time to adjust the gripper's trajectory in advance, controlling the gripper to perform segmented grasping actions. 2. The humanoid robot adaptive gripper grasping device according to claim 1, characterized in that the visual feedback module has a built-in camera, and the step of performing fine-grained object classification of environmental information using an improved YOLOv7 model, identifying the shape type and three-dimensional pose of each object, and selecting the object to be grasped includes: loading the YOLOv7 model and a fine-grained classifier, generating a depth map of the object after setting camera parameters, and obtaining the depth information of the target object; performing target detection on the depth map, obtaining the bounding box coordinates, object type, and confidence information of all objects in the depth map, selecting the object to be grasped based on the bounding box coordinates, object type, and confidence information, extracting the depth information of the region of the object to be grasped from the depth map according to the bounding box coordinates of the object to be grasped, and calculating the average depth; converting the pixel coordinates into three-dimensional coordinates in the world coordinate system according to the pixel coordinates and depth information of the center point of the object to be grasped, performing fine-grained shape classification of the region of the object to be grasped, and determining the shape type of the target object; Based on the shape type of the object to be grasped, a size estimation algorithm is invoked to estimate the size information of the object. The position, shape type, and size information of the target object are integrated to generate the first dataset required for grasping. 3. The humanoid robot adaptive gripper grasping device according to claim 1, characterized in that, before controlling the gripper to perform the grasping action, it includes constructing a hand-eye transformation matrix for mapping the three-dimensional pose of the object to be grasped to the robot base coordinate system: obtaining a gripper pose list and a calibration board pose list, traversing each pair of adjacent poses, multiplying the inverse of the gripper pose matrix of the previous robot with the gripper pose matrix of the current robot to obtain a robot pose transformation matrix, multiplying the current calibration board pose matrix with the inverse of the previous calibration board pose matrix to obtain a calibration board pose transformation matrix; extracting a first rotation matrix and a first translation vector from all robot pose transformation matrices, extracting a second rotation matrix and a second translation vector from all calibration board pose transformation matrices, deriving a hand-eye rotation matrix based on the first rotation matrix and the second rotation matrix, and deriving a hand-eye translation vector based on the first rotation matrix, the hand-eye rotation matrix, the first translation vector, and the second translation vector;Construct a homogeneous transformation matrix by filling the hand-eye translation vector and the hand-eye rotation matrix into the homogeneous transformation matrix to obtain the hand-eye transformation matrix. Claims 1 / 3 Page 2 CN 120962662 A 4. The humanoid robot adaptive gripper grasping device according to claim 1, characterized in that, before the gripper contacts the object to be grasped, it includes: initializing gripper dynamic model parameters, gripper physical parameters, and MPC parameters, wherein the gripper dynamic model parameters include a state transition matrix, a control matrix, an output matrix, and a direct transfer matrix; the gripper physical parameters include the gripper opening width and a gripping force threshold; and the MPC parameters include a prediction time domain, a control time domain, a state error weight, and a control input weight; adjusting the state error weight, the initial gripping force of the gripper, and the gripping force threshold according to the fragility of the object to be grasped; generating a first prediction matrix based on the power operation of the state transition matrix, generating a second prediction matrix based on the product operation of the power of the state transition matrix and the control matrix, and constructing an objective function based on the first and second prediction matrices; under the constraints of the control time domain and the gripping force threshold, the objective function defines an optimization problem through a quadratic combination of the state error weight and the control input weight, and obtains the gripping force value of the gripper through convex optimization. 5. The humanoid robot adaptive gripper grasping device according to claim 1, characterized in that, when the gripper contacts the object to be grasped, constraining the gripping force of the gripper based on the gripping force of the gripper, the shape type of the object, and the three-dimensional pose includes: selecting a corresponding gripping strategy according to the shape type of the object to be grasped, calculating the initial opening and closing width of the gripper and the target gripping force, acquiring contact deformation data through a force sensor during the pre-gripping stage, and evaluating the softness of the object material based on the strain rate of change; dynamically adjusting the stiffness coefficient and the maximum allowable gripping force threshold of the gripper's force controller according to the softness evaluation result, generating gripper motion commands that satisfy the force-position constraints using a model predictive controller, and executing a progressive gripping operation until the gripper reaches a stable grasping state; wherein, when the shape type of the object to be grasped is a cylinder, the initial opening and closing width of the gripper is calculated based on the diameter measurement value. The contact point distribution is determined based on the cylinder height; when the object to be grasped is a cuboid, the initial opening and closing width of the gripper is calculated based on the minimum edge distance principle, and the contact area is determined based on the center point of the surface; when the object to be grasped is an irregular object, the maximum stable grasping width is extracted based on the convex hull algorithm, and the centroid-aligned gripping point is calculated. 6. The humanoid robot adaptive gripper grasping device according to claim 1, characterized in that controlling the robotic arm to move using a low-space-occupancy configuration strategy includes: calculating the spatial constraint degree based on the three-dimensional Euclidean distance between the object to be grasped and the obstacle; when the spatial constraint degree is greater than a preset threshold, firstly calculating the optimal direction vector for the gripper to approach the object to be grasped; based on the optimal direction vector, generating the initial pose of the robotic arm with continuous S-shaped bending characteristics through inverse kinematics solution, and then...The joint angles of the robotic arm are iteratively adjusted until the robotic arm reaches the minimum space occupation state. Based on the adjusted joint angles and the preset gripper movement speed, a multi-segment motion trajectory of the gripper is generated. When the spatial constraint is less than a preset threshold, a grasping trajectory is generated based on the shape type and three-dimensional pose of the object to be grasped. 7. The humanoid robot adaptive gripper grasping device according to claim 1, characterized in that, when controlling the gripper to perform the grasping action, it further includes: calculating the initial joint angle configuration of the robotic arm based on the target position of the gripper and inverse kinematics; obtaining the space volume occupied by the robotic arm of the humanoid robot in the current configuration, calculating the position error between the actual position and the target position of the gripper, calculating the minimum distance between the robotic arm and the obstacle, and obtaining a comprehensive score based on the space volume occupied by the robotic arm, the position error and the minimum distance, wherein the comprehensive score is used to measure the quality of the current joint angle configuration; taking the initial joint angle configuration of the robotic arm as the starting point for optimization, minimizing the comprehensive score through the L-BFGS-B algorithm, so as to continuously adjust the joint angle of the robotic arm to obtain the optimal joint angle of the robotic arm. 8. The humanoid robot adaptive gripper grasping device according to claim 1, characterized in that, the gripping force of the gripper satisfies the relationship: Ftarget=μ×m×g×SF; where μ represents the coefficient of friction, m represents the mass of the object, g represents the gravitational acceleration, and SF represents the safety factor. 9. The humanoid robot adaptive gripper grasping device according to claim 1, characterized in that it is further provided with a force sensor, and when adjusting the gripping force of the gripper, it includes using a PI controller to adjust the gripping force in real time according to the feedback of the force sensor, satisfying the relationship: Fadjust=Ftarget+Kpe+Ki×∫edt; where, Fadjust represents the adjusted gripper gripping force, Ftarget represents the target gripping force, Kp represents the proportional gain coefficient, e represents the error between the force sensor and the target gripping force, Ki represents the integral gain coefficient, and ∫edt represents the integral of the error with respect to time. Claims 3 / 3 Page 4 CN 120962662 A Humanoid Robot Adaptive Gripper Grasping Device Based on Visual Feedback Technical Field

[0001] This invention relates to the field of industrial robot technology, and in particular to a humanoid robot adaptive gripper grasping device based on visual feedback. Background Art

[0002] In the field of industrial automation, the application of robot technology is becoming increasingly widespread, especially in high-precision grasping tasks, where its efficiency and accuracy are crucial to the production process.

[0003] Currently, industrial robots generally adopt grasping mechanisms based on fixed programs or simple sensor feedback. This typically relies on precise programming and predetermined grasping paths, and is achieved through monocular vision systems or basic force sensors.Object recognition and grasping, for example, some robots use traditional electric door pushing mechanisms, operated by gatekeepers in fixed positions, lacking flexibility and intelligence.

[0004] Existing robot grasping systems perform poorly when handling complex or irregularly shaped objects because they often require customized grippers and grasping strategies for specific objects. In addition, these systems have difficulty achieving adaptation and optimizing the grasping path when facing multiple objects or narrow spaces, which limits the application potential of robots in changing environments, especially in scenarios requiring high precision and high adaptability, such as automated warehouse management, precision assembly, or operation tasks in special environments. Summary of the Invention

[0005] In view of the above-mentioned defects, the purpose of this invention is to propose a humanoid robot adaptive gripper grasping device based on visual feedback, which aims to address the shortcomings of insufficient adaptability when handling complex shaped objects and low grasping accuracy and efficiency in dynamic environments.

[0006] To achieve this objective, the present invention adopts the following technical solution:

[0007] A humanoid robot adaptive gripper grasping device based on visual feedback, the grasping device comprising a visual feedback module, a gripper control module, and a robotic arm equipped with grippers, the robotic arm comprising multiple joints;

[0008] The visual feedback module is used to perform fine-grained object classification of environmental information using an improved YOLOv7 model, identify the shape type and three-dimensional pose of each object, select the object to be grasped, and send the shape type and three-dimensional pose of the object to be grasped to the gripper control module. The improvement includes adding a fine-grained classifier to the YOLOv7 model for classifying cylinders, cuboids, and irregular objects;

[0009] The gripper control module is used to receive the shape type and three-dimensional pose of the object to be grasped sent by the visual feedback module. When space is limited, based on the improved random sampling path planning algorithm, the robotic arm is controlled to move using a low space occupation configuration strategy to generate a collision-free trajectory of the gripper from the initial position to the object to be grasped. The low space occupation configuration strategy includes the coordinated bending of joints to form a spatial compression posture and real-time calculation of the minimum swing parameters of each joint motion sequence. The swing parameters include joint angle constraints and motion direction priority. When the gripper approaches the object to be grasped, the gripper is controlled to move within a preset speed threshold range. When the gripper contacts the object to be grasped, the gripping force of the gripper is constrained based on the gripping force of the gripper, the shape type of the object, and the three-dimensional pose.

[0010] Wherein, when generating a collision-free trajectory of the object to be grasped, the minimum Euclidean distance between the robotic arm and the obstacle is calculated in real time to adjust the gripper motion trajectory in advance and control the gripper to perform segmented grasping actions. Instruction manual, page 1 / 14, CN 120962662 A

[0011] Preferably, the visual feedback module has a built-in camera, and the environmental information is processed by the improved YOLOv7 model.The process of fine-grained object classification, identifying the shape type and 3D pose of each object, and selecting the object to be grasped includes:

[0012] Loading the YOLOv7 model and fine-grained classifier, generating a depth map of the object after setting camera parameters, and obtaining the depth information of the target object;

[0013] Performing object detection on the depth map, obtaining the bounding box coordinates, object type, and confidence information of all objects in the depth map, selecting the object to be grasped based on the bounding box coordinates, object type, and confidence information, extracting the depth information of the region of the object to be grasped from the depth map according to the bounding box coordinates of the object to be grasped, and calculating the average depth;

[0014] Converting the pixel coordinates to 3D coordinates in the world coordinate system according to the pixel coordinates and depth information of the center point of the object to be grasped, performing fine-grained shape classification on the region of the object to be grasped, and determining the shape type of the target object;

[0015] Calling a size estimation algorithm according to the shape type of the object to be grasped, estimating the size information of the object to be grasped, integrating the position, shape type, and size information of the target object, and generating the first dataset required for grasping.

[0016] Preferably, before controlling the gripper to perform the grasping action, the method includes constructing a hand-eye transformation matrix for mapping the three-dimensional pose of the object to be grasped to the robot base coordinate system:

[0017] Obtain the pose list of the gripper and the pose list of the calibration board, traverse each pair of adjacent poses, multiply the inverse of the gripper pose matrix of the previous robot by the gripper pose matrix of the current robot to obtain the robot pose transformation matrix, and multiply the current calibration board pose matrix by the inverse of the previous calibration board pose matrix to obtain the calibration board pose transformation matrix;

[0018] Extract the first rotation matrix and the first translation vector from all robot pose transformation matrices, extract the second rotation matrix and the second translation vector from all calibration board pose transformation matrices, derive the hand-eye rotation matrix based on the first rotation matrix and the second rotation matrix, and derive the hand-eye translation vector based on the first rotation matrix, the hand-eye rotation matrix, the first translation vector, and the second translation vector;

[0019] A homogeneous transformation matrix is ​​constructed by filling the hand-eye translation vector and hand-eye rotation matrix into the homogeneous transformation matrix to obtain the hand-eye transformation matrix.

[0020] Preferably, before the gripper contacts the object to be grasped, the following steps are included:

[0021] Initializing the gripper dynamics model parameters, gripper physical parameters, and MPC parameters, wherein the gripper dynamics model parameters include a state transition matrix, a control matrix, an output matrix, and a direct transfer matrix; the gripper physical parameters include the gripper opening width and the gripping force threshold; and the MPC parameters include the prediction time domain, the control time domain, the state error weight, and the control input weight;

[0022] Adjusting the state error weight, the initial gripping force, and the gripping force threshold of the gripper according to the fragility of the object to be grasped;

[0023] Generating a first prediction matrix based on the power operation of the state transition matrix, and generating a first prediction matrix based on the power operation of the state transition matrix and the control...The matrix multiplication operation generates the second prediction matrix, and the objective function is constructed based on the first prediction matrix and the second prediction matrix;

[0024] Under the constraints of the control time domain and the clamping force threshold, the objective function defines the optimization problem by the quadratic combination of the state error weight and the control input weight, and obtains the clamping force value of the gripper by convex optimization.

[0025] Preferably, constraining the gripping force of the gripper based on the gripping force of the gripper, the shape type of the object, and the three-dimensional pose when the gripper contacts the object to be grasped includes:

[0026] Selecting a corresponding gripping strategy according to the shape type of the object to be grasped, calculating the initial opening and closing width of the gripper and the target gripping force, acquiring contact deformation data through a force sensor during the pre-gripping stage, and evaluating the softness of the object material based on the strain rate of change;

[0027] Dynamically adjusting the stiffness coefficient and the maximum allowable gripping force threshold value of the gripper's force controller according to the softness evaluation result, using a model predictive controller to generate gripper motion commands that satisfy the force-position constraints, and executing a progressive gripping operation until the gripper reaches a stable gripping state;

[0028] Specifically, when the object to be grasped is a cylinder, the initial opening and closing width of the gripper is calculated based on the diameter measurement, and the contact point distribution is determined according to the cylinder height; when the object to be grasped is a cuboid, the initial opening and closing width of the gripper is calculated based on the minimum edge distance principle, and the contact area is determined according to the center point of the surface; when the object to be grasped is an irregular object, the maximum stable gripping width is extracted based on the convex hull algorithm, and the centroid-aligned gripping point is calculated.

[0029] Preferably, controlling the robotic arm to move using a low-space-occupancy configuration strategy includes:

[0030] Calculating the spatial constraint degree based on the three-dimensional Euclidean distance between the object to be grasped and the obstacle;

[0031] When the spatial constraint degree is greater than a preset threshold, firstly calculating the optimal direction vector for the gripper to approach the object to be grasped, and based on the optimal direction vector, generating the initial pose of the robotic arm with continuous S-shaped bending characteristics through inverse kinematics solution, and iteratively adjusting the joint angles of the robotic arm until the robotic arm reaches the minimum space-occupancy state, and generating a multi-segment motion trajectory of the gripper according to the adjusted joint angles and the preset gripper movement speed;

[0032] When the spatial constraint degree is less than a preset threshold, generating a grasping trajectory based on the shape type and three-dimensional pose of the object to be grasped.

[0033] Preferably, when controlling the gripper to perform the grasping action, the method further includes:

[0034] calculating the initial joint angle configuration of the robotic arm based on the target position of the gripper and inverse kinematics;

[0035] obtaining the space volume occupied by the robotic arm of the humanoid robot in the current configuration, calculating the position error between the actual position and the target position of the gripper, calculating the minimum distance between the robotic arm and the obstacle, and based on the space volume occupied by the robotic arm and its position...The error and minimum distance are used to obtain a comprehensive score, which is used to measure the quality of the current joint angle configuration;

[0036] The initial joint angle configuration of the robotic arm is used as the starting point for optimization. The comprehensive score is minimized by the L-BFGS-B algorithm to continuously adjust the joint angle of the robotic arm and obtain the optimal joint angle of the robotic arm.

[0037] Preferably, the gripping force of the gripper satisfies the following relationship:

[0038] Ftarget=μ×m×g×SF;

[0039] Wherein, μ represents the coefficient of friction, m represents the mass of the object, g represents the gravitational acceleration, and SF represents the safety factor.

[0040] Preferably, a force sensor is also provided, and when adjusting the clamping force of the gripper, the gripping force is adjusted in real time using a PI controller based on the feedback from the force sensor, satisfying the following relationship:

[0041] Fadjust=Ftarget+Kpe+Ki×∫ed t;

[0042] Wherein, Fadjust represents the adjusted gripper clamping force, Ftarget represents the target clamping force, Kp represents the proportional gain coefficient, e represents the error between the force sensor and the target clamping force, Ki represents the integral gain coefficient, and ∫ed t represents the integral of the error over time.

[0043] One of the above technical solutions has the following advantages or beneficial effects:

[0044] By integrating visual feedback and adaptive path planning technology, the present invention can effectively solve the shortcomings of traditional gripping devices in the recognition of complex-shaped objects and adaptability to dynamic environments. First, the improved YOLOv7 model achieves accurate classification of cylinders, cuboids, and irregular objects by adding a fine-grained classifier. Combined with the 3D pose recognition of the visual feedback module, it can significantly improve the recognition accuracy of the shape type and 3D pose of the object to be grasped, providing reliable data support for subsequent grasping strategies. Second, the gripper control module adopts a low-space-occupancy configuration strategy, forming a spatial compression posture through joint coordinated bending, and calculating the minimum swing parameters of joint movement in real time. This generates a collision-free trajectory in narrow or obstacle-filled scenarios, enhancing the robot arm's movement flexibility in confined environments. Furthermore, the dynamic constraint mechanism of gripping force based on object shape type and 3D pose, combined with segmented grasping actions and real-time trajectory adjustment, can prevent objects from being grasped due to insufficient gripping force. (Instruction manual page 3 / 14 7 CN 120962662 A) The slippage caused by this prevents damage to fragile objects due to overload clamping. Through the synergistic effect of visual perception, path optimization, and mechanical control, this invention achieves stable grasping of objects of different shapes in complex scenarios, while also balancing grasping efficiency and safety, effectively improving the operational capabilities of humanoid robots in unstructured environments.

[0045] To more clearly illustrate the technical solutions in the embodiments of this invention or the prior art, the embodiments or the prior art will be described below.The accompanying drawings used in the technical description are briefly introduced. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0046] Figure 1 is a structural schematic diagram of the humanoid robot adaptive gripper grasping device based on visual feedback provided in an embodiment of the present invention. Detailed Description

[0047] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0048] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further restrictions, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0049] In the field of industrial automation, the application of robotics technology is becoming increasingly widespread, especially in high-precision grasping tasks, where its efficiency and accuracy are crucial to the production process.

[0050] Currently, industrial robots generally adopt grasping mechanisms based on fixed programs or simple sensor feedback. This typically relies on precise programming and predetermined grasping paths, and uses monocular vision systems or basic force sensors to achieve basic object recognition and grasping. For example, some robots use traditional electric door-pushing mechanisms, operated by gatekeepers in fixed positions, lacking flexibility and intelligence.

[0051] Existing robot grasping systems perform poorly when handling complex or irregularly shaped objects because they often require customized grippers and grasping strategies for specific objects. Furthermore, these systems struggle to adapt and optimize grasping paths when facing multiple objects or narrow spaces, limiting the robot's application potential in variable environments, especially in scenarios requiring high precision and adaptability, such as automated warehouse management, precision assembly, or operations in special environments.

[0052] A humanoid robot adaptive gripper grasping device based on visual feedback, as shown in Figure 1, is a preferred embodiment of the present invention. The grasping device includes a visual feedback module, a gripper control module, and a robotic arm equipped with grippers. The robotic arm includes multiple joints.

[0053] The visual feedback module is used to perform fine-grained object classification of environmental information using an improved YOLOv7 model.After identifying the shape type and 3D pose of each object, the object to be grasped is selected, and the shape type and 3D pose of the object to be grasped are sent to the gripper control module. The improvement includes adding a fine-grained classifier to the YOLOv7 model for classifying cylinders, cuboids, and irregular objects;

[0054] The gripper control module is used to receive the shape type and 3D pose of the object to be grasped sent by the visual feedback module. In three-dimensional pose, when space is limited, based on an improved random sampling path planning algorithm, the robotic arm is controlled to move using a low-space-occupancy configuration strategy to generate a collision-free trajectory of the gripper from its initial position to the object to be grasped. The low-space-occupancy configuration strategy includes the coordinated bending of joints to form a spatial compression posture, and real-time calculation of the minimum swing parameters of each joint motion sequence. The swing parameters include joint angle constraints and motion direction priority. When the gripper approaches the object to be grasped, it is controlled to move within a preset speed threshold range. When the gripper contacts the object to be grasped, the gripping force of the gripper is constrained based on the gripping force of the gripper, the shape and type of the object, and the three-dimensional pose.

[0055] Wherein, when generating a collision-free trajectory of the object to be grasped, the minimum Euclidean distance between the robotic arm and the obstacle is calculated in real time to adjust the gripper motion trajectory in advance, and the gripper is controlled to perform segmented grasping actions.

[0056] In this embodiment, the visual feedback module is a binocular camera system optimized based on Isaac Sim, which integrates an improved YOLOv7 target detection algorithm to detect and estimate the pose of the target object. Specifically, the visual feedback module acquires image information of the target object through a binocular camera and uses an improved YOLOv7 algorithm for target detection and pose estimation. This algorithm adds a fine-grained classification branch to the traditional YOLOv7, enabling it to recognize objects of different shapes, such as cylinders, cuboids, and irregular objects. In the implementation process, the binocular camera system first acquires image data of the target object with an input resolution of 640×480. The improved YOLOv7 algorithm processes the image, identifies the target object, and estimates its pose, improving the recognition accuracy for complex-shaped objects and optimizing the inference speed to 25 FPS. Assuming the target object is a cuboid, the visual feedback system acquires its image through the binocular camera. The improved YOLOv7 algorithm identifies the bounding box of the cuboid and obtains its pose information, including position and orientation, by solving the PnP problem (based on cv2.solvePnP).

[0057] In this embodiment, the adaptive gripper adopts a two-finger gripper with a parallelogram hinge structure and a built-in strain gauge force sensor. It performs force-position hybrid control based on visual feedback information to grasp the target object. An improvement on the traditional random sampling path planning algorithm, it can generate a path based on obstacle information in the environment and the kinematic model of the robotic arm.The collision path algorithm constructs a search tree by randomly sampling in the configuration space and uses a heuristic function to guide the search direction to find a feasible path from the starting point to the target point. The low-space-occupancy configuration strategy uses coordinated bending of joints to form a spatial compression posture, reducing the space occupied by the robotic arm during movement. Multiple joints bend according to a certain cooperative relationship to form a specific posture, achieving the effect of spatial compression. After receiving the shape type and 3D pose information of the object to be grasped from the visual feedback module, the gripper control module first determines whether the current space is restricted (e.g., by detecting whether the distance between surrounding obstacles and the robotic arm is less than a preset threshold). If the space is restricted, the gripper control module calls an improved random sampling path planning algorithm to generate a collision-free trajectory from the initial gripper position to the object to be grasped, based on the kinematic model of the robotic arm and obstacle information in the environment. During trajectory generation, the robotic arm employs a low-space-occupancy configuration strategy, using coordinated joint bending to create a spatial compression posture, reducing the space occupied by the arm during movement. The minimum swing parameters of each joint's motion sequence are calculated in real-time, including joint angle constraints and motion direction priorities, to ensure the robotic arm does not collide with obstacles during movement. When the gripper approaches the object to be grasped, it moves within a preset speed threshold range, for example, limiting the speed to below 0.03 m / s to improve grasping accuracy. When the gripper contacts the object, the gripping force is constrained based on the gripping force, the object's shape, and its 3D pose. For example, the gripping force is reduced for fragile objects; increased for smooth surfaces to prevent slippage; and appropriate force correction factors are set for objects that are not kicked: standard gripping force for rigid objects; a correction factor of 0.6-0.8 for deformable objects to prevent excessive deformation; a lower correction factor of 0.3-0.5 for fragile objects; and a higher correction factor for high-density objects and low-friction surfaces. The extension speed of the robotic arm and the gripping force of the gripper can be adjusted at different times during different grasping phases. Approach phase (t0-t1): The duration is calculated based on distance and safe acceleration, the maximum approach speed is set, and the speed limit is reduced for fragile objects. Initial contact phase (t1-t2): 100-200ms is maintained, using extremely low-speed contact, and the contact force threshold is set to 0.5-1.0N. Force growth phase (t2-t3): The force growth slope is calculated based on the target force and the basic growth time, and different maximum force growth rates are set for different types of objects, dynamically adjusting the growth rate through strain feedback. Stable grasping phase (t3-t4): The duration is determined according to task requirements, the force value is maintained within ±5% of the target force, and micro-vibrations are applied to improve grasping stability. Release phase (t4-t5): Generally, the force is set to decrease.The rate is 1.5-2 times the force growth rate, and a gentler release curve is used for special objects.

[0058] In the specific implementation process, such as using a vision system to process the images acquired by the left and right cameras to obtain the position information of the glass bottle, extracting the position information of the glass bottle, and setting the size (radius and height), shape type, material, surface characteristics and estimated mass of the glass bottle according to known conditions, forming a dictionary containing various information of the glass bottle, which is convenient for subsequent steps to call. Take 40% of the height of the glass bottle as the gripping height, so that the gripping point is close to the center of gravity of the bottle to improve gripping stability. Based on the original position of the glass bottle, adjust the height along the Z-axis to obtain the specific gripping position. By calculating the approach vector perpendicular to the axis of the bottle, the gripper can accurately and stably contact the bottle body, and obtain the direction vector that the gripper should follow when approaching the bottle body. According to the estimated mass and gravitational acceleration of the glass bottle, calculate the basic force and multiply it by a safety factor of 2.5 to obtain the maximum gripping force. Set the holding force to 95% of the maximum gripping force to maintain a high continuous gripping force, and generate a force curve configuration file containing gripping force related information. After the inspection is completed, the texture of the gripper fingertip is switched to a high-friction type, which can increase the friction with the surface of the glass bottle. The robotic arm is controlled to move to the calculated gripping position and move in the set approach direction. The robot controls the gripper to perform the gripping action and apply the corresponding gripping force according to the generated force curve configuration file.

[0059] In addition, in one embodiment of the present invention, a corresponding adaptive adjustment mechanism is also provided under vibration conditions: 1. Vibration detection and analysis: The vibration detection algorithm is adopted to analyze the vibration characteristics through IMU sensor and force sensor data: the root mean square calculation and FFT spectrum analysis of the acceleration data are performed to extract the main frequency and amplitude characteristics of the vibration. The vibration degree is classified into three levels: low, medium and high according to the vibration characteristics; 2. Dynamic adjustment of gripping strategy: Force adjustment compensator: The gripping force is dynamically adjusted based on the vibration frequency and amplitude. Different compensation coefficients are used for different frequency vibrations: a larger frequency compensation factor is used for low frequency vibration (<8Hz), and a larger frequency compensation factor is used for high frequency vibration. Using different adjustment strategies, a safety factor is calculated by combining multiple factors, with a maximum limit of 2.5. The impedance control parameters are adaptively adjusted, and the stiffness and damping parameters are dynamically adjusted according to the vibration level: low vibration: increase stiffness, slightly increase damping; medium vibration: decrease stiffness, significantly increase damping; high vibration: significantly decrease stiffness, significantly increase damping. 3. The vision-force collaborative anti-interference control system realizes a multi-modal information fusion and prediction compensation mechanism: using a Kalman filter to fuse visual and force sensing data, predicting the future position of the target object based on the vibration mode, adjusting the robotic arm control parameters according to the vibration characteristics, and then calculating and applying the compensated grasping force in real time.

[0060] Preferably, the visual feedback module has a built-in camera, and the improved YOLOv7 model is used to process environmental information.The process of fine-grained object classification, identifying the shape type and 3D pose of each object, and selecting the object to be grasped includes:

[0061] Loading the YOLOv7 model and fine-grained classifier, generating a depth map of the object after setting camera parameters, and obtaining the depth information of the target object;

[0062] The YOLOv7 model is an advanced object detection algorithm with real-time performance and high accuracy. It detects and locates objects in the image through a deep learning network and outputs information such as the bounding box coordinates, category, and confidence score of the object. The fine-grained classifier refers to the classification branch added on the basis of the YOLOv7 model, which is used to classify objects in a more detailed way, such as distinguishing objects of different shapes (cylinders, cuboids, irregular objects, etc.). Camera parameter settings include the camera's intrinsic parameters (such as focal length, principal point coordinates, etc.) and extrinsic parameters (such as the camera's position and pose in the world coordinate system), which are used to convert points in the image coordinate system into points in the world coordinate system. Depth map generation is achieved by using two images acquired by a binocular camera and calculating the depth information of each pixel using the parallax principle to generate a depth map. Each pixel value in the depth map represents the distance of that point from the camera (page 6 / 14 of the manual, 10 CN 120962662 A).

[0063] In the logistics sorting scenario, packages of various shapes are placed on the conveyor belt, which need to be grasped and sorted by a robotic arm. The visual feedback module first loads the YOLOv7 model and fine-grained classifier, and then sets the parameters of the binocular camera so that it can accurately capture images of the packages on the conveyor belt. After the binocular camera acquires the images of the packages, it generates a depth map and obtains the depth information of each package from it. For example, the depth range of package A is [100, 150] mm, and the depth range of package B is [150, 200] mm, etc., which lays the foundation for subsequent object recognition and pose estimation.

[0064] Target detection is performed on the depth map to obtain the bounding box coordinates, object type, and confidence information of all objects within the depth map. Based on the bounding box coordinates, object type, and confidence information, objects to be captured are selected. The depth information of the region to be captured is extracted from the depth map according to the bounding box coordinates of the object to be captured, and the average depth is calculated.

[0065] The average depth calculation involves extracting the depth information of the region to be captured from the depth map, calculating the average depth value of all pixels in that region, and using this average depth as the object's average depth to estimate the distance between the object and the camera.

[0066] The generated depth map is input into the YOLOv7 model for target detection. The bounding box coordinates, object type, and confidence information of all objects within the depth map are obtained. According to preset selection rules (such as selecting the object with the highest confidence, selecting objects of a specific type, etc.), objects to be captured are selected from the detection results. Based on the bounding box coordinates of the object to be captured, the depth information of the region to be captured is extracted from the depth map.The depth information of the object region is extracted from the depth map, and the average value of the extracted depth information is calculated to obtain the average depth of the object to be grasped, providing data for subsequent three-dimensional pose calculation.

[0067] For example, in the above logistics sorting scenario, the YOLOv7 model performs target detection on the depth map, outputs the bounding box coordinates, type (such as cylindrical cans, cuboid boxes, etc.) and confidence information of all packages on the conveyor belt, selects the object to be grasped in order of confidence from high to low, assuming that the cuboid box with the highest confidence is selected as the target, and then extracts the depth information of its region from the depth map according to the bounding box coordinates of the box, calculates that the average depth of the box is 180mm, indicating that the average distance of the box from the camera is 180mm, providing a distance reference for subsequent grasping actions.

[0068] Based on the pixel coordinates and depth information of the center point of the object to be grasped, the pixel coordinates are converted into three-dimensional coordinates in the world coordinate system. Fine-grained shape classification is performed on the area of ​​the object to be grasped to determine the shape type of the target object.

[0069] The conversion from pixel coordinates to the world coordinate system is achieved by using the camera's intrinsic and extrinsic parameters to convert the pixel coordinates and depth information of the object in the image into three-dimensional coordinates in the world coordinate system. Through coordinate transformation, the position and orientation of the object in actual space can be determined.

[0070] In the implementation process, the pixel coordinates (e.g., coordinates (u, v)) and depth information (e.g., depth value d) of the center point of the object to be grasped are first obtained. Using the camera's intrinsic and extrinsic parameter matrices, the pixel coordinates (u, v) and depth value d are converted into three-dimensional coordinates (X, Y, Z) in the world coordinate system. Shape analysis is performed on the area of ​​the object to be grasped to extract the shape features of the object (e.g., edge contours, symmetry, etc.). The extracted shape features are input into a fine-grained classifier to determine the shape type of the target object.

[0071] For example, in a logistics sorting scenario, the pixel coordinates of the center point of the box to be grasped are (320, 240), and the depth information is 180mm. Using the camera's intrinsic parameter matrix (including parameters such as focal length and principal point coordinates) and extrinsic parameter matrix (including the camera's position and orientation in the world coordinate system), the pixel coordinates and depth value are converted into three-dimensional coordinates in the world coordinate system. Assuming the obtained three-dimensional coordinates are (100, 200, 180)mm. Then, the shape of the box area is analyzed, its edge contour and other features are extracted, and input into a fine-grained classifier to determine that the shape type of the box is a cuboid, providing a basis for the subsequent grasping strategy.

[0072] According to the shape type of the object to be grasped, the size estimation algorithm is called to estimate the size information of the object to be grasped. The position, shape type and size information of the target object are integrated to generate the first dataset required for grasping. Specification 7 / 14 pages 11 CN 120962662 A

[0073] Size estimation algorithms estimate the actual size of an object based on its shape and features in the image. For example, for a cylinder, its diameter and height in the image can be measured and combined with depth information to calculate the actual diameter and height. For a cuboid, its length, width, and height in the image can be measured and combined with depth information to calculate the actual size. The first dataset is a dataset that integrates the position (three-dimensional coordinates), shape, and size information of the target object to guide the gripper's grasping action.

[0074] For example, in a logistics sorting scenario, after determining that the shape of the box to be grasped is a cuboid, its length, width, and height in the image are measured to be 100 pixels, 80 pixels, and 60 pixels, respectively. Combined with depth information of 180 mm and parameters such as the camera's focal length, the actual size of the box is calculated to be 200 mm long, 160 mm wide, and 120 mm high. The box's position (100, 200, 180) mm, shape type (cubic prism), and size information (200 mm × 160 mm × 120 mm) are integrated into the first dataset to provide detailed data guidance for the robotic arm's gripping action, ensuring that the gripper can accurately adapt to the size of the box and achieve stable gripping.

[0075] Preferably, before controlling the gripper to perform the grasping action, the method includes constructing a hand-eye transformation matrix for mapping the three-dimensional pose of the object to be grasped to the robot base coordinate system:

[0076] Obtain the pose list of the gripper and the pose list of the calibration board, traverse each pair of adjacent poses, multiply the inverse of the gripper pose matrix of the previous robot with the gripper pose matrix of the current robot to obtain the robot pose transformation matrix, and multiply the current calibration board pose matrix with the inverse of the previous calibration board pose matrix to obtain the calibration board pose transformation matrix;

[0077] Wherein, the gripper pose list includes recording the pose information of the gripper in different positions and postures of the robotic arm, including position coordinates and posture angles; the calibration board pose list includes recording the pose information of the calibration board in different positions and postures, used for hand-eye calibration; the robot pose transformation matrix represents the transformation relationship of the robot from one pose to another, and is obtained by multiplying the inverse of the previous pose matrix with the current pose matrix. The calibration board pose transformation matrix represents the transformation relationship of the calibration board from one pose to another, and is obtained by multiplying the current calibration board pose matrix by the inverse of the previous calibration board pose matrix.

[0078] For example, in a laboratory object grasping environment, in order to calibrate the hand-eye relationship between the gripper of the robotic arm and the camera, the robotic arm is first moved to multiple different positions and postures, and the gripper pose information (such as position coordinates (x, y, z) and attitude angles (roll, pitch, yaw)) is recorded at each position to form a gripper pose list. At the same time, a calibration board of known size is placed at each position, and the pose information of the calibration board (such as its position and attitude in the world coordinate system) is recorded to form a calibration board pose list.The calibration board pose list is generated. Then, each pair of adjacent gripper poses and calibration board poses is traversed, for example, the first and second poses, to obtain the corresponding gripper pose matrices Trobot1 and Trobot2, and the robot pose transformation matrix Trobot_transform = Trobot2 × Trobot1 - 1 is calculated; similarly, the calibration board pose matrices Tboard1 and Tboard2 are obtained, and the calibration board pose transformation matrix Tboard_transform = Tboard2 × Tboard1 - 1 is calculated.

[0079] Extract the first rotation matrix and the first translation vector from all robot pose transformation matrices, extract the second rotation matrix and the second translation vector from all calibration board pose transformation matrices, derive the hand-eye rotation matrix based on the first rotation matrix and the second rotation matrix, and derive the hand-eye translation vector based on the first rotation matrix, the hand-eye rotation matrix, the first translation vector, and the second translation vector;

[0080] The first rotation matrix and the first translation vector are the rotation and translation parts extracted from the robot pose transformation matrix, respectively, representing the rotation and translation changes of the robot between adjacent poses; the second rotation matrix and the second translation vector are the rotation and translation parts extracted from the calibration board pose transformation matrix, respectively, representing the rotation and translation changes of the calibration board between adjacent poses; the hand-eye rotation matrix represents the rotation relationship of the camera relative to the gripper, obtained by comparing the rotation parts of the robot pose transformation matrix and the calibration board pose transformation matrix; the hand-eye translation vector represents the translation relationship of the camera relative to the gripper, calculated by combining the first rotation matrix, the hand-eye rotation matrix, the first translation vector, and the second translation vector. (Page 8 / 14, 12 CN) 120962662 A arrived.

[0081] Construct a homogeneous transformation matrix by filling the hand-eye translation vector and hand-eye rotation matrix into the homogeneous transformation matrix to obtain the hand-eye transformation matrix.

[0082] The hand-eye transformation matrix represents the transformation relationship between the camera coordinate system and the gripper coordinate system. It is used to transform the three-dimensional pose of the object to be grasped from the camera coordinate system to the robot base coordinate system, thereby guiding the grasping action of the robotic arm.

[0083] This embodiment describes in detail the process of constructing the hand-eye transformation matrix before controlling the gripper to perform the grasping action. By obtaining the pose list of the gripper and the calibration plate, the robot pose transformation matrix and the calibration plate pose transformation matrix are calculated, and then the rotation matrix and translation vector are extracted to finally obtain the hand-eye rotation matrix and translation vector, and construct the homogeneous transformation matrix. This process can ensure that the three-dimensional pose information of the object obtained by the camera can be accurately mapped to the robot base coordinate system, providing a key coordinate transformation basis for the precise grasping of the robotic arm. In practical applications, such as the precise grasping and assembly of parts on industrial production lines, the accuracy of the hand-eye transformation matrix directly affects the success rate and efficiency of the robotic arm's grasping. This embodiment demonstrates this.This method can effectively improve the accuracy of hand-eye calibration, thereby enhancing the robotic arm's ability to operate and adapt in complex environments.

[0084] Preferably, before the gripper contacts the object to be gripped, the following steps are included:

[0085] Initializing gripper dynamics model parameters, gripper physical parameters, and MPC parameters. The gripper dynamics model parameters include a state transition matrix, a control matrix, an output matrix, and a direct transfer matrix. The gripper physical parameters include the gripper's opening width and the gripping force threshold. The MPC parameters include a prediction time domain, a control time domain, a state error weight, and a control input weight.

[0086] The gripper dynamics model parameters are used to describe the state change law of the gripper under different control inputs and are the basis for predicting the future state of the gripper in the MPC algorithm. The gripper physical parameters involve the range of the gripper's opening width (maximum and minimum opening width) and the gripping force threshold. These parameters determine the physical limits and safety range of the gripper in actual operation. The MPC parameters include a prediction time domain (the time range for the algorithm to predict the future state), a control time domain (the time range for the control input to act in each optimization calculation), a state error weight (measuring the importance of the state deviation), and a control input weight (measuring the cost of changes in the control input). These parameters directly affect the optimization objective and control performance of the MPC algorithm.

[0087] For example, in a food processing workshop, a robotic arm needs to grab fragile eggs for packaging. Based on the characteristics of the eggs and the design of the gripper, the gripper dynamic model parameters are initialized: the state transition matrix A describes the rate of change of the opening width of the gripper under no external force, assumed to be 0.8 (meaning that the opening width is reduced to 80% of the original value per unit time); the control matrix B describes the effect of motor torque on the opening width, assumed to be 0.5 (meaning that the opening width changes by 0.5 mm per unit torque); the output matrix C directly outputs the opening width, which is 1; the direct transmission matrix D is 0. The gripper physical parameters are initialized to an opening width of 100 mm and a gripping force threshold of 5 N (exceeding 5 N may crush the eggs). The MPC parameters are initialized to a prediction time domain of 0.5 seconds, a control time domain of 0.2 seconds, a state error weight [20,10] (paying more attention to the error of the opening width), and a control input weight [2] (appropriately considering energy consumption).

[0088] The state error weight, the initial gripping force of the gripper, and the gripping force threshold are adjusted according to the fragility of the object to be gripped;

[0089] Fragility indicates the sensitivity of the object to the gripping force, and is usually classified according to the material and structural characteristics of the object, such as fragile, relatively fragile, normal, and strong; the state error weight adjustment is to adjust the element values ​​in the state error weight matrix according to the fragility of the object, so as to change the emphasis on the opening width error and the gripping force error; the initial gripping force is the initial force applied by the gripper when it first contacts the object, and is adjusted according to the fragility of the object. Fragile objects require a smaller initial gripping force.The initial clamping force; the clamping force threshold adjustment is to adjust the maximum allowable value of the clamping force according to the fragility of the object. The clamping force threshold of fragile objects should be lower. Instruction manual 9 / 14 pages 13 CN 120962662 A

[0090] In the food processing workshop, the eggs to be grasped are fragile objects. According to their fragility, the state error weight is adjusted from [20,10] to [10,20], paying more attention to the control accuracy of the clamping force; the initial clamping force is adjusted from 3N to 1N to reduce the initial impact on the egg; the clamping force threshold is adjusted from 5N to 3N to ensure that the clamping force does not exceed the egg's tolerance range and avoid crushing the egg.

[0091] A first prediction matrix is ​​generated based on the power operation of the state transition matrix, and a second prediction matrix is ​​generated based on the product operation of the power of the state transition matrix and the control matrix. An objective function is constructed based on the first and second prediction matrices.

[0092] The power operation of the state transition matrix is ​​to predict the future evolution trend of the system state by calculating the power of the state transition matrix A (e.g., A^k, where k is the time step). The first prediction matrix is ​​generated by the power operation of the state transition matrix and represents the natural evolution of the system state in the prediction time domain. The second prediction matrix is ​​generated by the product operation of the power of the state transition matrix and the control matrix B and represents the influence of the control input on the system state in the prediction time domain. In the MPC algorithm, the objective function is a function used to measure the control effect. It usually includes the weighted sum of the state error and the control input. The optimal control input is solved by minimizing the objective function.

[0093] Under the constraints of the control time domain and the clamping force threshold, the objective function defines the optimization problem by the quadratic combination of the state error weight and the control input weight. The clamping force value of the gripper is obtained by solving the problem through convex optimization.

[0094] In the prediction time domain, the actual time period for applying the control input, for example, in a prediction time domain of 0.5 seconds, the control time domain is 0.2 seconds, indicating that the control input is applied only in the first 0.2 seconds, after which the system evolves freely. The quadratic combination combines the state error and the control input into the objective function in the form of a quadratic form, which is convenient for solving convex optimization problems; convex optimization is a mathematical optimization method used to solve for the minimum value of a convex function, and is often used in the MPC algorithm to solve for the optimal control input.

[0095] In the food processing workshop, the control time domain is 0.2 seconds, the discrete time is 0.1 seconds, the number of time steps for applying the control input is 2 steps, and a clamping force threshold constraint (maximum clamping force is 4N) is added to the objective function. The minimum value of the objective function J is solved by the convex optimization algorithm to obtain the optimal motor torque control input u, thereby obtaining the clamping force value of the gripper. Assuming the obtained control input u is [0.8, 0.6] (in N·m), the corresponding clamping force is 3N, which satisfies the clamping force threshold constraint, ensuring that the clamping force will not exceed the egg's tolerance range when grasping the egg, thus achieving stable and safe grasping.

[0096] Preferably, constraining the gripping force of the gripper based on the gripping force of the gripper, the shape type of the object, and the three-dimensional pose when the gripper contacts the object to be grasped includes:

[0097] Selecting a corresponding gripping strategy according to the shape type of the object to be grasped, calculating the initial opening and closing width of the gripper and the target gripping force, acquiring contact deformation data through a force sensor during the pre-gripping stage, and evaluating the softness of the object material based on the strain change rate;

[0098] The gripping strategy selects different gripping methods and parameters based on the shape and type of the object to achieve the best gripping effect. For example, a cylinder may be gripped using a wraparound gripping method, a cuboid using a surface contact gripping method, and an irregular object using an adaptive gripping method. The initial opening width is the opening width of the gripper when it first contacts the object. It is calculated based on the size and shape of the object to ensure that the gripper can smoothly approach and contact the object. The target gripping force is the expected gripping force calculated based on the object's weight, coefficient of friction, and shape to achieve stable gripping. A force sensor is installed on the gripper to measure the gripping force in real time. It is usually based on the principle of strain gauges and calculates the force by detecting changes in strain. The contact deformation data is the deformation data of the object's surface when the gripper contacts the object. It is acquired by the force sensor and reflects the material properties and stress conditions of the object. The strain change rate is the rate of change of strain of the strain gauge in the force sensor over time. It is used to assess the softness of the object's material. Objects with high softness have a larger strain change rate when subjected to force, and vice versa.

[0099] In specific implementation, the corresponding clamping strategy is selected according to the shape type of the object to be grasped: a. Cylindrical object: a surround clamping method is adopted, with the grippers surrounding the object from both sides; b. Cuboid: a surface contact clamping method is adopted, with the grippers clamping parallel to one face of the object; c. Irregular object: an adaptive clamping method is adopted, with the grippers automatically adjusting the clamping angle and position according to the shape of the object; the initial opening and closing width of the grippers is calculated: a. For a cylinder, the initial opening and closing width is equal to the diameter of the object plus a certain safety margin (e.g., 2mm); b. For a cuboid, the initial opening and closing width is equal to the minimum side length of the object plus a safety margin; c. For irregular objects, the initial opening and closing width is calculated based on the convex hull size of the object. Calculating the target clamping force: a. Based on the object's weight, coefficient of friction, and safety factor, calculate using the formula F = μ × m × g × k, where μ is the coefficient of friction, m is the object's mass, g is the acceleration due to gravity, and k is the safety factor (usually taken as 1.2–2.0). During the pre-clamping stage, the grippers approach the object at a low speed. Upon contact, the force sensor begins acquiring contact deformation data. Calculating the strain rate of change involves calculating the strain rate of change by performing time derivative calculations on the strain data acquired by the force sensor. The strain rate of change is used to assess the flexibility of the object's material.Softness: a. A large strain rate indicates that the material is relatively soft; b. A small strain rate indicates that the material is relatively hard.

[0100] The stiffness coefficient and the maximum allowable clamping force threshold of the gripper are dynamically adjusted according to the softness assessment results. A model prediction controller is used to generate gripper motion commands that satisfy the force-position constraints, and a progressive clamping operation is performed until the gripper reaches a stable grasping state;

[0101] Specifically, according to the softness assessment results, the stiffness coefficient and the maximum allowable clamping force threshold of the force controller are dynamically adjusted: a. For soft objects, the stiffness coefficient is reduced (e.g., from 100N / m to 50N / m), and the maximum allowable clamping force threshold is reduced (e.g., from 10N to 3N); b. For hard objects, the stiffness coefficient and the maximum allowable clamping force threshold can be appropriately increased; Model predictive controllers are used to generate gripper motion commands: a. A force-position hybrid control model for the gripper is established, including a dynamic model of the gripper and a feedback model of the force sensor; b. In each control cycle, the opening and closing width and gripping force changes of the gripper over a future period are predicted; c. Based on the prediction results, the current control input (such as motor torque) is optimized to ensure that the opening and closing width and gripping force of the gripper meet the force-position constraints and are as close as possible to the target values. Progressive gripping operations are performed: a. In the initial stage, the gripper is slowly closed with a small gripping force, and the gripping force and the deformation of the object are monitored in real time; b. Based on the monitoring data, the gripping force is gradually increased until a stable gripping state is reached, i.e., the gripping force reaches the target value and the object does not slip significantly or deform excessively.

[0102] Specifically, when the shape of the object to be grasped is a cylinder, the initial opening and closing width of the gripper is calculated based on the diameter measurement value, and the contact point distribution is determined according to the height of the cylinder; when the shape of the object to be grasped is a cuboid, the initial opening and closing width of the gripper is calculated based on the minimum edge distance principle, and the contact area is determined according to the center point of the surface; when the shape of the object to be grasped is an irregular object, the maximum stable grasping width is extracted based on the convex hull algorithm, and the centroid aligned gripping point is calculated.

[0103] Wherein, the diameter measurement value represents the diameter of the cylinder measured by the visual feedback module; the contact points are distributed on the surface of the cylinder, and the position distribution of the gripper contact is determined according to the height and diameter of the cylinder to achieve a uniform gripping force distribution; the minimum edge distance principle selects the shortest edge among the edges of the cuboid as the gripping direction to ensure that the gripper can firmly grip the object; the face center point is the center point of each face of the cuboid, used to determine the contact area of ​​the gripper, so that the gripper can grip the object parallel to the face; the convex hull algorithm is a computational geometry algorithm used to extract the convex hull of the object, that is, the smallest convex polygon or convex polyhedron containing all points of the object, thereby determining the maximum stable gripping width of the object; the centroid aligned gripping point is determined according to the position of the centroid of the object, so that the gripping force of the gripper passes through the centroid of the object, improving the stability and balance of gripping.

[0104] This embodiment details how, when the gripper contacts the object to be grasped, a gripping strategy is selected based on the object's shape, the initial opening and closing width and target gripping force are calculated, and during the pre-gripping stage, contact deformation data is obtained through a force sensor to assess the object's material softness. The force controller parameters are then dynamically adjusted, and a model predictive controller is used to generate gripper motion commands that satisfy force-position constraints, executing a progressive gripping operation until a stable gripping state is achieved. Furthermore, for objects of different shapes, corresponding methods for calculating the initial opening and closing width and distributing contact points are employed to ensure the gripper can adapt to the gripping needs of various shapes. This series of technical measures effectively improves the adaptive gripping capability of the robotic arm in complex environments, especially when handling fragile, soft, or irregularly shaped objects, enabling precise and stable gripping, improving production efficiency and product quality, and expanding the application potential of the robotic arm in more fields.

[0105] Preferably, controlling the robotic arm to move using a low-space-occupancy configuration strategy includes:

[0106] Calculating the spatial constraint degree based on the three-dimensional Euclidean distance between the object to be grasped and the obstacle;

[0107] The three-dimensional Euclidean distance refers to the straight-line distance between two points in three-dimensional space; the spatial constraint degree represents the degree of spatial restriction experienced by the robotic arm when moving around the object to be grasped. It is determined by calculating the ratio of the minimum three-dimensional Euclidean distance between the object to be grasped and the obstacle to parameters such as the size of the robotic arm. The larger the spatial constraint degree, the more restricted the space.

[0108] For example, inside a narrow industrial pipe, the object to be grasped is located at coordinates (100, 200, 300) mm, and the obstacle (inner wall of the pipe) is located at coordinates (150, 200, 300) mm. The calculated three-dimensional Euclidean distance d = 50 mm, and the maximum extension length L of the robotic arm is 500 mm. Therefore, the spatial constraint degree S = 50 / 500 = 0.1, indicating that the space is relatively restricted and a low-space-occupancy configuration strategy is required.

[0109] When the spatial constraint degree is greater than the preset threshold, the optimal direction vector of the gripper approaching the object to be grasped is first calculated. Based on the optimal direction vector, the initial pose of the robotic arm with continuous S-shaped bending characteristics is generated by inverse kinematics solution. The joint angles of the robotic arm are iteratively adjusted until the robotic arm reaches the minimum space occupation state. The multi-segment motion trajectory of the gripper is generated according to the adjusted joint angles and the preset gripper movement speed.

[0110] Wherein, the preset threshold is used to determine whether the space is restricted. It is set according to the working space and application scenario of the robotic arm. For example, it is set to 0.2. When the spatial constraint degree is less than 0.2, the space is considered restricted. The optimal direction vector represents the direction vector that minimizes the space occupied by the robotic arm when the gripper approaches the object to be grasped. It points to the direction between the object to be grasped and the obstacle.The direction with the largest gap; Inverse kinematics solution means solving the angle of each joint according to the target position and direction of the end effector (gripper) of the robotic arm, so that the robotic arm can reach the target position and direction; Continuous S-shaped bending feature means that the joints of the robotic arm bend according to a certain rule to form a continuous S-shaped curve, reducing the space occupation of the robotic arm during the movement process; Iterative adjustment optimizes the pose of the robotic arm by adjusting the joint angles of the robotic arm multiple times, so as to achieve the minimum space occupation state; Multi-segment motion trajectory divides the motion trajectory of the gripper into multiple stages, each stage corresponding to different joint angles and positions, so as to achieve a smooth and collision-free grasping action.

[0111] For example, in the narrow industrial pipe, the spatial constraint S = 0.1, which is less than the preset threshold of 0.2. Then, the optimal direction vector of the gripper approaching the object to be grasped is calculated, which is assumed to be vector (-1,0,0), that is, along the negative X-axis direction. Based on this direction vector, the target position of the gripper is set to (90, 200, 300) mm, with the direction consistent with the optimal direction vector. The initial pose of the robotic arm is generated by inverse kinematics solution, so that the robotic arm presents a continuous S-shaped bend, reducing the space occupation in the pipe. Then, the joint angle is iteratively adjusted, and the space occupation volume is calculated after each adjustment, gradually optimizing to the minimum space occupation state. Finally, based on the adjusted joint angle and the preset gripper movement speed of 0.03 m / s, a multi-segment motion trajectory of the gripper is generated: first, the gripper moves along the optimal direction vector at a speed of 0.03 m / s, approaching the object to a certain distance; then, the joint angle of the robotic arm is adjusted, and the position and direction of the gripper are fine-tuned to ensure accurate grasping of the object; finally, the grasping action is completed, and the entire process avoids collision with the inner wall of the pipe.

[0112] When the spatial constraint degree is less than the preset threshold, the grasping trajectory is generated based on the shape type and three-dimensional pose of the object to be grasped.

[0113] When the spatial constraint degree is less than the preset threshold, such as on an open workbench, the object to be grasped is a cuboid box, the spatial constraint degree S = 0.5, which is greater than the preset threshold of 0.2. According to the shape and three-dimensional pose of the box (position is (300, 400, 500) mm, and the orientation is horizontal), a parallel grasping trajectory is generated. The gripper approaches the box from above in parallel, aligns with one face of the box, and then closes the gripper to complete the grasping action. The whole process does not need to consider the spatial constraint problem, and the grasping trajectory is simple and direct.

[0114] This invention determines whether the space is restricted by calculating the spatial constraint degree. When the space is restricted, the optimal direction vector is calculated, and the initial pose of the robotic arm with continuous S-shaped bending characteristics is generated by inverse kinematics solution. The joint angles are iteratively adjusted to the minimum space occupation state, and multiple motion trajectories are generated to ensure that the gripper can smoothly and accurately approach and grasp the object.It avoids collisions with obstacles. When the space is relatively open, the grasping trajectory is directly generated based on the shape and three-dimensional pose of the object, simplifying the control process. This strategy effectively improves the adaptability of the robotic arm in different spatial environments, especially in narrow or obstacle-filled scenarios, which can significantly improve the success rate and efficiency of grasping, expand the application range of the robotic arm, and meet the grasping needs in more complex environments.

[0115] Preferably, when controlling the gripper to perform the grasping action, the method further includes:

[0116] calculating the initial joint angle configuration of the robotic arm based on the target position of the gripper and inverse kinematics;

[0117] obtaining the space volume occupied by the robotic arm of the humanoid robot in the current configuration, calculating the position error between the actual position and the target position of the gripper, calculating the minimum distance between the robotic arm and the obstacle, and obtaining a comprehensive score based on the space volume occupied by the robotic arm, the position error and the minimum distance, wherein the comprehensive score is used to measure the quality of the current joint angle configuration;

[0118] The space volume occupied by the robotic arm represents the three-dimensional space volume occupied by the robotic arm in the current configuration, which can be obtained by calculating the geometry and position of each link of the robotic arm; the minimum distance between the robotic arm and the obstacle is calculated by detecting the geometric relationship between each part of the robotic arm and the obstacle; the comprehensive score is a weighted combination of the space volume occupied by the robotic arm, the position error and the minimum distance to obtain a score that measures the quality of the current joint angle configuration, and the lower the comprehensive score, the better the joint angle configuration.

[0119] The comprehensive score is calculated as w1 * volume of space occupied by the robotic arm + w2 * position error between the actual position and the target position of the gripper - w3 * minimum distance between the robotic arm and the obstacle. Here, w1, w2, and w3 are weighting coefficients, set according to actual needs, for example, w1 = 0.5, w2 = 2.0, w3 = 0.3. On an automated assembly line, the current joint angles of the robotic arm are configured as θ1 = 30°, θ2 = 45°, θ3 = 60°, θ4 = 0°, θ5 = 0°, θ6 = 0°. The calculated volume of space occupied by the robotic arm is V = 0.1 cubic meters, the actual position of the gripper is (200, 300, 390) mm, the position error between the gripper and the target position (200, 300, 400) mm is eposition = 10 mm, the minimum distance between the robotic arm and the obstacle is dmin = 50 mm, and the weighting coefficients are w1 = 0.5, w2 = 2.0, w3 = 0. 3. Calculate the comprehensive score: Comprehensive score = 0.5 × 0.1 + 2.0 × 0.01 + 0.3 × 0.05 = 0.05 + 0.02 + 6 = 0.085.

[0120] Taking the initial joint angle configuration of the robotic arm as the starting point for optimization, the comprehensive score is minimized using the L-BFGS-B algorithm to continuously adjust the joint angles of the robotic arm and obtain the optimal joint angles.

[0121] The L-BFGS-B algorithm is a BFGS algorithm with limited memory, suitable for optimization problems with boundary constraints. It uses gradient information to optimize by approximating the Hessian matrix, and can efficiently handle high-dimensional optimization problems.

[0122] Specifically, assuming that on an automated assembly line, the initial joint angle configuration θ1=30°, θ2=45°, θ3=60°, θ4=0°, θ5=0°, θ6=0° is used as the optimization starting point, and the boundary constraints of the joint angles are set. The L-BFGS-B algorithm is applied for optimization, and the joint angles are adjusted in each iteration, and the comprehensive score is calculated. After multiple iterations, the comprehensive score gradually decreases, and finally converges to the minimum value. The joint angle configuration corresponding to this minimum value is θ1=32°, θ2=48°, θ3=55°, θ4=5°, θ5=3°, θ6=2°. At this time, the space occupied by the robotic arm is reduced, the position error is close to zero, the minimum distance is increased, and the comprehensive score reaches the optimal level, ensuring that the robotic arm occupies the least space, has the highest position accuracy, and maintains a safe distance from obstacles during the grasping process.

[0123] When controlling the gripper to perform the grasping action, the initial joint angle configuration of the robotic arm is calculated based on the target position of the gripper and inverse kinematics. A comprehensive score is calculated by obtaining the space volume occupied by the robotic arm, the position error and the minimum distance. The comprehensive score is minimized using the AL-BFGS-B algorithm on pages 13 / 14 of the specification (CN 120962662) to optimize the joint angle configuration of the robotic arm and obtain the optimal joint angle. This process ensures that the robotic arm can move in the optimal posture during the grasping process, which reduces space occupation and improves the grasping accuracy and safety, and avoids collisions with obstacles. Especially in complex working environments, by optimizing the joint angle configuration, the working efficiency and reliability of the robotic arm can be effectively improved, and its adaptability in diverse scenarios can be enhanced.

[0124] Preferably, the gripping force of the gripper satisfies the following relationship:

[0125] Ftarget=μ×m×g×SF;

[0126] Wherein, μ represents the coefficient of friction, m represents the mass of the object, g represents the gravitational acceleration, and SF represents the safety factor.

[0127] The friction coefficient μ can be obtained through experimental measurement or by consulting a material friction coefficient table. A suitable safety factor SF is selected based on the stability and reliability requirements of the gripping action. Specifically, when gripping an object with a mass of 0.5 kg, the friction coefficient μ of the contact surface between the object and the gripper is 0.4, and the safety factor k is 1.5. The minimum gripping force required by the gripper, calculated using the formula, is F = 0.4 × 0.5 × 9.8 × 1.5 = 2.94 N. Therefore, the gripping force of the gripper should be at least 2.94 N to ensure stable gripping of the object and prevent the object from slipping due to insufficient gripping force.

[0128] Preferably, a force sensor is also provided, and adjusting the gripping force of the gripper includes using a PI controller based on the force sensor...The feedback of the device adjusts the clamping force in real time, satisfying the following relationship:

[0129] Fadjust=Ftarget+Kpe+Ki×∫e dt;

[0130] Wherein, Fadjust represents the adjusted clamping force of the gripper, Ftarget represents the target clamping force, Kp represents the proportional gain coefficient, e represents the error between the force sensor and the target clamping force, Ki represents the integral gain coefficient, and ∫e dt represents the integral of the error with respect to time.

[0131] Specifically, the force sensor is installed on the gripper and is used to measure the clamping force in real time; the target clamping force is the desired clamping force set according to the characteristics of the object and the grasping requirements; the proportional gain coefficient is used to adjust the response speed and amplitude of the control input to the error; the integral gain coefficient is used to eliminate the steady-state error of the system. The integral of the error with respect to time represents the accumulation of the error between the force sensor measurement value and the target clamping force over time, and is used to reflect the cumulative deviation of the system.

[0132] When gripping a fragile object, the target clamping force Ftarget is set to 3N. The clamping force Fclamping measured in real time by the force sensor is initially 2.5N. The proportional gain coefficient Kp = 0.8, the integral gain coefficient Ki = 0.2, and the calculation error e = 3 - 2.5 = 0.5N. Assuming that the error remains constant within time t, the integral term is 0.2 × 0.5 × t. The adjusted clamping force Fadjust = 3 + 0.8 × 0.5 + 0.2 × 0.5 × t. As time goes on, Fadjust gradually increases, causing the clamping force of the gripper to gradually increase and approach 3N, ensuring stable gripping of the fragile object without damage.

[0133] In the description of this specification, references to the terms "one embodiment", "some embodiments", "illustrative embodiment", "example", "specific example", or "some examples", etc., mean that the specific features, structures, materials, or characteristics described in connection with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiments or examples. Moreover, the specific features, structures, materials or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0134] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents. Specification 14 / 14 pages 18 CN 120962662 A Figure 1 Specification Drawing 1 / 1 page 19 CN 120962662 A Abstract The present invention relates to the field of industrial robotics andproposes an adaptive robotic gripper for humanoid robots based on visual feedback. The gripper comprises a visual feedback module, a gripper control module, and a robotic arm equipped with a gripper, wherein the robotic arm includes multiple joints. The visual feedback module is used to perform fine-grained object classification of environmental information via an improved YOLOv7 model, and to send the shape type and 3D pose of the object to be grasped to the gripper control module. The gripper control module is used to receive the shape type and 3D pose of the object to be grasped sent by the visual feedback module, to control the robotic arm to move using a low-space-occupancy configuration strategy, and to generate a collision-free trajectory for the gripper from its initial position to the object to be grasped. When the gripper approaches the target object, it is controlled to move within a specified speed threshold range; when the gripper makes contact with the target object, thegripper’s gripping force is constrained based on the gripper’s gripping force, the object’s shape type, and its 3D pose.

Claims

1. A humanoid robot adaptive gripper grasping device based on visual feedback, characterized in that, The gripping device includes a visual feedback module, a gripper control module, and a robotic arm equipped with grippers. The robotic arm includes multiple joints. The visual feedback module is used to perform fine-grained object classification of environmental information using an improved YOLOv7 model. After identifying the shape type and three-dimensional pose of each object, it selects the object to be grasped and sends the shape type and three-dimensional pose of the object to be grasped to the gripper control module. The improvement includes adding a fine-grained classifier to the YOLOv7 model for classifying cylinders, cuboids and irregular objects. The gripper control module receives the shape and three-dimensional pose of the object to be grasped from the visual feedback module. When space is limited, it controls the robotic arm to move using a low-space-occupancy configuration strategy based on an improved random sampling path planning algorithm, generating a collision-free trajectory for the gripper from its initial position to the object to be grasped. The low-space-occupancy configuration strategy includes the coordinated bending of joints to form a spatial compression posture and the real-time calculation of the minimum swing parameters of each joint motion sequence. The swing parameters include joint angle constraints and motion direction priority. When the gripper approaches the object to be grasped, it controls the gripper to move within a preset speed threshold range. When the gripper contacts the object to be grasped, the gripping force of the gripper is constrained based on the gripping force of the gripper, the shape and type of the object, and the three-dimensional pose. Specifically, when generating a collision-free trajectory for the object to be grasped, the minimum Euclidean distance between the robotic arm and the obstacle is calculated in real time to adjust the gripper's movement trajectory in advance, thereby controlling the gripper to perform segmented grasping actions.

2. The humanoid robot adaptive gripper grasping device according to claim 1, characterized in that, The visual feedback module has a built-in camera. The process of performing fine-grained object classification of environmental information using an improved YOLOv7 model, identifying the shape, type, and 3D pose of each object, and then selecting the object to be grasped includes: Load the YOLOv7 model and fine-grained classifier, and generate a depth map of the object after setting the camera parameters to obtain the depth information of the target object; Target detection is performed on the depth map to obtain the bounding box coordinates, object type and confidence information of all objects in the depth map. Based on the bounding box coordinates, object type and confidence information of the objects, the object to be captured is selected. The depth information of the region of the object to be captured is extracted from the depth map according to the bounding box coordinates of the object to be captured, and the average depth is calculated. Based on the pixel coordinates and depth information of the center point of the object to be grasped, the pixel coordinates are converted into three-dimensional coordinates in the world coordinate system, and the object area to be grasped is classified in fine-grained shape to determine the shape type of the target object. Based on the shape type of the object to be grasped, a size estimation algorithm is invoked to estimate the size information of the object. The position, shape type, and size information of the target object are integrated to generate the first dataset required for grasping.

3. The humanoid robot adaptive gripper grasping device according to claim 1, characterized in that, Before controlling the gripper to perform the grasping action, the process includes constructing a hand-eye transformation matrix to map the 3D pose of the object to be grasped to the robot's base coordinate system: Obtain the pose list of the gripper and the pose list of the calibration board. Iterate through each pair of adjacent poses, invert the gripper pose matrix of the previous robot and multiply it with the gripper pose matrix of the current robot to obtain the robot pose transformation matrix. Multiply the current calibration board pose matrix with the inverse of the previous calibration board pose matrix to obtain the calibration board pose transformation matrix. Extract the first rotation matrix and the first translation vector from all robot pose transformation matrices. Extract the second rotation matrix and the second translation vector from all calibration board pose transformation matrices. Derive the hand-eye rotation matrix based on the first rotation matrix and the second rotation matrix. Derive the hand-eye translation vector based on the first rotation matrix, the hand-eye rotation matrix, the first translation vector, and the second translation vector. Construct a homogeneous transformation matrix by filling the hand-eye translation vector and the hand-eye rotation matrix into the homogeneous transformation matrix to obtain the hand-eye transformation matrix.

4. The humanoid robot adaptive gripper grasping device according to claim 1, characterized in that, Before the grippers contact the object to be gripped, the following steps are taken: Initialize the gripper dynamics model parameters, gripper physical parameters, and MPC parameters. The gripper dynamics model parameters include the state transition matrix, control matrix, output matrix, and direct transfer matrix. The gripper physical parameters include the gripper opening width and gripping force threshold. The MPC parameters include prediction time domain, control time domain, state error weight, and control input weight. Adjust the state error weight, the initial gripping force of the gripper, and the gripping force threshold according to the fragility of the object to be grasped; The first prediction matrix is ​​generated by exponentiation of the state transition matrix, the second prediction matrix is ​​generated by multiplication of the state transition matrix by exponentiation and the control matrix, and the objective function is constructed based on the first and second prediction matrices. Under the constraints of control time domain and clamping force threshold, the objective function defines the optimization problem by a quadratic combination of state error weights and control input weights, and the clamping force value of the gripper is obtained by convex optimization.

5. The humanoid robot adaptive gripper grasping device according to claim 1, characterized in that, When the gripper contacts the object to be grasped, the gripping force of the gripper is constrained based on the gripper's gripping force, the object's shape and type, and its 3D pose. Select the appropriate gripping strategy based on the shape and type of the object to be gripped, calculate the initial opening and closing width of the gripper and the target gripping force, acquire contact deformation data through force sensors during the pre-gripping stage, and evaluate the softness of the object material based on the strain change rate. The stiffness coefficient and maximum allowable gripping force threshold of the gripper are dynamically adjusted based on the softness assessment results. A model predictive controller is used to generate gripper motion commands that satisfy the force-position constraints, and a progressive gripping operation is performed until the gripper reaches a stable grasping state. Specifically, when the object to be grasped is a cylinder, the initial opening and closing width of the gripper is calculated based on the diameter measurement, and the contact point distribution is determined according to the cylinder height; when the object to be grasped is a cuboid, the initial opening and closing width of the gripper is calculated based on the minimum edge distance principle, and the contact area is determined according to the center point of the surface; when the object to be grasped is an irregular object, the maximum stable gripping width is extracted based on the convex hull algorithm, and the centroid-aligned gripping point is calculated.

6. The humanoid robot adaptive gripper grasping device according to claim 1, characterized in that, Controlling the robotic arm to move using a low-space-occupancy configuration strategy includes: The spatial constraint degree is calculated based on the three-dimensional Euclidean distance between the object to be grasped and the obstacle. When the spatial constraint is greater than the preset threshold, the optimal direction vector of the gripper approaching the object to be grasped is first calculated. Based on the optimal direction vector, the initial pose of the robotic arm with continuous S-shaped bending characteristics is generated by inverse kinematics solution. The joint angles of the robotic arm are iteratively adjusted until the robotic arm reaches the minimum space occupation state. The multi-segment motion trajectory of the gripper is generated according to the adjusted joint angles and the preset gripper movement speed. When the spatial constraint is less than a preset threshold, a grasping trajectory is generated based on the shape type and three-dimensional pose of the object to be grasped.

7. The humanoid robot adaptive gripper grasping device according to claim 1, characterized in that, The process of controlling the gripper to perform the grasping action also includes: The initial joint angle configuration of the robotic arm is calculated based on the target position of the gripper and inverse kinematics. The system obtains the space volume occupied by the robotic arm of the humanoid robot in the current configuration, calculates the position error between the actual position and the target position of the gripper, calculates the minimum distance between the robotic arm and the obstacle, and obtains a comprehensive score based on the space volume occupied by the robotic arm, the position error and the minimum distance. The comprehensive score is used to measure the quality of the current joint angle configuration. The initial joint angle configuration of the robotic arm is used as the starting point for optimization. The comprehensive score is minimized by the L-BFGS-B algorithm to continuously adjust the joint angles of the robotic arm and obtain the optimal joint angles.

8. The humanoid robot adaptive gripper grasping device according to claim 1, characterized in that, The clamping force of the grippers satisfies the following relationship: F target =μ×m×g×SF; Where μ represents the coefficient of friction, m represents the mass of the object, g represents the gravitational acceleration, and SF represents the safety factor.

9. The humanoid robot adaptive gripper grasping device according to claim 1, characterized in that, It also includes a force sensor, which, when adjusting the clamping force of the gripper, utilizes a PI controller to adjust the clamping force in real time based on feedback from the force sensor, satisfying the following relationship: F adjust =F target +K p e+K i ×∫edt; Among them, F adjust F represents the adjusted clamping force of the grippers. target K represents the target clamping force. p K represents the proportional gain coefficient, e represents the error between the force sensor and the target clamping force, and K represents the proportional gain coefficient. i ∫edt represents the integral gain coefficient, and ∫edt represents the integral of the error with respect to time.

Images