An adaptive gripping device for humanoid robots based on visual feedback

Abstract

The present application relates to the technical field of industrial robots, and proposes a humanoid robot adaptive gripper grabbing device based on visual feedback, the grabbing device comprising a visual feedback module, a gripper control module and a mechanical arm provided with the gripper, the mechanical arm comprising a plurality of joints; the visual feedback module is used for performing fine-grained object classification on environmental information through an improved YOLOv7 model, and sending the shape type and three-dimensional pose of the object to be grabbed to the gripper control module; the gripper control module is used for receiving the shape type and three-dimensional pose of the object to be grabbed sent by the visual feedback module, controlling the mechanical arm to move in a low-space-occupancy configuration strategy, and generating a collision-free trajectory of the gripper from an initial position to the object to be grabbed; when the gripper approaches the object to be grabbed, the gripper is controlled to move within a speed threshold range, and when the gripper contacts the object to be grabbed, the gripping force of the gripper is constrained based on the gripping force of the gripper, the shape type and three-dimensional pose of the object.

Description

Technical Field

[0001] This invention relates to the field of industrial robot technology, and in particular to an adaptive gripper for humanoid robots based on visual feedback. Background Technology

[0002] In the field of industrial automation, the application of robotics technology is becoming increasingly widespread, especially in high-precision grasping tasks, where its efficiency and accuracy are crucial to the production process.

[0003] Currently, industrial robots generally adopt grasping mechanisms based on fixed programs or simple sensor feedback. This usually relies on precise programming and predetermined grasping paths, and achieves basic object recognition and grasping through monocular vision systems or basic force sensors. For example, some robots use traditional electric door-pushing mechanisms, operated by doormen in fixed positions, which lack flexibility and intelligence.

[0004] Existing robotic grasping systems perform poorly when handling complex or irregularly shaped objects because they often require customized grippers and grasping strategies for specific objects. Furthermore, these systems struggle to adapt and optimize grasping paths when facing multiple objects or confined spaces, which limits the robot's potential for application in variable environments, especially in scenarios requiring high precision and adaptability, such as automated warehouse management, precision assembly, or operations in special environments. Summary of the Invention

[0005] To address the aforementioned shortcomings, the present invention aims to propose a humanoid robot adaptive gripper grasping device based on visual feedback, which addresses the deficiencies of insufficient adaptability when handling complex-shaped objects and low grasping accuracy and efficiency in dynamic environments.

[0006] To achieve this objective, the present invention adopts the following technical solution:

[0007] A humanoid robot adaptive gripper grasping device based on visual feedback, the gripping device includes a visual feedback module, a gripper control module, and a robotic arm equipped with grippers, the robotic arm including multiple joints;

[0008] The visual feedback module is used to perform fine-grained object classification of environmental information using an improved YOLOv7 model. After identifying the shape type and three-dimensional pose of each object, it selects the object to be grasped and sends the shape type and three-dimensional pose of the object to be grasped to the gripper control module. The improvement includes adding a fine-grained classifier to the YOLOv7 model for classifying cylinders, cuboids and irregular objects.

[0009] The gripper control module receives the shape and three-dimensional pose of the object to be grasped from the visual feedback module. When space is limited, it controls the robotic arm to move using a low-space-occupancy configuration strategy based on an improved random sampling path planning algorithm, generating a collision-free trajectory for the gripper from its initial position to the object to be grasped. The low-space-occupancy configuration strategy includes the coordinated bending of joints to form a spatial compression posture, and real-time calculation of the minimum swing parameters of each joint motion sequence. The swing parameters include joint angle constraints and motion direction priorities. When the gripper approaches the object to be grasped, it controls the gripper to move within a preset speed threshold range. When the gripper contacts the object to be grasped, it constrains the gripping force of the gripper based on the gripping force of the gripper, the shape and three-dimensional pose of the object.

[0010] Specifically, when generating a collision-free trajectory for the object to be grasped, the minimum Euclidean distance between the robotic arm and the obstacle is calculated in real time to adjust the gripper's movement trajectory in advance, thereby controlling the gripper to perform segmented grasping actions.

[0011] Preferably, the visual feedback module has a built-in camera, and the step of performing fine-grained object classification of environmental information using an improved YOLOv7 model, identifying the shape type and 3D pose of each object, and then selecting the object to be grasped includes:

[0012] Load the YOLOv7 model and fine-grained classifier, and generate a depth map of the object after setting the camera parameters to obtain the depth information of the target object;

[0013] Target detection is performed on the depth map to obtain the bounding box coordinates, object type and confidence information of all objects in the depth map. Based on the bounding box coordinates, object type and confidence information of the objects, the object to be captured is selected. The depth information of the region of the object to be captured is extracted from the depth map according to the bounding box coordinates of the object to be captured, and the average depth is calculated.

[0014] Based on the pixel coordinates and depth information of the center point of the object to be grasped, the pixel coordinates are converted into three-dimensional coordinates in the world coordinate system, and the object area to be grasped is classified in fine-grained shape to determine the shape type of the target object.

[0015] Based on the shape type of the object to be grasped, a size estimation algorithm is invoked to estimate the size information of the object. The position, shape type, and size information of the target object are integrated to generate the first dataset required for grasping.

[0016] Preferably, before controlling the gripper to perform the grasping action, a hand-eye transformation matrix is ​​constructed to map the three-dimensional pose of the object to be grasped to the robot's base coordinate system:

[0017] Obtain the pose list of the gripper and the pose list of the calibration board. Iterate through each pair of adjacent poses, invert the gripper pose matrix of the previous robot and multiply it with the gripper pose matrix of the current robot to obtain the robot pose transformation matrix. Multiply the current calibration board pose matrix with the inverse of the previous calibration board pose matrix to obtain the calibration board pose transformation matrix.

[0018] Extract the first rotation matrix and the first translation vector from all robot pose transformation matrices. Extract the second rotation matrix and the second translation vector from all calibration board pose transformation matrices. Derive the hand-eye rotation matrix based on the first rotation matrix and the second rotation matrix. Derive the hand-eye translation vector based on the first rotation matrix, the hand-eye rotation matrix, the first translation vector, and the second translation vector.

[0019] Construct a homogeneous transformation matrix by filling the hand-eye translation vector and the hand-eye rotation matrix into the homogeneous transformation matrix to obtain the hand-eye transformation matrix.

[0020] Preferably, the process includes the following steps before the grippers contact the object to be grasped:

[0021] Initialize the gripper dynamics model parameters, gripper physical parameters, and MPC parameters. The gripper dynamics model parameters include the state transition matrix, control matrix, output matrix, and direct transfer matrix. The gripper physical parameters include the gripper opening width and gripping force threshold. The MPC parameters include prediction time domain, control time domain, state error weight, and control input weight.

[0022] Adjust the state error weight, the initial gripping force of the gripper, and the gripping force threshold according to the fragility of the object to be grasped;

[0023] The first prediction matrix is ​​generated by exponentiation of the state transition matrix, the second prediction matrix is ​​generated by multiplication of the state transition matrix by exponentiation and the control matrix, and the objective function is constructed based on the first and second prediction matrices.

[0024] Under the constraints of control time domain and clamping force threshold, the objective function defines the optimization problem by a quadratic combination of state error weights and control input weights, and the clamping force value of the gripper is obtained by convex optimization.

[0025] Preferably, constraining the gripping force of the gripper based on the gripping force of the gripper, the shape and type of the object, and the three-dimensional pose when the gripper contacts the object to be grasped includes:

[0026] Select the appropriate gripping strategy based on the shape and type of the object to be gripped, calculate the initial opening and closing width of the gripper and the target gripping force, acquire contact deformation data through force sensors during the pre-gripping stage, and evaluate the softness of the object material based on the strain change rate.

[0027] The stiffness coefficient and maximum allowable gripping force threshold of the gripper are dynamically adjusted based on the softness assessment results. A model predictive controller is used to generate gripper motion commands that satisfy the force-position constraints, and a progressive gripping operation is performed until the gripper reaches a stable grasping state.

[0028] Specifically, when the object to be grasped is a cylinder, the initial opening and closing width of the gripper is calculated based on the diameter measurement, and the contact point distribution is determined according to the cylinder height; when the object to be grasped is a cuboid, the initial opening and closing width of the gripper is calculated based on the minimum edge distance principle, and the contact area is determined according to the center point of the surface; when the object to be grasped is an irregular object, the maximum stable gripping width is extracted based on the convex hull algorithm, and the centroid-aligned gripping point is calculated.

[0029] Preferably, controlling the robotic arm to move using a low-space-occupancy configuration strategy includes:

[0030] The spatial constraint degree is calculated based on the three-dimensional Euclidean distance between the object to be grasped and the obstacle.

[0031] When the spatial constraint is greater than the preset threshold, the optimal direction vector of the gripper approaching the object to be grasped is first calculated. Based on the optimal direction vector, the initial pose of the robotic arm with continuous S-shaped bending characteristics is generated by inverse kinematics solution. The joint angles of the robotic arm are iteratively adjusted until the robotic arm reaches the minimum space occupation state. The multi-segment motion trajectory of the gripper is generated according to the adjusted joint angles and the preset gripper movement speed.

[0032] When the spatial constraint is less than a preset threshold, a grasping trajectory is generated based on the shape type and three-dimensional pose of the object to be grasped.

[0033] Preferably, when controlling the gripper to perform the grasping action, the method further includes:

[0034] The initial joint angle configuration of the robotic arm is calculated based on the target position of the gripper and inverse kinematics.

[0035] The system obtains the space volume occupied by the robotic arm of the humanoid robot in the current configuration, calculates the position error between the actual position and the target position of the gripper, calculates the minimum distance between the robotic arm and the obstacle, and obtains a comprehensive score based on the space volume occupied by the robotic arm, the position error and the minimum distance. The comprehensive score is used to measure the quality of the current joint angle configuration.

[0036] The initial joint angle configuration of the robotic arm is used as the starting point for optimization. The comprehensive score is minimized by the L-BFGS-B algorithm to continuously adjust the joint angles of the robotic arm and obtain the optimal joint angles.

[0037] Preferably, the clamping force of the grippers satisfies the following relationship:

[0038] F target =μ×m×g×SF;

[0039] Where μ represents the coefficient of friction, m represents the mass of the object, g represents the gravitational acceleration, and SF represents the safety factor.

[0040] Preferably, a force sensor is also provided, and when adjusting the clamping force of the gripper, a PI controller is used to adjust the clamping force in real time based on the feedback from the force sensor, satisfying the following relationship:

[0041] F adjust =F target +K p e+K i ×∫ed t;

[0042] Among them, F adjust F represents the adjusted clamping force of the grippers. target K represents the target clamping force. p K represents the proportional gain coefficient, e represents the error between the force sensor and the target clamping force, and K represents the proportional gain coefficient. i Let ∫e dt represent the integral gain coefficient, and let ∫e dt represent the integral of the error with respect to time.

[0043] One of the above technical solutions has the following advantages or beneficial effects:

[0044] This invention integrates visual feedback and adaptive path planning technologies to effectively address the shortcomings of traditional grasping devices in recognizing complex-shaped objects and adapting to dynamic environments. First, the improved YOLOv7 model achieves accurate classification of cylinders, cuboids, and irregular objects by adding a fine-grained classifier. Combined with the 3D pose recognition of the visual feedback module, it can significantly improve the recognition accuracy of the shape type and 3D pose of the object to be grasped, providing reliable data support for subsequent grasping strategies. Second, the gripper control module adopts a low-space-occupancy configuration strategy, forming a spatial compression posture through joint coordinated bending, and calculating the minimum swing parameters of joint movement in real time. This generates a collision-free trajectory in narrow or obstacle-filled scenarios, enhancing the robot arm's mobility in confined environments. Furthermore, the dynamic constraint mechanism of gripping force based on object shape type and 3D pose, combined with segmented grasping actions and real-time trajectory adjustment, can avoid slippage caused by insufficient gripping force and prevent damage to fragile objects caused by overload. Through the synergistic effect of visual perception, path optimization, and mechanical control, this invention achieves stable grasping of objects of different shapes in complex scenarios, while also balancing grasping efficiency and safety, effectively improving the humanoid robot's ability to operate in unstructured environments. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0046] Figure 1 This is a schematic diagram of the structure of the humanoid robot adaptive gripper grasping device based on visual feedback provided in an embodiment of the present invention. Detailed Implementation

[0047] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0048] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0049] In the field of industrial automation, the application of robotics technology is becoming increasingly widespread, especially in high-precision grasping tasks, where its efficiency and accuracy are crucial to the production process.

[0050] Currently, industrial robots generally adopt grasping mechanisms based on fixed programs or simple sensor feedback. This usually relies on precise programming and predetermined grasping paths, and achieves basic object recognition and grasping through monocular vision systems or basic force sensors. For example, some robots use traditional electric door-pushing mechanisms, operated by doormen in fixed positions, which lack flexibility and intelligence.

[0051] Existing robotic grasping systems perform poorly when handling complex or irregularly shaped objects because they often require customized grippers and grasping strategies for specific objects. Furthermore, these systems struggle to adapt and optimize grasping paths when facing multiple objects or confined spaces, which limits the robot's potential for application in variable environments, especially in scenarios requiring high precision and adaptability, such as automated warehouse management, precision assembly, or operations in special environments.

[0052] A humanoid robot adaptive gripper grasping device based on visual feedback, such as Figure 1 As shown, in a preferred embodiment of the present invention, the gripping device includes a visual feedback module, a gripper control module, and a robotic arm equipped with grippers, the robotic arm including multiple joints;

[0053] The visual feedback module is used to perform fine-grained object classification of environmental information using an improved YOLOv7 model. After identifying the shape type and three-dimensional pose of each object, it selects the object to be grasped and sends the shape type and three-dimensional pose of the object to be grasped to the gripper control module. The improvement includes adding a fine-grained classifier to the YOLOv7 model for classifying cylinders, cuboids and irregular objects.

[0054] The gripper control module receives the shape and three-dimensional pose of the object to be grasped from the visual feedback module. When space is limited, it controls the robotic arm to move using a low-space-occupancy configuration strategy based on an improved random sampling path planning algorithm, generating a collision-free trajectory for the gripper from its initial position to the object to be grasped. The low-space-occupancy configuration strategy includes the coordinated bending of joints to form a spatial compression posture, and real-time calculation of the minimum swing parameters of each joint motion sequence. The swing parameters include joint angle constraints and motion direction priorities. When the gripper approaches the object to be grasped, it controls the gripper to move within a preset speed threshold range. When the gripper contacts the object to be grasped, it constrains the gripping force of the gripper based on the gripping force of the gripper, the shape and three-dimensional pose of the object.

[0055] Specifically, when generating a collision-free trajectory for the object to be grasped, the minimum Euclidean distance between the robotic arm and the obstacle is calculated in real time to adjust the gripper's movement trajectory in advance, thereby controlling the gripper to perform segmented grasping actions.

[0056] In this embodiment, the visual feedback module is based on an Isaac Sim optimized binocular camera system, integrating an improved YOLOv7 object detection algorithm to detect and estimate the pose of target objects. Specifically, the visual feedback module acquires image information of the target object through the binocular camera and uses the improved YOLOv7 algorithm for object detection and pose estimation. This algorithm adds a fine-grained classification branch to the traditional YOLOv7, enabling it to recognize objects of different shapes, such as cylinders, cuboids, and irregular objects. In the specific implementation process, the binocular camera system first acquires image data of the target object with an input resolution of 640×480. The improved YOLOv7 algorithm processes the image, identifies the target object, and estimates its pose, improving the recognition accuracy for complex-shaped objects and optimizing the inference speed to 25 FPS. Assuming the target object is a cuboid, the visual feedback system acquires its image through a binocular camera. The improved YOLOv7 algorithm identifies the bounding box of the cuboid and obtains its pose information, including position and orientation, by solving the PnP problem (based on cv2.solvePnP).

[0057] In this embodiment, the adaptive gripper employs a parallelogram hinge structure with two fingers and a built-in strain gauge force sensor. It uses a force-position hybrid control based on visual feedback to grasp the target object. An improvement upon the traditional random sampling path planning algorithm generates a collision-free path based on obstacle information in the environment and the robotic arm's kinematic model. This algorithm constructs a search tree by randomly sampling in the configuration space and uses a heuristic function to guide the search direction, finding a feasible path from the starting point to the target point. A low-space-occupancy configuration strategy uses coordinated joint bending to create a spatial compression posture, reducing the robotic arm's space occupation during movement. Multiple joints bend according to a certain cooperative relationship to form a specific posture, achieving the effect of space compression. After receiving the shape type and three-dimensional pose information of the object to be grasped from the visual feedback module, the gripper control module first determines whether the current space is restricted (e.g., by detecting whether the distance between surrounding obstacles and the robotic arm is less than a preset threshold). If space is limited, the gripper control module invokes an improved random sampling path planning algorithm to generate a collision-free trajectory from the gripper's initial position to the object to be grasped, based on the robotic arm's kinematic model and obstacle information in the environment. During trajectory generation, the robotic arm employs a low-space-occupancy configuration strategy, using coordinated joint bending to create a spatial compression posture, reducing the space occupied by the robotic arm during movement. The minimum swing parameters of each joint's motion sequence are calculated in real-time, including joint angle constraints and motion direction priorities, to ensure the robotic arm does not collide with obstacles during movement. When the gripper approaches the object to be grasped, it moves within a preset speed threshold range, for example, limiting the speed to below 0.03 m / s to improve grasping accuracy. When the gripper contacts the object, the gripping force is constrained based on the gripper's clamping force, the object's shape, and its 3D pose. For example, reduce the clamping force for fragile objects; increase the clamping force for smooth objects to prevent slippage; and set appropriate force correction factors for objects that are not kicked: use standard gripping force for rigid objects; use a correction factor of 0.6-0.8 for deformable objects to prevent excessive deformation; use a lower correction factor of 0.3-0.5 for fragile objects to prevent damage; and use a higher correction factor for high-density objects and low-friction surfaces.The extension and retraction speed of the robotic arm and the gripping force of the gripper can be adjusted at different times during different grasping phases. Approach phase (t0-t1): The duration is calculated based on distance and safe acceleration, the maximum approach speed is set, and the speed limit is reduced for fragile objects. Initial contact phase (t1-t2): Lasts 100-200ms, using extremely low contact speed, with a contact force threshold set at 0.5-1.0N. Force growth phase (t2-t3): The force growth slope is calculated based on the target force and the basic growth time, setting different maximum force growth rates for different types of objects, and dynamically adjusting the growth rate through strain feedback. Stable grasping phase (t3-t4): The duration is determined according to task requirements, the force value is maintained within ±5% of the target force, and micro-vibrations are applied to improve grasping stability. Release phase (t4-t5): Generally, the force descent rate is set to 1.5-2 times the force growth rate, with a gentler release curve used for special objects.

[0058] In the specific implementation process, a vision system is used to process images acquired by the left and right cameras to obtain the position information of the glass bottle. This position information is extracted, and based on known conditions, the bottle's dimensions (radius and height), shape type, material, surface characteristics, and estimated mass are set to create a dictionary containing various information about the glass bottle for easy retrieval in subsequent steps. 40% of the bottle's height is chosen as the gripping height, bringing the gripping point close to the bottle's center of gravity to improve gripping stability. Based on the bottle's original position, the height is adjusted along the Z-axis to obtain the specific gripping position. By calculating the approach vector perpendicular to the bottle's axis, the gripper can accurately and stably contact the bottle. The direction vector that the gripper should follow when approaching the bottle is obtained. Based on the estimated mass of the glass bottle and gravitational acceleration, the base force is calculated and multiplied by a safety factor of 2.5 to obtain the maximum gripping force. The holding force is set to 95% of the maximum gripping force to maintain a high and continuous gripping force, and a force curve configuration file containing gripping force-related information is generated. After the inspection is completed, the texture of the gripper fingertip is switched to a high-friction type to increase the friction with the glass bottle surface. The robotic arm is then controlled to move to the calculated gripping position and move in the set approach direction. The robot controls the gripper to perform the gripping action and apply the corresponding clamping force according to the generated force curve configuration file.

[0059] In addition, in one embodiment of the present invention, a corresponding adaptive adjustment mechanism is also provided under vibration conditions: 1. Vibration detection and analysis: A vibration detection algorithm is adopted to analyze vibration characteristics through IMU sensor and force sensor data: Root mean square calculation and FFT spectrum analysis are performed on acceleration data to extract the main vibration frequency and amplitude characteristics, and the vibration level is classified into three levels: low, medium, and high according to the vibration characteristics; 2. Dynamic adjustment of gripping strategy: Force adjustment compensator: The clamping force is dynamically adjusted based on the vibration frequency and amplitude, and different compensation coefficients are used for different frequency vibrations: a larger frequency compensation factor is used for low frequency vibration (<8Hz), and a larger compensation factor is used for high frequency vibration. Using different adjustment strategies, a safety factor is calculated by considering multiple factors, with a maximum limit of 2.5. The impedance control parameters are adaptively adjusted, and the stiffness and damping parameters are dynamically adjusted according to the vibration level: low vibration: increase stiffness and slightly increase damping; medium vibration: decrease stiffness and significantly increase damping; high vibration: significantly decrease stiffness and significantly increase damping. 3. Vision-force coordinated anti-interference control - The system realizes a multi-modal information fusion and prediction compensation mechanism: using a Kalman filter to fuse visual and force sensing data, predicting the future position of the target object based on the vibration mode, adjusting the robotic arm control parameters according to the vibration characteristics, and then calculating and applying the compensated grasping force in real time.

[0060] Preferably, the visual feedback module has a built-in camera, and the step of performing fine-grained object classification of environmental information using an improved YOLOv7 model, identifying the shape type and 3D pose of each object, and then selecting the object to be grasped includes:

[0061] Load the YOLOv7 model and fine-grained classifier, and generate a depth map of the object after setting the camera parameters to obtain the depth information of the target object;

[0062] The YOLOv7 model is an advanced object detection algorithm characterized by real-time performance and high accuracy. It uses a deep learning network to detect and locate objects in images, outputting information such as bounding box coordinates, category, and confidence score. A fine-grained classifier is an additional classification branch added to the YOLOv7 model for more detailed object classification, such as distinguishing between objects of different shapes (cylinders, cuboids, irregular objects, etc.). Camera parameter settings include intrinsic parameters (such as focal length and principal point coordinates) and extrinsic parameters (such as the camera's position and pose in the world coordinate system), used to convert points in the image coordinate system to points in the world coordinate system. Depth map generation involves capturing two images from a stereo camera and calculating the depth information of each pixel using the parallax principle to generate a depth map. Each pixel value in the depth map represents the distance of that point from the camera.

[0063] In logistics sorting scenarios, packages of various shapes are placed on conveyor belts, requiring robotic arms to grasp and sort them. The visual feedback module first loads the YOLOv7 model and fine-grained classifier, and then sets the parameters of the binocular camera to accurately capture images of the packages on the conveyor belt. After the binocular camera acquires images of the packages, it generates depth maps to obtain the depth information of each package. For example, the depth range of package A is [100, 150] mm, and the depth range of package B is [150, 200] mm, etc., laying the foundation for subsequent object recognition and pose estimation.

[0064] Target detection is performed on the depth map to obtain the bounding box coordinates, object type and confidence information of all objects in the depth map. Based on the bounding box coordinates, object type and confidence information of the objects, the object to be captured is selected. The depth information of the region of the object to be captured is extracted from the depth map according to the bounding box coordinates of the object to be captured, and the average depth is calculated.

[0065] The average depth calculation involves extracting depth information of the area of ​​the object to be captured from the depth map, calculating the average depth value of all pixels in that area, and using this average depth as the object's average depth to estimate the distance between the object and the camera.

[0066] The generated depth map is input into the YOLOv7 model for object detection. The bounding box coordinates, object type, and confidence score of all objects within the depth map are obtained. Based on preset selection rules (such as selecting the object with the highest confidence score or selecting a specific object type), the object to be captured is selected from the detection results. Using the bounding box coordinates of the object to be captured, the depth information of the object's region is extracted from the depth map. The average value of the extracted depth information is calculated to obtain the average depth of the object to be captured, providing data for subsequent 3D pose calculations.

[0067] For example, in the above logistics sorting scenario, the YOLOv7 model performs object detection on the depth map and outputs the bounding box coordinates, type (such as cylindrical cans, cuboid boxes, etc.) and confidence information of all packages on the conveyor belt. The objects to be grasped are selected in descending order of confidence. Assuming that the cuboid box with the highest confidence is selected as the target, the depth information of its region is extracted from the depth map based on the bounding box coordinates of the box. The average depth of the box is calculated to be 180mm, which means that the average distance of the box from the camera is 180mm, providing a distance reference for subsequent grasping actions.

[0068] Based on the pixel coordinates and depth information of the center point of the object to be grasped, the pixel coordinates are converted into three-dimensional coordinates in the world coordinate system, and the object area to be grasped is classified in fine-grained shape to determine the shape type of the target object.

[0069] The conversion from pixel coordinates to world coordinates utilizes the camera's intrinsic and extrinsic parameters to convert the pixel coordinates and depth information of objects in an image into three-dimensional coordinates in the world coordinate system. Through coordinate transformation, the position and orientation of the object in actual space can be determined.

[0070] During implementation, the pixel coordinates (e.g., coordinates (u,v)) and depth information (e.g., depth value d) of the center point of the object to be grasped are first obtained. Using the camera's intrinsic and extrinsic parameter matrices, the pixel coordinates (u,v) and depth value d are converted into three-dimensional coordinates (X,Y,Z) in the world coordinate system. Shape analysis is then performed on the region of the object to be grasped to extract the object's shape features (e.g., edge contours, symmetry, etc.). The extracted shape features are then input into a fine-grained classifier to determine the shape type of the target object.

[0071] In a logistics sorting scenario, the pixel coordinates of the center point of the box to be grasped are (320, 240), and the depth information is 180mm. Using the camera's intrinsic parameter matrix (including parameters such as focal length and principal point coordinates) and extrinsic parameter matrix (including the camera's position and orientation in the world coordinate system), these pixel coordinates and depth values ​​are converted into three-dimensional coordinates in the world coordinate system, assuming the obtained three-dimensional coordinates are (100, 200, 180)mm. Then, shape analysis is performed on the box area, extracting its edge contours and other features, which are input into a fine-grained classifier to determine that the box's shape is a cuboid, providing a basis for subsequent grasping strategies.

[0072] Based on the shape type of the object to be grasped, a size estimation algorithm is invoked to estimate the size information of the object. The position, shape type, and size information of the target object are integrated to generate the first dataset required for grasping.

[0073] Size estimation algorithms estimate the actual size of an object based on its shape and features in an image. For example, for a cylinder, its diameter and height can be calculated by measuring its diameter and height in the image and combining this with depth information. Similarly, for a cuboid, its length, width, and height can be calculated by measuring its length, width, and height in the image and combining this with depth information. The first dataset is a dataset that integrates the target object's position (3D coordinates), shape, and size information to guide the gripper's grasping action.

[0074] In a logistics sorting scenario, after determining that the shape of the box to be grasped is a cuboid, its length, width, and height in the image are measured to be 100 pixels, 80 pixels, and 60 pixels, respectively. Combined with the depth information of 180mm and parameters such as the camera's focal length, the actual dimensions of the box are calculated to be 200mm long, 160mm wide, and 120mm high. The box's position (100, 200, 180)mm, shape type (cubic prism), and size information (200mm × 160mm × 120mm) are integrated into a first dataset, providing detailed data guidance for the robotic arm's grasping actions, ensuring that the gripper can accurately adapt to the box's size and achieve stable grasping.

[0075] Preferably, before controlling the gripper to perform the grasping action, a hand-eye transformation matrix is ​​constructed to map the three-dimensional pose of the object to be grasped to the robot's base coordinate system:

[0076] Obtain the pose list of the gripper and the pose list of the calibration board. Iterate through each pair of adjacent poses, invert the gripper pose matrix of the previous robot and multiply it with the gripper pose matrix of the current robot to obtain the robot pose transformation matrix. Multiply the current calibration board pose matrix with the inverse of the previous calibration board pose matrix to obtain the calibration board pose transformation matrix.

[0077] The system includes several key components: a gripper pose list (recording the gripper's pose information at different positions and orientations, including position coordinates and orientation angles), a calibration board pose list (recording the calibration board's pose information at different positions and orientations, used for hand-eye calibration), a robot pose transformation matrix (representing the robot's transformation from one pose to another, obtained by multiplying the inverse of the previous pose matrix by the current pose matrix), and a calibration board pose transformation matrix (representing the calibration board's transformation from one pose to another, obtained by multiplying the current calibration board pose matrix by the inverse of the previous calibration board pose matrix).

[0078] For example, in a laboratory object-grabbing environment, to calibrate the hand-eye relationship between the gripper of a robotic arm and the camera, the robotic arm is first moved to multiple different positions and orientations, and the gripper's pose information (such as position coordinates (x, y, z) and orientation angles (roll, pitch, yaw)) is recorded at each position, forming a gripper pose list. Simultaneously, a calibration board of known size is placed at each position, and its pose information (such as its position and orientation in the world coordinate system) is recorded, forming a calibration board pose list. Then, each pair of adjacent gripper and calibration board poses is traversed, for example, the first and second poses, to obtain the corresponding gripper pose matrices Trobot1 and Trobot2, and the robot pose transformation matrix Trobot_transform = Trobot2 × Trobot1-1 is calculated; similarly, the calibration board pose matrices Tboard1 and Tboard2 are obtained, and the calibration board pose transformation matrix Tboard_transform = Tboard2 × Tboard1-1 is calculated.

[0079] Extract the first rotation matrix and the first translation vector from all robot pose transformation matrices. Extract the second rotation matrix and the second translation vector from all calibration board pose transformation matrices. Derive the hand-eye rotation matrix based on the first rotation matrix and the second rotation matrix. Derive the hand-eye translation vector based on the first rotation matrix, the hand-eye rotation matrix, the first translation vector, and the second translation vector.

[0080] The first rotation matrix and the first translation vector are the rotation and translation components extracted from the robot pose transformation matrix, respectively, representing the rotation and translation changes of the robot between adjacent poses; the second rotation matrix and the second translation vector are the rotation and translation components extracted from the calibration board pose transformation matrix, respectively, representing the rotation and translation changes of the calibration board between adjacent poses; the hand-eye rotation matrix represents the rotational relationship of the camera relative to the gripper, obtained by comparing the rotational components of the robot pose transformation matrix and the calibration board pose transformation matrix; the hand-eye translation vector represents the translational relationship of the camera relative to the gripper, calculated by combining the first rotation matrix, the hand-eye rotation matrix, the first translation vector, and the second translation vector.

[0081] Construct a homogeneous transformation matrix by filling the hand-eye translation vector and the hand-eye rotation matrix into the homogeneous transformation matrix to obtain the hand-eye transformation matrix.

[0082] The hand-eye transformation matrix represents the transformation relationship between the camera coordinate system and the gripper coordinate system. It is used to transform the three-dimensional pose of the object to be grasped from the camera coordinate system to the robot base coordinate system, thereby guiding the grasping action of the robotic arm.

[0083] This embodiment details the process of constructing the hand-eye transformation matrix before controlling the gripper to perform the grasping action. By acquiring the pose lists of the gripper and calibration plate, the robot pose transformation matrix and the calibration plate pose transformation matrix are calculated. Then, the rotation matrix and translation vector are extracted, ultimately obtaining the hand-eye rotation matrix and translation vector, and constructing a homogeneous transformation matrix. This process ensures that the 3D pose information of the object acquired by the camera can be accurately mapped to the robot's base coordinate system, providing a crucial coordinate transformation foundation for the robotic arm's precise grasping. In practical applications, such as the precise grasping and assembly of parts on industrial production lines, the accuracy of the hand-eye transformation matrix directly affects the success rate and efficiency of the robotic arm's grasping. The method in this embodiment can effectively improve the accuracy of hand-eye calibration, thereby enhancing the robotic arm's operational capabilities and adaptability in complex environments.

[0084] Preferably, the process includes the following steps before the grippers contact the object to be grasped:

[0085] Initialize the gripper dynamics model parameters, gripper physical parameters, and MPC parameters. The gripper dynamics model parameters include the state transition matrix, control matrix, output matrix, and direct transfer matrix. The gripper physical parameters include the gripper opening width and gripping force threshold. The MPC parameters include prediction time domain, control time domain, state error weight, and control input weight.

[0086] The gripper dynamics model parameters describe the state changes of the gripper under different control inputs and are the basis for predicting the future state of the gripper in the MPC algorithm. The gripper physical parameters involve the range of the gripper's opening width (maximum and minimum opening width) and the gripping force threshold. These parameters determine the physical limits and safe range of the gripper in actual operation. The MPC parameters include the prediction time domain (the time range within which the algorithm predicts the future state), the control time domain (the time range within which the control input is effective in each optimization calculation), the state error weight (measuring the importance of the state deviation), and the control input weight (measuring the cost of changes in the control input). These parameters directly affect the optimization objective and control performance of the MPC algorithm.

[0087] For example, in a food processing workshop, a robotic arm needs to grab fragile eggs for packaging. Based on the characteristics of the eggs and the design of the gripper, the parameters of the gripper dynamic model are initialized: the state transition matrix A describes the rate of change of the opening width of the gripper under no external force, assumed to be 0.8 (meaning that the opening width is reduced to 80% of the original value per unit time); the control matrix B describes the effect of motor torque on the opening width, assumed to be 0.5 (meaning that the opening width changes by 0.5 mm per unit torque); the output matrix C directly outputs the opening width, which is 1; the direct transfer matrix D is 0. The physical parameters of the gripper are initialized to an opening width of 100 mm and a gripping force threshold of 5 N (exceeding 5 N may crush the eggs). The MPC parameters are initialized to a prediction time domain of 0.5 seconds, a control time domain of 0.2 seconds, a state error weight [20,10] (paying more attention to the error of the opening width), and a control input weight [2] (appropriately considering energy consumption).

[0088] Adjust the state error weight, the initial gripping force of the gripper, and the gripping force threshold according to the fragility of the object to be grasped;

[0089] Fragility indicates the object's sensitivity to clamping forces, and is usually classified according to the object's material and structural characteristics, such as fragile, relatively fragile, normal, and robust. State error weight adjustment adjusts the element values ​​in the state error weight matrix according to the object's fragility to change the emphasis on opening width error and clamping force error. Initial clamping force is the initial force applied by the grippers when they first contact the object, and is adjusted according to the object's fragility; fragile objects require a smaller initial clamping force. Clamping force threshold adjustment adjusts the maximum allowable value of the clamping force according to the object's fragility; fragile objects should have a lower clamping force threshold.

[0090] In a food processing workshop, eggs to be handled are considered fragile objects. Based on their fragility, the state error weight is adjusted from [20,10] to [10,20], placing greater emphasis on the precision of clamping force control; the initial clamping force is adjusted from 3N to 1N to reduce the initial impact on the egg; and the clamping force threshold is adjusted from 5N to 3N to ensure that the clamping force does not exceed the egg's tolerance range and to avoid crushing the egg.

[0091] The first prediction matrix is ​​generated by exponentiation of the state transition matrix, the second prediction matrix is ​​generated by multiplication of the state transition matrix by exponentiation and the control matrix, and the objective function is constructed based on the first and second prediction matrices.

[0092] The exponentiation of the state transition matrix is ​​performed by calculating the power of the state transition matrix A (e.g., A^k, where k is the time step) to predict the future evolution trend of the system state. The first prediction matrix is ​​generated by the exponentiation of the state transition matrix and represents the natural evolution of the system state in the prediction time domain. The second prediction matrix is ​​generated by multiplying the state transition matrix by the power of the state transition matrix and the control matrix B, representing the influence of the control input on the system state in the prediction time domain. In the MPC algorithm, the objective function is a function used to measure the control effect. It usually includes the weighted sum of the state error and the control input. The optimal control input is solved by minimizing the objective function.

[0093] Under the constraints of control time domain and clamping force threshold, the objective function defines the optimization problem by a quadratic combination of state error weights and control input weights, and the clamping force value of the gripper is obtained by convex optimization.

[0094] Within the prediction time domain, the actual time period for applying control input is defined. For example, if the prediction time domain is 0.5 seconds and the control time domain is 0.2 seconds, it means that control input is applied only within the first 0.2 seconds, after which the system evolves freely. Quadratic combination combines the state error and control input into a quadratic objective function, facilitating the solution of convex optimization problems. Convex optimization is a mathematical optimization method used to find the minimum value of a convex function, and it is often used in MPC algorithms to find the optimal control input.

[0095] In a food processing workshop, the control time domain is 0.2 seconds, the discrete time is 0.1 seconds, and the number of time steps for applying the control input is 2. A clamping force threshold constraint (maximum clamping force of 4N) is added to the objective function. The minimum value of the objective function J is solved using a convex optimization algorithm to obtain the optimal motor torque control input u, thereby obtaining the clamping force value of the gripper. Assuming the solved control input u is [0.8, 0.6] (in N·m), the corresponding clamping force value is 3N, satisfying the clamping force threshold constraint. This ensures that the clamping force will not exceed the egg's tolerance range when grasping it, achieving stable and safe grasping.

[0096] Preferably, constraining the gripping force of the gripper based on the gripping force of the gripper, the shape and type of the object, and the three-dimensional pose when the gripper contacts the object to be grasped includes:

[0097] Select the appropriate gripping strategy based on the shape and type of the object to be gripped, calculate the initial opening and closing width of the gripper and the target gripping force, acquire contact deformation data through force sensors during the pre-gripping stage, and evaluate the softness of the object material based on the strain change rate.

[0098] The gripping strategy selects different gripping methods and parameters based on the shape and type of the object to achieve the best gripping effect. For example, a cylinder may be gripped using a wraparound gripping method, a cuboid using a surface contact gripping method, and an irregular object using an adaptive gripping method. The initial opening width is the opening width of the gripper when it first contacts the object. It is calculated based on the size and shape of the object to ensure that the gripper can smoothly approach and contact the object. The target gripping force is the expected gripping force calculated based on the object's weight, coefficient of friction, and shape to achieve stable gripping. A force sensor is installed on the gripper to measure the gripping force in real time. It is usually based on the principle of strain gauges and calculates the force by detecting changes in strain. The contact deformation data is the deformation data of the object's surface when the gripper contacts the object. It is acquired by the force sensor and reflects the material properties and stress conditions of the object. The strain change rate is the rate of change of strain of the strain gauge in the force sensor over time. It is used to assess the softness of the object's material. Objects with high softness have a larger strain change rate when subjected to force, and vice versa.

[0099] In practice, the appropriate gripping strategy is selected based on the shape of the object to be gripped: a. Cylinder: surround gripping is used, with the grippers surrounding the object from both sides; b. Cuboid: surface contact gripping is used, with the grippers clamping parallel to one face of the object; c. Irregular object: adaptive gripping is used, with the grippers automatically adjusting the gripping angle and position according to the shape of the object; the initial opening and closing width of the grippers is calculated as follows: a. For cylinder, the initial opening and closing width is equal to the object's diameter plus a certain safety margin (e.g., 2mm); b. For cuboid, the initial opening and closing width is equal to the object's minimum side length plus a safety margin; c. For irregular object, the initial opening and closing width is calculated based on the object's convex hull size. Calculating the target clamping force: a. Based on the object's weight, coefficient of friction, and safety factor, calculate using the formula F = μ × m × g × k, where μ is the coefficient of friction, m is the object's mass, g is the acceleration due to gravity, and k is the safety factor (usually taken as 1.2–2.0). During the pre-clamping stage, the grippers approach the object at a low speed. When contact is made, the force sensor begins to acquire contact deformation data. Calculating the strain change rate: The strain change rate is obtained by calculating the time derivative of the strain data acquired by the force sensor. The softness of the object's material is assessed based on the strain change rate: a. A larger strain change rate indicates a softer material; b. A smaller strain change rate indicates a harder material.

[0100] The stiffness coefficient and maximum allowable gripping force threshold of the gripper are dynamically adjusted based on the softness assessment results. A model predictive controller is used to generate gripper motion commands that satisfy the force-position constraints, and a progressive gripping operation is performed until the gripper reaches a stable grasping state.

[0101] Specifically, based on the softness assessment results, the stiffness coefficient and maximum allowable clamping force threshold of the force controller are dynamically adjusted: a. For soft objects, the stiffness coefficient is reduced (e.g., from 100 N / m to 50 N / m), and the maximum allowable clamping force threshold is lowered (e.g., from 10 N to 3 N); b. For hard objects, the stiffness coefficient and maximum allowable clamping force threshold can be appropriately increased; Model predictive controllers are used to generate gripper motion commands: a. A force-position hybrid control model for the gripper is established, including the dynamic model of the gripper and the feedback model of the force sensor; b. In each control cycle, the opening and closing width and clamping force changes of the gripper over a future period are predicted; c. Based on the prediction results, the current control input (e.g., motor torque) is optimized to ensure that the opening and closing width and clamping force of the gripper meet the force-position constraints and are as close as possible to the target values. Perform progressive gripping operation: a. In the initial stage, slowly close the gripper with a small gripping force, and monitor the gripping force and the deformation of the object in real time; b. Based on the monitoring data, gradually increase the gripping force until a stable gripping state is reached, that is, the gripping force reaches the target value and the object does not slip significantly or deform excessively.

[0102] Specifically, when the object to be grasped is a cylinder, the initial opening and closing width of the gripper is calculated based on the diameter measurement, and the contact point distribution is determined according to the cylinder height; when the object to be grasped is a cuboid, the initial opening and closing width of the gripper is calculated based on the minimum edge distance principle, and the contact area is determined according to the center point of the surface; when the object to be grasped is an irregular object, the maximum stable gripping width is extracted based on the convex hull algorithm, and the centroid-aligned gripping point is calculated.

[0103] The diameter measurement represents the cylinder diameter obtained through the visual feedback module; the contact points are distributed on the cylinder surface, and the position of the gripper contact is determined based on the cylinder's height and diameter to achieve a uniform gripping force distribution; the minimum edge distance principle selects the shortest edge among the various sides of the cuboid as the gripping direction to ensure that the gripper can firmly grip the object; the face center point is the center point of each face of the cuboid, used to determine the contact area of ​​the gripper, so that the gripper can grip parallel to the object's face; the convex hull algorithm is a computational geometry algorithm used to extract the convex hull of the object, that is, the smallest convex polygon or convex polyhedron containing all points of the object, thereby determining the maximum stable gripping width of the object; the centroid aligned gripping point is determined based on the object's centroid position, so that the gripping force of the gripper passes through the object's centroid, improving the stability and balance of the gripping.

[0104] This embodiment details how, when the gripper contacts the object to be grasped, a gripping strategy is selected based on the object's shape, the initial opening and closing width and target gripping force are calculated, and during the pre-gripping stage, contact deformation data is acquired using a force sensor to assess the object's material softness. This allows for dynamic adjustment of the force controller parameters, and a model predictive controller is used to generate gripper motion commands that satisfy force-position constraints, executing a progressive gripping operation until a stable gripping state is achieved. Furthermore, different initial opening and closing width calculation and contact point distribution methods are employed for objects of different shapes to ensure the gripper can adapt to the gripping needs of various shapes. This series of technical measures effectively improves the robotic arm's adaptive gripping capability in complex environments, especially when handling fragile, soft, or irregularly shaped objects, enabling precise and stable gripping, improving production efficiency and product quality, and expanding the application potential of robotic arms in more fields.

[0105] Preferably, controlling the robotic arm to move using a low-space-occupancy configuration strategy includes:

[0106] The spatial constraint degree is calculated based on the three-dimensional Euclidean distance between the object to be grasped and the obstacle.

[0107] Three-dimensional Euclidean distance refers to the straight-line distance between two points in three-dimensional space; spatial constraint degree represents the degree of spatial restriction that a robotic arm is subject to when moving around an object to be grasped. It is determined by calculating the ratio of the minimum three-dimensional Euclidean distance between the object to be grasped and the obstacle to parameters such as the size of the robotic arm. The greater the spatial constraint degree, the more restricted the space.

[0108] For example, inside a narrow industrial pipe, the object to be grasped is located at coordinates (100, 200, 300) mm, and the obstacle (the inner wall of the pipe) is located at coordinates (150, 200, 300) mm. The calculated three-dimensional Euclidean distance d = 50 mm, and the maximum extension length L of the robotic arm is 500 mm. Therefore, the spatial constraint S = 50 / 500 = 0.1, indicating that the space is relatively limited, requiring a low-space-occupancy configuration strategy.

[0109] When the spatial constraint is greater than the preset threshold, the optimal direction vector of the gripper approaching the object to be grasped is first calculated. Based on the optimal direction vector, the initial pose of the robotic arm with continuous S-shaped bending characteristics is generated by inverse kinematics solution. The joint angles of the robotic arm are iteratively adjusted until the robotic arm reaches the minimum space occupation state. The multi-segment motion trajectory of the gripper is generated according to the adjusted joint angles and the preset gripper movement speed.

[0110] The preset threshold is used to determine whether the space is restricted. It is set according to the working space and application scenario of the robotic arm. For example, it is set to 0.2. When the spatial constraint is less than 0.2, the space is considered restricted. The optimal direction vector represents the direction vector that minimizes the space occupied by the robotic arm when the gripper approaches the object to be grasped. It points to the direction with the largest gap between the object to be grasped and the obstacle. The inverse kinematics solution means that the angles of each joint are solved according to the target position and direction of the robotic arm's end effector (gripper) so that the robotic arm can reach the target position and direction. The continuous S-shaped bending feature means that the joints of the robotic arm bend according to a certain pattern to form a continuous S-shaped curve, reducing the space occupied by the robotic arm during the movement. The iterative adjustment optimizes the pose of the robotic arm by adjusting the joint angles of the robotic arm multiple times to achieve the minimum space occupation state. The multi-segment motion trajectory divides the motion trajectory of the gripper into multiple stages. Each stage corresponds to different joint angles and positions to achieve a smooth and collision-free grasping action.

[0111] For example, inside a narrow industrial pipe, the spatial constraint S = 0.1, which is less than the preset threshold of 0.2. The optimal direction vector for the gripper to approach the object to be grasped is then calculated, assumed to be vector (-1, 0, 0), i.e., along the negative X-axis. Based on this direction vector, the target position of the gripper is set to (90, 200, 300) mm, with the direction consistent with the optimal direction vector. Through inverse kinematics, the initial pose of the robotic arm is generated, causing the arm to exhibit a continuous S-shaped bend, reducing its space occupation within the pipe. The joint angles are then iteratively adjusted, calculating the space occupied after each adjustment, gradually optimizing to the minimum space occupation state. Finally, based on the adjusted joint angles and the preset gripper movement speed of 0.03 m / s, a multi-segment motion trajectory of the gripper is generated: first, the gripper moves along the optimal direction vector at a speed of 0.03 m / s, approaching the object to a certain distance; then, the joint angles of the robotic arm are adjusted, fine-tuning the gripper position and direction to ensure accurate object grasping; finally, the grasping action is completed, avoiding collisions with the pipe wall throughout the entire process.

[0112] When the spatial constraint is less than a preset threshold, a grasping trajectory is generated based on the shape type and three-dimensional pose of the object to be grasped.

[0113] When the spatial constraint is less than a preset threshold, such as on an open workbench where the object to be grasped is a cuboid box, the spatial constraint S = 0.5, which is greater than the preset threshold of 0.2. Based on the box's shape and 3D pose (position (300, 400, 500) mm, horizontal orientation), a parallel grasping trajectory is generated. The gripper approaches the box from above, aligns with one face, and then closes to complete the grasping action. The entire process does not require consideration of spatial constraints, and the grasping trajectory is simple and direct.

[0114] This invention determines whether a space is restricted by calculating spatial constraints. When space is restricted, it calculates the optimal direction vector and generates an initial pose of the robotic arm with continuous S-shaped bending characteristics through inverse kinematics. It then iteratively adjusts joint angles to minimize space occupancy, generating multiple motion trajectories to ensure the gripper can smoothly and accurately approach and grasp the object while avoiding collisions with obstacles. When the space is relatively open, the grasping trajectory is directly generated based on the object's shape and 3D pose, simplifying the control process. This strategy effectively improves the robotic arm's adaptability to different spatial environments, especially in narrow or obstacle-filled scenarios, significantly increasing the success rate and efficiency of grasping, expanding the application range of the robotic arm, and meeting grasping needs in more complex environments.

[0115] Preferably, when controlling the gripper to perform the grasping action, the method further includes:

[0116] The initial joint angle configuration of the robotic arm is calculated based on the target position of the gripper and inverse kinematics.

[0117] The system obtains the space volume occupied by the robotic arm of the humanoid robot in the current configuration, calculates the position error between the actual position and the target position of the gripper, calculates the minimum distance between the robotic arm and the obstacle, and obtains a comprehensive score based on the space volume occupied by the robotic arm, the position error and the minimum distance. The comprehensive score is used to measure the quality of the current joint angle configuration.

[0118] The volume occupied by the robotic arm represents the three-dimensional space volume occupied by the robotic arm in its current configuration, which can be calculated by analyzing the geometry and position of each link of the robotic arm. The minimum distance between the robotic arm and the obstacle is calculated by detecting the geometric relationship between each part of the robotic arm and the obstacle. The comprehensive score is a weighted combination of the volume occupied by the robotic arm, the positional error, and the minimum distance to obtain a score that measures the quality of the current joint angle configuration. The lower the comprehensive score, the better the joint angle configuration.

[0119] The comprehensive score is calculated using the formula: w1 * volume of space occupied by the robotic arm + w2 * positional error between the actual and target positions of the gripper - w3 * minimum distance between the robotic arm and the obstacle. Here, w1, w2, and w3 are weighting coefficients set according to actual needs, for example, w1 = 0.5, w2 = 2.0, w3 = 0.3. On an automated assembly line, the current joint angles of the robotic arm are configured as θ1 = 30°, θ2 = 45°, θ3 = 60°, θ4 = 0°, θ5 = 0°, θ6 = 0°. The calculated volume of space occupied by the robotic arm is V = 0.1 cubic meters, the actual position of the gripper is (200, 300, 390) mm, and the positional error between the gripper and the target position (200, 300, 400) mm is e. position =10mm, the minimum distance d between the robotic arm and the obstacle min=50mm, weighting coefficients w1=0.5, w2=2.0, w3=0.3, calculate the comprehensive score: comprehensive score=0.5×0.1+2.0×0.01+0.3×0.05=0.05+0.02+6=0.085.

[0120] The initial joint angle configuration of the robotic arm is used as the starting point for optimization. The comprehensive score is minimized by the L-BFGS-B algorithm to continuously adjust the joint angles of the robotic arm and obtain the optimal joint angles.

[0121] The L-BFGS-B algorithm is a BFGS algorithm with limited memory, suitable for optimization problems with boundary constraints. It optimizes by approximating the Hessian matrix and utilizing gradient information, and can efficiently handle high-dimensional optimization problems.

[0122] Specifically, assuming an automated assembly line, with an initial joint angle configuration of θ1 = 30°, θ2 = 45°, θ3 = 60°, θ4 = 0°, θ5 = 0°, and θ6 = 0° as the optimization starting point, boundary constraints are set for the joint angles. The L-BFGS-B algorithm is applied for optimization, adjusting the joint angles in each iteration and calculating the comprehensive score. After multiple iterations, the comprehensive score gradually decreases, eventually converging to the minimum value corresponding to the joint angle configuration of θ1 = 32°, θ2 = 48°, θ3 = 55°, θ4 = 5°, θ5 = 3°, and θ6 = 2°. At this point, the robotic arm's occupied space volume decreases, the positional error approaches zero, the minimum distance increases, and the comprehensive score reaches its optimal level, ensuring that the robotic arm occupies minimal space, achieves maximum positional accuracy, and maintains a safe distance from obstacles during the grasping process.

[0123] When controlling the gripper to perform the grasping action, the initial joint angle configuration of the robotic arm is calculated based on the target position of the gripper and inverse kinematics. A comprehensive score is calculated by obtaining the space volume occupied by the robotic arm, position error, and minimum distance. The L-BFGS-B algorithm is used to minimize the comprehensive score and optimize the joint angle configuration of the robotic arm to obtain the optimal joint angle. This process ensures that the robotic arm can move in the optimal posture during the grasping process, which reduces space occupation, improves grasping accuracy and safety, and avoids collisions with obstacles. Especially in complex working environments, by optimizing the joint angle configuration, the working efficiency and reliability of the robotic arm can be effectively improved, and its adaptability in diverse scenarios can be enhanced.

[0124] Preferably, the clamping force of the grippers satisfies the following relationship:

[0125] F target =μ×m×g×SF;

[0126] Where μ represents the coefficient of friction, m represents the mass of the object, g represents the gravitational acceleration, and SF represents the safety factor.

[0127] The coefficient of friction μ can be obtained through experimental measurement or by consulting a material friction coefficient table. A suitable safety factor SF should be selected based on the stability and reliability requirements of the gripping mechanism. Specifically, when gripping an object with a mass of 0.5 kg, the coefficient of friction μ between the object and the gripper contact surface is 0.4, and the safety factor k is 1.5. The minimum gripping force required by the gripper, calculated using the formula, is F = 0.4 × 0.5 × 9.8 × 1.5 = 2.94 N. Therefore, the gripping force of the gripper should be at least 2.94 N to ensure stable gripping of the object and prevent it from slipping due to insufficient gripping force.

[0128] Preferably, a force sensor is also provided, and when adjusting the clamping force of the gripper, a PI controller is used to adjust the clamping force in real time based on the feedback from the force sensor, satisfying the following relationship:

[0129] F adjust =F target +K p e+K i ×∫e dt;

[0130] Among them, F adjust F represents the adjusted clamping force of the grippers. target K represents the target clamping force. p K represents the proportional gain coefficient, e represents the error between the force sensor and the target clamping force, and K represents the proportional gain coefficient. i Let ∫e dt represent the integral gain coefficient, and let ∫e dt represent the integral of the error with respect to time.

[0131] Specifically, a force sensor is mounted on the gripper to measure the gripping force in real time; the target gripping force is the desired gripping force set according to the object's characteristics and gripping requirements; the proportional gain coefficient is used to adjust the response speed and magnitude of the control input to errors; and the integral gain coefficient is used to eliminate the system's steady-state error. The integral of the error over time represents the accumulation of the error between the force sensor measurement and the target gripping force over time, reflecting the system's cumulative deviation.

[0132] When grasping a fragile object, the target clamping force F target The clamping force F is set to 3N and measured in real time by the force sensor. 夹 Initially 2.5N, proportional gain coefficient K p =0.8, integral gain coefficient K i =0.2, calculation error e = 3 - 2.5 = 0.5N, assuming the error remains constant within time t, the integral term is 0.2 × 0.5 × t, the adjusted clamping force F adjust= 3 + 0.8 × 0.5 + 0.2 × 0.5 × t. As time goes on, F... adjust The gripping force of the jaws gradually increases, approaching 3N, ensuring stable gripping of fragile objects without damage.

[0133] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0134] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A visual feedback based adaptive gripper grasping device for humanoid robots, characterized by, The grabbing device comprises a visual feedback module, a gripper control module and a mechanical arm provided with a gripper, and the mechanical arm comprises a plurality of joints. The visual feedback module is used for selecting a to-be-grabbed object by performing fine-grained object classification on environmental information through an improved YOLOv7 model, identifying the shape type and three-dimensional pose of each object, and sending the shape type and three-dimensional pose of the to-be-grabbed object to the gripper control module, and the improvement comprises adding a fine-grained classifier for classifying cylinders, cuboids and irregular objects in the YOLOv7 model. The gripper control module is used for receiving the shape type and three-dimensional pose of the to-be-grabbed object sent by the visual feedback module, and when the space is limited, controlling the mechanical arm to adopt a low-space-occupancy configuration strategy to move based on an improved random sampling path planning algorithm, generating a collision-free trajectory of the gripper from an initial position to the to-be-grabbed object, and the low-space-occupancy configuration strategy comprises cooperative bending of the joints to form a space compression posture, and real-time calculation of minimum swing parameters of each joint movement sequence, the swing parameters comprising joint angle constraints and movement direction priorities. The gripper is controlled to move within a preset speed threshold range when the gripper approaches the to-be-grabbed object, and the gripping force of the gripper is constrained based on the gripping force of the gripper, the shape type and three-dimensional pose of the object when the gripper contacts the to-be-grabbed object. When generating the collision-free trajectory of the to-be-grabbed object, the gripper movement trajectory is adjusted in advance by real-time calculation of the minimum Euclidean distance between the mechanical arm and the obstacle, and the gripper is controlled to perform a segmented grabbing action.

2. The anthropomorphic robot adaptive gripper grasping device according to claim 1, wherein, The visual feedback module comprises a camera, and the selection of the to-be-grabbed object after the fine-grained object classification on the environmental information through the improved YOLOv7 model and the identification of the shape type and three-dimensional pose of each object comprise: loading the YOLOv7 model and the fine-grained classifier, generating a depth map about the object after setting the camera parameters, and obtaining the depth information of the target object; performing target detection on the depth map to obtain the bounding box coordinates, object category and confidence information of all objects in the depth map, selecting the to-be-grabbed object based on the bounding box coordinates, object category and confidence information of the object, extracting the depth information of the to-be-grabbed object region from the depth map according to the bounding box coordinates of the to-be-grabbed object, and calculating the average depth; converting the pixel coordinates into three-dimensional coordinates in the world coordinate system according to the pixel coordinates and depth information of the center point of the to-be-grabbed object, performing fine-grained shape classification on the to-be-grabbed object region, and determining the shape type of the target object; calling a size estimation algorithm to estimate the size information of the to-be-grabbed object according to the shape type of the to-be-grabbed object, and integrating the position, shape type and size information of the target object to generate a first data set required for grabbing.

3. The anthropomorphic robot adaptive gripper grasping device according to claim 1, wherein, Before controlling the gripper to perform the grabbing action, an eye-in-hand transformation matrix for mapping the three-dimensional pose of the to-be-grabbed object to the robot base coordinate system is constructed. The pose list of the gripper and the pose list of the calibration plate are acquired, each pair of adjacent poses is traversed, the pose transformation matrix of the robot is obtained by multiplying the inverse of the pose matrix of the gripper of the previous robot with the pose matrix of the gripper of the current robot, and the pose transformation matrix of the calibration plate is obtained by multiplying the inverse of the pose matrix of the previous calibration plate with the pose matrix of the current calibration plate; The first rotation matrix and the first translation vector are extracted from all the robot pose transformation matrices, the second rotation matrix and the second translation vector are extracted from all the calibration plate pose transformation matrices, the hand-eye rotation matrix is obtained according to the first rotation matrix and the second rotation matrix, and the hand-eye translation vector is obtained according to the first rotation matrix, the hand-eye rotation matrix, the first translation vector and the second translation vector; The hand-eye translation vector and the hand-eye rotation matrix are filled into the homogeneous transformation matrix to obtain the hand-eye transformation matrix.

4. The anthropomorphic robot adaptive gripper grasping device according to claim 1, wherein, Before the gripper contacts the object to be grasped, the method comprises: initializing the gripper dynamics model parameters, the gripper physical parameters and the MPC parameters, wherein the gripper dynamics model parameters comprise a state transition matrix, a control matrix, an output matrix and a direct transfer matrix, the gripper physical parameters comprise an opening width of the gripper and a clamping force threshold, and the MPC parameters comprise a prediction time domain, a control time domain, a state error weight and a control input weight; adjusting the state error weight, the initial clamping force of the gripper and the clamping force threshold according to the fragility of the object to be grasped; generating a first prediction matrix based on the power operation of the state transition matrix, generating a second prediction matrix based on the product operation of the power of the state transition matrix and the control matrix, and constructing a target function according to the first prediction matrix and the second prediction matrix; under the constraints of the control time domain and the clamping force threshold, the target function defines an optimization problem through the quadratic combination of the state error weight and the control input weight, and the clamping force value of the gripper is obtained by solving the optimization problem through convex optimization.

5. The anthropomorphic robot adaptive gripper grasping device according to claim 1, wherein, When the gripper contacts the object to be grasped, the method comprises: selecting a corresponding clamping strategy according to the shape type of the object to be grasped, calculating the initial opening width and the target clamping force of the gripper, and obtaining the contact deformation data through the force sensor in the pre-clamping stage, and evaluating the softness of the object based on the strain rate; dynamically adjusting the stiffness coefficient and the maximum allowable clamping force threshold of the force controller of the gripper according to the softness evaluation result, generating the gripper motion instruction that meets the force-position constraint by using the model predictive controller, and performing the gradual clamping operation until the gripper reaches the stable grasping state; wherein, when the shape type of the object to be grasped is a cylinder, the initial opening width of the gripper is calculated based on the diameter measurement value, and the contact point distribution is determined according to the cylinder height; when the shape type of the object to be grasped is a cuboid, the initial opening width of the gripper is calculated based on the minimum edge distance principle, and the contact area is determined according to the face center point; when the shape type of the object to be grasped is an irregular object, the maximum stable grasping width is extracted based on the convex hull algorithm, and the centroid is aligned with the clamping point.

6. The anthropomorphic robot adaptive gripper grasping device according to claim 1, wherein, The method comprises: calculating the spatial constraint degree based on the three-dimensional Euclidean distance between the object to be grasped and the obstacle; When the space constraint degree is greater than the preset threshold, first, an optimal direction vector of the gripper approaching the object to be grabbed is calculated, based on the optimal direction vector, an initial pose of the mechanical arm with continuous S-shaped bending characteristics is generated through inverse kinematics solving, and the joint angle of the mechanical arm is iteratively adjusted until the mechanical arm reaches a minimum space occupation state, and a multi-segment motion trajectory of the gripper is generated according to the adjusted joint angle and a preset gripper motion speed; When the space constraint degree is less than the preset threshold, a grabbing trajectory is generated based on the shape type and three-dimensional pose of the object to be grabbed.

7. The anthropomorphic robot adaptive gripper grasping device according to claim 1, wherein, When controlling the gripper to perform the grabbing action, further comprising: According to the target position of the gripper and the inverse kinematics, an initial joint angle configuration of the mechanical arm is calculated; The volume of the space occupied by the mechanical arm of the humanoid robot in the current configuration is obtained, the position error between the actual position and the target position of the gripper is calculated, the minimum distance between the mechanical arm and the obstacle is calculated, and a comprehensive score is obtained based on the volume of the space occupied by the mechanical arm, the position error and the minimum distance, the comprehensive score is used to measure the pros and cons of the current joint angle configuration; The initial joint angle configuration of the mechanical arm is taken as the starting point of optimization, and the comprehensive score is minimized through the L-BFGS-B algorithm to continuously adjust the joint angle of the mechanical arm, and the optimal joint angle of the mechanical arm is obtained.

8. The anthropomorphic robot adaptive gripper grasping device according to claim 1, wherein, The clamping force of the gripper satisfies the relationship: F target = μ x m x g x SF; Wherein, μ represents the friction coefficient, m represents the mass of the object, g represents the acceleration of gravity, and SF represents the safety factor.

9. The anthropomorphic robot adaptive gripper grasping device according to claim 1, wherein, There is also a force sensor, and when adjusting the clamping force of the gripper, it includes using a PI controller to adjust the clamping force in real time according to the feedback of the force sensor, satisfying the relationship: F adjust = F target + K p e + K i x ∫ edt; where F adjust represents the adjusted clamping force of the clamping jaw, F target represents the target clamping force, K p represents the proportional gain coefficient, e represents the error of the force sensor and the target clamping force, K i represents the integral gain coefficient, and ∫edt represents the integral of the error with respect to time.

Images