Method and apparatus for object picking and carrying by humanoid robot

Abstract

The application provides a humanoid robot article grabbing and carrying method and device, and relates to the technical field of robot control. The method comprises the following steps: determining an optimal path in response to a movement control instruction, and controlling the humanoid robot to move according to the optimal path; when a coarse positioning completion condition is met, iteratively generating a micro-motion control amount of a waist part of the robot in a near-field fine adjustment mode according to visual pose information and a preset tolerance interval, so as to adjust the standing position of the humanoid robot by using the micro-motion control amount; performing visual positioning on a mechanical arm and a target article twice, and controlling the mechanical arm to execute a first movement track and a second movement track respectively, so as to calibrate a grabbing position; controlling a dexterous hand to complete grabbing of the target article, and generating a third movement track of a double-arm according to a preset carrying pose and a current pose of the mechanical arm, so as to move the target article to the preset carrying pose. By using the above humanoid robot article grabbing and carrying method and device, the success rate of article grabbing and carrying is improved.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and more specifically, to a method and apparatus for humanoid robot to grasp and transport objects. Background Technology

[0002] With the continuous development of humanoid robot control technology, humanoid robots are being used more and more widely in various fields. In practical applications, when humanoid robots grasp and move objects, they usually adopt a serial process of "navigating to a fixed target point → stopping → performing one-time visual recognition → direct grasping".

[0003] However, the control process for humanoid robots to grasp and move objects has the following drawbacks: First, positioning errors are prone to occur after reaching the target point, causing the object to often be at the edge of the camera's field of view, directly affecting the grasping success rate; second, positioning deviations are easily generated when lighting changes, the object is partially obscured, or the viewing angle is poor, resulting in insufficient end-effector positioning accuracy, which amplifies the impact of the low-stiffness humanoid arm model error on the final grasping action; third, for dual-arm collaborative handling tasks, existing methods often plan the trajectories of the two arms separately and then execute them in parallel, which cannot strictly guarantee the consistency of the movement rhythm of the two arms. When the time evolution of the left and right dexterous hand trajectories is not synchronized, it is easy to cause the object to be squeezed, misaligned, or slipped, reducing the grasping success rate. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a humanoid robot method for grasping and transporting objects, so as to overcome at least one of the above-mentioned defects.

[0005] In a first aspect, embodiments of this application provide a method for a humanoid robot to grasp and transport objects, including:

[0006] In response to the movement control command for the humanoid robot, the optimal path for moving from the initial pose to the target work area is determined, and the humanoid robot is controlled to move according to the optimal path;

[0007] When the humanoid robot meets the preset coarse positioning completion conditions, it switches to the near-field fine-tuning mode based on visual feedback. In the near-field fine-tuning mode, the visual pose information of the target object is obtained according to the positioning code. Based on the visual pose information and the preset tolerance range, the micro-motion control quantity of the robot's waist is generated iteratively to adjust the humanoid robot's position until the precise positioning completion conditions are met.

[0008] The robotic arm and the target object are initially located visually to obtain the first grasping point pose. Based on the first grasping point pose and the first robotic arm pose, a first moving trajectory for the two arms to move closer together is generated. The robotic arm is controlled to execute the first moving trajectory to move to the grasping position.

[0009] Secondary visual localization is performed on the robotic arm and the target object to obtain the pose of the second grasping point. Based on the pose of the second grasping point, the pose of the second robotic arm, and the geometric constraints of the target object, a second movement trajectory for the two arms to move closer together is generated. The robotic arm is controlled to execute the second movement trajectory to calibrate the grasping position.

[0010] The dexterous hand is controlled to grasp the target object. A third movement trajectory of the two arms is generated based on the preset handling posture and the current posture of the robotic arm. The robotic arm is then controlled to move the target object to the preset handling posture according to the third movement trajectory.

[0011] Optionally, the preset tolerance range includes three dimensions. Based on the visual pose information and the preset tolerance range, the robot's waist micro-motion control quantity is iteratively generated to adjust the humanoid robot's position. This includes the following steps: in each round of visual measurement, acquiring the visual pose information of the target object in the robot's waist coordinate system, and determining the three-dimensional deviation vector between the visual pose information and the preset alignment posture; when the deviation vector in any dimension of the three-dimensional deviation vector exceeds the corresponding preset tolerance range, generating a micro-motion control command for the humanoid robot in that dimension; after executing the micro-motion control command, the humanoid robot enters a stable waiting state and performs the next round of visual measurement until the deviation vector in any dimension of the three-dimensional deviation vector does not exceed the corresponding preset tolerance range.

[0012] Optionally, the target movement path is generated in the following way: the target movement amount is determined based on the target grasping point pose and the target robotic arm pose; the target movement path is generated based on the target movement amount; the target movement path is processed for progress alignment to obtain the target movement trajectory; wherein, the target movement trajectory is any one of the first movement trajectory, the second movement trajectory, and the third movement trajectory, and the target grasping point pose, the target robotic arm pose, and the target movement amount correspond to the target movement trajectory.

[0013] Optionally, the target movement path includes two initial movement paths corresponding to the left and right robotic arms. The step of performing progress alignment processing on the target movement path to obtain the target movement trajectory includes: determining the number of geometric waypoints for each initial movement path; determining the movement path with the largest number of geometric waypoints among the two initial movement paths as the reference movement path; determining the initial movement paths other than the reference movement path as the specified movement paths; using the reference movement path to perform progress alignment processing on the specified movement paths; and performing unified time parameterization processing on the progress-aligned initial movement paths to generate the target movement trajectory after the left and right robotic arms are synchronized in time.

[0014] Optionally, the step of performing progress alignment processing on a specified movement path using a reference movement path includes: for each initial movement path, determining the path arc length of each waypoint on the initial movement path; normalizing the path arc length of each waypoint to obtain a progress array for the initial movement path, the progress array including a first progress array corresponding to the reference movement path and a second progress array corresponding to the specified movement path; using the elements in the first progress array as standard progress, linearly interpolating the joint values ​​corresponding to the target progress interval in the joint space using the standard progress to obtain the progress-aligned movement path, wherein the target progress interval is the progress interval determined based on the standard progress in the second progress array.

[0015] Optionally, the method further includes: setting multiple marker points on the two dexterous hands of the humanoid robot, including a first marker point located at the web of the thumb, a second marker point located on the thumb, and a third marker point located at the base of the index finger; performing planar fitting on the two dexterous hands based on the multiple marker points. Using the first marker point as the origin, the direction pointing to the second marker point as the X-axis, and the plane normal vector as the Z-axis, the Y-axis direction is determined according to the right-hand rule to determine the hand posture; and determining the current pose of the dexterous hand based on the geometric center of the hand and the hand posture.

[0016] Optionally, both the first gripping point pose and the second gripping point pose include the gripping point poses of the left and right robotic arms respectively. The first gripping point pose and the second gripping point pose are determined by the following method: visually locating the target object to determine the center point pose of the target object; and determining the gripping point poses of the left and right robotic arms respectively corresponding to the first gripping point pose and the second gripping point pose based on the center point pose and the physical dimensions of the target object.

[0017] Optionally, the positioning code includes a symmetrically distributed first positioning code and a second positioning code. The steps of visually positioning the target item and determining the center point pose of the target item include: determining the pose of the first positioning code and the pose of the second positioning code respectively, and determining the center pose of the target item based on the pose of the first positioning code and the pose of the second positioning code.

[0018] Optionally, the coarse positioning completion condition of the humanoid robot can be determined by the following method: during the movement of the humanoid robot, the translation error and heading error between the humanoid robot and the target work area are obtained; when the translation error is less than or equal to a preset distance threshold, the heading error is less than or equal to a preset angle threshold, and the continuous holding time is greater than or equal to a preset time threshold, the humanoid robot is determined to meet the coarse positioning completion condition.

[0019] Secondly, embodiments of this application also provide a humanoid robot object grasping and handling device, the device comprising:

[0020] The motion control module is used to respond to motion control commands for the humanoid robot, determine the optimal path from the initial pose to the target work area, and control the humanoid robot to move according to the optimal path;

[0021] The precise positioning module is used to switch to the near-field fine-tuning mode based on visual feedback when the humanoid robot meets the preset coarse positioning completion conditions. In the near-field fine-tuning mode, the visual pose information of the target object is obtained according to the positioning code. Based on the visual pose information and the preset tolerance range, the micro-motion control quantity of the robot's waist is iteratively generated to adjust the humanoid robot's position until the precise positioning completion conditions are met.

[0022] The first trajectory generation module is used to perform initial visual positioning of the robotic arm and the target object, obtain the first grasping point pose, generate a first movement trajectory for the two arms to move closer together based on the first grasping point pose and the first robotic arm pose, and control the robotic arm to execute the first movement trajectory to move to the grasping position.

[0023] The second trajectory generation module is used to perform secondary visual positioning of the robotic arm and the target object to obtain the pose of the second grasping point. Based on the pose of the second grasping point and the pose of the second robotic arm, a second movement trajectory is generated for the two arms to move closer together. The robotic arm is controlled to execute the second movement trajectory to calibrate the grasping position.

[0024] The third trajectory generation module is used to control the dexterous hand to complete the grasping of the target object. It generates the third movement trajectory of the two arms according to the preset handling posture and the current posture of the robotic arm, so as to control the robotic arm to move the target object to the preset handling posture according to the third movement trajectory.

[0025] The embodiments of this application bring the following beneficial effects:

[0026] This application provides a humanoid robot object grasping and handling method and apparatus. After the humanoid robot arrives at the target work area, it does not directly grasp the object. Instead, it uses visual feedback to obtain relative deviations and performs micro-motion corrections on the humanoid robot, causing the object to re-enter the visual center area, thus compensating for the positioning error caused by bipedal walking. Subsequently, a two-stage visual positioning mechanism controls the robotic arm to move to the grasping position, improving end-effector positioning accuracy and reducing the impact of low-rigidity humanoid arm model errors. Finally, in a dual-arm collaborative handling scenario, the trajectories of the left and right robotic arms are unified on the same time axis for interpolation and synchronous execution, avoiding problems such as squeezing, offset, or unstable gripping caused by asynchronous movements of the left and right arms. Compared with existing humanoid robot object grasping and handling methods, this improves the grasping stability and success rate of the humanoid robot in production line material handling scenarios.

[0027] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0028] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 A flowchart of the humanoid robot object grasping and handling method provided in an embodiment of this application is shown;

[0030] Figure 2 A flowchart illustrating the steps for determining the optimal path provided in an embodiment of this application is shown;

[0031] Figure 3 A schematic diagram of the location code of the target item provided in the embodiments of this application is shown;

[0032] Figure 4 A schematic diagram showing the location of the marker points provided in the embodiments of this application is shown;

[0033] Figure 5 A flowchart illustrating the steps for generating a target movement trajectory provided in an embodiment of this application is shown;

[0034] Figure 6 A flowchart illustrating the progress alignment process steps provided in an embodiment of this application is shown;

[0035] Figure 7 This paper shows a schematic diagram of the structure of the humanoid robot object grasping and handling device provided in an embodiment of this application;

[0036] Figure 8 A schematic diagram of the structure of the electronic device provided in the embodiments of this application is shown. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0038] To facilitate understanding of this embodiment, the following description uses the humanoid robot object grasping and handling method provided in this application embodiment applied to a humanoid robot object grasping and handling system as an example to illustrate the exemplary steps provided in this application embodiment. The humanoid robot object grasping and handling system includes a controller, a navigation and obstacle avoidance module, a vision recognition module, a walking control module, an upper limb control module, and a dexterous hand control module.

[0039] Please see Figure 1 , Figure 1 This is a flowchart illustrating a method for grasping and transporting objects using a humanoid robot, as provided in an embodiment of this application. Figure 1 As shown in the embodiment of this application, the humanoid robot object grasping and handling method includes:

[0040] Step S101: In response to the movement control command for the humanoid robot, determine the optimal path from the initial pose to the target work area, and control the humanoid robot to move according to the optimal path.

[0041] The movement control command refers to the command that controls the humanoid robot to move to the target work area. The movement control command is sent from the controller to the navigation and obstacle avoidance module, which determines the optimal path based on the movement control command.

[0042] The initial pose refers to the pose of the humanoid robot at its starting position, that is, the pose before executing movement control commands. The target work area refers to the target location that the humanoid robot is expected to reach.

[0043] Humanoid robots include humanoid robots with different modes of locomotion, such as bipedal humanoid robots, wheeled humanoid robots with differential wheels, and wheeled humanoid robots with omnidirectional wheels.

[0044] The following reference Figure 2 This section will introduce the process of determining the optimal path.

[0045] Figure 2A flowchart illustrating the steps for determining the optimal path provided in an embodiment of this application is shown, as follows: Figure 2 As shown, the steps for determining the optimal path include:

[0046] Step S1011: Determine the reference path for moving from the initial pose to the target work area.

[0047] The navigation and obstacle avoidance module completes environmental mapping of the work area through manual remote control, and obtains a static map of the work area. The static map is stored in a static map file, which contains a two-dimensional occupancy map image in PGM format, a two-dimensional occupancy map description file in YAML format, and a three-dimensional point cloud map.

[0048] The navigation and obstacle avoidance module generates a global cost map based on the static map. After receiving the movement control command for the humanoid robot, the navigation and obstacle avoidance module extracts the target work area from the movement control command and solves the reference path from the initial pose to the target work area on the global cost map.

[0049] Step S1012: Generate multiple short-term predicted trajectories based on the reference path;

[0050] The navigation and obstacle avoidance module obtains image data collected by the LiDAR from the visual recognition module, maintains the local obstacle layer and safety expansion zone of the working area based on the image data, and constructs a cost map based on the local obstacle layer and safety expansion zone.

[0051] The process of moving to the target work area includes multiple control cycles. In each control cycle, multiple short-term predicted trajectories are generated in parallel based on the reference path and the local obstacle layer.

[0052] Step S1013: Determine the comprehensive movement cost of each short-term predicted trajectory, and determine the short-term predicted trajectory with the minimum comprehensive movement cost as the optimal path, so as to control the humanoid robot to move according to the optimal path.

[0053] For each short-term predicted trajectory, determine the path tracking error, obstacle distance, and speed smoothness of the short-term predicted trajectory. Based on the path tracking error, obstacle distance, and speed smoothness, determine the comprehensive movement cost of the short-term predicted trajectory.

[0054] The short-time predicted trajectory with the minimum overall movement cost among multiple short-time predicted trajectories is selected as the optimal path for the current control period. The optimal path includes the linear velocity and angular velocity of each path point. By determining the overall movement cost and selecting the optimal path, local trajectory optimization can be achieved within the current control period.

[0055] In one embodiment, when the humanoid robot moves according to the optimal path, if the navigation and obstacle avoidance module detects temporary blockages or local minima during the movement—for example, if the navigation and obstacle avoidance module detects local optimization failure, obstacles occupying the path for a long time, or increased oscillations—then it selects a recovery behavior from preset recovery behaviors as the target recovery behavior and executes the target recovery behavior to solve the temporary blockage or local minima problem. The preset recovery behaviors include at least one of the following: the humanoid robot rotating in place, short-range backward movement, and global replanning. Additionally, if the navigation and obstacle avoidance module detects localization degradation during the movement, it increases the particle number or performs a small-range scan to restore convergence.

[0056] In one embodiment, in order to improve the adaptability of the humanoid robot object grasping and handling system and make it applicable to humanoid robots with different modes of movement, the walking control module can set up its own control model for humanoid robots with different modes of movement. The control model includes a dynamic model and a kinematic model.

[0057] When controlling the humanoid robot's movement, the walking control module can acquire the target control model and target constraint set corresponding to the robot's movement mode, i.e., switch the control model and constraint set according to the robot's movement mode. Then, the linear velocity and angular velocity in the optimal path are input into the target control model to obtain control quantities that can be executed by the mobile chassis, thereby controlling the humanoid robot's movement according to the target control model and target constraint set under the optimal path.

[0058] During the movement of the humanoid robot, it is necessary to detect the current pose of the humanoid robot in real time to determine whether the humanoid robot meets the coarse localization completion conditions. At this time, the navigation and obstacle avoidance module will adopt a time-holding plus hysteresis double threshold rule to determine whether the humanoid robot has entered the coarse localization area based on the positional relationship between the current pose of the humanoid robot and the target working area.

[0059] Specifically, the translational and heading errors between the humanoid robot's current pose and the target work area can be determined. When the translational error meets the first error condition, the heading error meets the second error condition, and the continuous holding time meets the preset duration condition, it is determined that the robot has entered the coarse positioning area and meets the coarse positioning completion condition. When the translational error does not meet the first error condition, the heading error does not meet the second error condition, or the continuous holding time does not meet the preset duration condition, it is determined that the robot has not entered the coarse positioning area and does not meet the coarse positioning completion condition.

[0060] If, during the holding time, the translation error does not meet the first error condition or the heading error does not meet the second error condition, the holding time is reset and the holding time is recalculated, while confirming that the coarse positioning area has not been entered.

[0061] Specifically, when the translation error is less than or equal to a preset distance threshold, the translation error is determined to meet the first error condition; when the translation error is greater than the preset distance threshold, the translation error is determined to not meet the first error condition. When the heading error is less than or equal to a preset angle threshold, the heading error is determined to meet the second error condition; when the heading error is greater than the preset angle threshold, the heading error is determined to not meet the second error condition. The continuous holding time refers to the continuous duration during which both the translation error and the heading error meet the second error condition. When the continuous holding time is greater than a preset time threshold, the continuous holding time is determined to meet the preset duration condition; when the continuous holding time is less than or equal to the preset time threshold, the continuous holding time is determined to not meet the preset duration condition. The entry distance threshold, entry angle threshold, and preset time threshold are different for humanoid robots with different movement modes.

[0062] Step S102: When the humanoid robot meets the preset coarse positioning completion conditions, switch to the near-field fine-tuning mode based on visual feedback. In the near-field fine-tuning mode, obtain the visual pose information of the target object according to the positioning code. Based on the visual pose information and the preset tolerance range, iteratively generate the micro-motion control quantity of the robot's waist, so as to adjust the humanoid robot's position using the micro-motion control quantity until the precise positioning completion conditions are met.

[0063] The target item can refer to the item to be grabbed. For example, the target item could be a container.

[0064] After the humanoid robot enters the coarse positioning area, it switches to the near-field fine-tuning mode based on visual feedback. In the near-field fine-tuning mode, the end-effector pose is visually closed-loop fine-tuned to achieve precise positioning.

[0065] Specifically, after determining that the coarse positioning completion conditions are met, the navigation obstacle avoidance module sends a coarse positioning completion signal to the controller. After receiving the coarse positioning completion signal, the controller enters the near-domain fine-tuning mode.

[0066] In one embodiment, when determining the visual pose information of a target item, the controller instructs the visual recognition module to identify multiple positioning codes set on the target item. The visual recognition module determines the visual pose information of the target item based on the multiple positioning codes set on the target item. The visual pose information includes the current pose of the target item. The visual recognition module uses an RGB-D camera as its core sensor and has a target detection model set internally.

[0067] The following reference Figure 3 Let's introduce the location code of the target item.

[0068] Figure 3 This illustration shows a schematic diagram of the location code of the target item provided in an embodiment of this application, such as... Figure 3As shown, the target item 200 is a cube. Two positioning codes, 201 and 202, are symmetrically arranged on the left and right sides of the top cover of the target item 200. The visual recognition module acquires color and depth images of the positioning codes using an RGB-D camera. These images are then input into the target detection model to determine the point cloud data of the region where each positioning code is located. The target detection model then performs plane fitting and noise removal on the point cloud data to determine the planar image of each positioning code and, based on the planar image, determines the geometric center point of each positioning code. Finally, based on the geometric center point of each positioning code, the visual pose information of the target item is determined.

[0069] In one embodiment, precise positioning of the humanoid robot can be achieved through multiple rounds of visual measurement.

[0070] Specifically, in each round of visual measurement, the visual pose information of the target object in the robot's waist coordinate system is acquired, and a three-dimensional deviation vector between the visual pose information and the preset alignment posture is determined. When the deviation vector in any dimension of the three-dimensional deviation vector exceeds the corresponding preset tolerance range, a micro-motion control command for the humanoid robot in that dimension is generated. After executing the micro-motion control command, the humanoid robot enters a stable waiting state and performs the next round of near-field fine-tuning until the deviation vector in any dimension of the three-dimensional deviation vector does not exceed the corresponding preset tolerance range. The preset tolerance range includes three dimensions: the first dimension corresponding to the forward and backward direction, the second dimension corresponding to the left and right direction, and the third dimension corresponding to the yaw angle direction.

[0071] For example: First, the visual pose information of the target object in the camera coordinate system is obtained through the visual recognition module, and then the relative pose of the target object in the waist coordinate system is obtained through transformation. The relative pose is denoted as: ; ,in, Indicates the forward distance between the target object and the humanoid robot. This indicates the left or right offset of the target item. Indicates the height of the target item. This indicates the yaw angle. Simultaneously, a preset alignment posture is established for the arms to stably grasp and transport the target object. This preset alignment posture is denoted as: , Then, the three-dimensional deviation vector is calculated, which can be expressed as: , , .

[0072] To ensure the correct relative position of the arms during grasping and handling, preset tolerance ranges can be set for each of the three degrees of freedom, for example: , , The preset tolerance range is set to ensure that the target item is within the workable area of ​​the coordinated action of the two arms. If the target item is too far from the workable area, such as if it is too far to the left or right or too skewed at an angle, the two arms may not be able to make contact at the same time or provide stable support. Therefore, the preset tolerance range is both the standard for determining alignment and the constraint condition for the feasibility of the two arms to grasp.

[0073] After each acquisition of visual pose, the system determines which dimensions need correction based on the 3D deviation vector. Only dimensions exceeding the preset tolerance range are given micro-motion control commands in the waist coordinate system. These micro-motion control commands include micro-motion control quantities, denoted as u. It then performs a short translation or a small rotation. After executing the micro-motion control command, it enters a static waiting state to ensure that the camera's field of view is stable and does not shake. Then, it performs visual measurement again and determines the three-dimensional vector deviation. The above process is repeated iteratively until all three-dimensional deviation vectors fall within the preset tolerance range, thus confirming that the conditions for accurate positioning are met, and thus completing the accurate stance suitable for stable dual-arm grasping.

[0074] When the conditions for accurate positioning are met, the vision recognition module sends an alignment completion signal to the controller. The controller then sends a vision positioning command to the vision recognition module based on the alignment completion signal, so that the vision recognition module can perform the first vision positioning.

[0075] It should be noted that the robot parameters set above are all configuration data, which can be tuned and transplanted for bipedal humanoid robots, wheeled humanoid robots, and wheeled composite robots, improving the adaptability of humanoid robot object grasping and handling systems. The robot parameters include, but are not limited to: map resolution, safety expansion distance, control cycle, prediction time domain, number of samples, cost weight, entry distance threshold, entry angle threshold, and preset time threshold.

[0076] Step S103: Perform initial visual localization of the robotic arm and the target object to obtain the first grasping point pose. Based on the first grasping point pose and the first robotic arm pose, generate a first moving trajectory for the two arms to move closer together. Control the robotic arm to execute the first moving trajectory to move to the grasping position.

[0077] During the initial visual positioning, the gripping point pose of the target item can be determined based on the geometric center point of each positioning code set on the target item.

[0078] For example, the midpoint of the line connecting the geometric centers of two positioning codes can be defined as the origin of the object coordinate system. The direction of the line connecting the two geometric centers is the Y-axis, and the normal vector of the plane containing the positioning codes is the Z-axis. The X-axis direction is determined according to the right-hand rule to establish the object coordinate system. Then, based on the position of the grasping point in the object coordinate system, the first grasping point pose in the camera coordinate system is determined. After coordinate transformation, the first grasping point pose is input into the controller of the humanoid robot object grasping and handling system. After obtaining the first grasping point pose, the controller sends trajectory planning instructions to the upper limb control module.

[0079] After receiving the trajectory planning instruction, the upper limb control module generates a first movement trajectory for the two arms to move closer together based on the first grasping point pose and the first robotic arm pose, and controls the left and right robotic arms to execute the first movement trajectory to move to the grasping position.

[0080] Step S104: Perform secondary visual positioning on the robotic arm and the target object to obtain the second grasping point pose. Based on the second grasping point pose, the second robotic arm pose, and the geometric constraints of the target object, generate a second movement trajectory for the two arms to move closer together. Control the robotic arm to execute the second movement trajectory to calibrate the grasping position.

[0081] After completing the initial visual positioning, a second visual positioning is performed using the same method as the initial visual positioning. Based on the pose of the second gripping point and the pose of the second robotic arm, a second movement trajectory is generated for the two arms to move closer together. The left and right robotic arms are then controlled to execute the second movement trajectory to calibrate the gripping position.

[0082] Step S105: Control the dexterous hand to complete the grasping of the target item. Generate a third movement trajectory for both arms based on the preset handling posture and the current posture of the robotic arm, so as to control the robotic arm to move the target item to the preset handling posture according to the third movement trajectory.

[0083] The preset transport posture can refer to the final posture of the target item being transported. For example, the preset transport posture can refer to the final posture after the target item is picked up and transported to another platform.

[0084] When controlling the robotic arm to grasp a target item, the robotic arm is actually controlled to move according to the second movement trajectory, and then the dexterous hand set on the robotic arm completes the grasping of the target item.

[0085] Before using a dexterous hand to grasp a target object, the position of the dexterous hand needs to be accurately located. To facilitate the identification of the dexterous hand's position, multiple markers can be set on each dexterous hand of the humanoid robot. These markers include a first marker set at the base of the thumb, a second marker set on the thumb, and a third marker set at the base of the index finger.

[0086] The following reference Figure 4 Let's introduce how to set the location of the marker points.

[0087] Figure 4 A schematic diagram showing the location of the marker points provided in an embodiment of this application is shown, such as... Figure 4 As shown, the dexterous hand includes a left dexterous hand 310 and a right dexterous hand 320. The left dexterous hand 310 is provided with a first marking point 311 at the web of the thumb, a second marking point 312 on the thumb, and a third marking point 313 at the base of the index finger; the right dexterous hand 320 is provided with a first marking point 321 at the web of the thumb, a second marking point 322 on the thumb, and a third marking point 323 at the base of the index finger.

[0088] For each dexterous hand, the visual recognition module performs planar fitting based on multiple marker points on the dexterous hand to determine the geometric center of the hand. Taking the first marker point as the origin, the direction pointing to the second marker point as the X-axis, and the plane normal vector as the Z-axis, the Y-axis direction is determined according to the right-hand rule to determine the hand posture. Based on the hand geometric center and hand posture, the six-dimensional pose of the dexterous hand is determined, and the six-dimensional pose provides high-precision input for dexterous hand control when grasping target objects.

[0089] Each dexterous hand has multiple degrees of freedom, and the control of these multiple degrees of freedom is achieved by the dexterous hand control module. The dexterous hand control module is connected to the controller via an RS485 bus, and the controller can send 12 data streams to the dexterous hand control module at once.

[0090] In one embodiment, in order to maintain controllability and reproducibility under conditions such as stalling / drifting, the previous dexterous hand pose can be stored in the state cache hand_cmd_cache of the dexterous hand control module, so as to save the dexterous hand's grasping pose through the state cache.

[0091] Specifically, before using the dexterous hand to grasp or move a target object, the dexterous hand control module receives the hand type and gear position from the controller. The hand type can refer to the identifier of the dexterous hand, including left hand, right hand, and both hands. The hand type is used to indicate the target dexterous hand that needs to be updated, and the gear position is used to indicate the type of grasping action.

[0092] After receiving the hand position and gear, the dexterous hand control module reads the previous dexterous hand pose from the state buffer. The previous dexterous hand pose includes complete 12 data streams (i.e., 12 joint angles). Then, it copies the previous dexterous hand pose as the reference pose and stores it in the preset pose parameter cmd, thus using the previous dexterous hand pose to ensure the continuity of the movement. The 12 data streams include 6 data streams from the left robotic arm and 6 data streams from the right robotic arm, and the 12 data streams are arranged with the 6 data streams from the left robotic arm first and the 6 data streams from the right robotic arm second. Each of the 6 data streams corresponds to one joint angle of the dexterous hand.

[0093] Then, the dexterous hand control module determines the grasping pose of the humanoid robot's dexterous hand based on the hand type, gripping position, and reference pose. For example, the dexterous hand control module selects the target angle template for the current frame from the preset angle template based on the hand type and grasping position. The target angle template contains the 6-way pose of each dexterous hand. Then, according to the data buffer offset rules (the offset of the 6-way data for the left robotic arm is 0, and the offset of the 6-way data for the right robotic arm is 6), the reference pose in the cmd is updated using the target angle template to obtain the grasping pose.

[0094] When the hand is not both hands, the grasping pose of the specified dexterous hand is kept unchanged according to the data cache offset rule, and the grasping pose of the target dexterous hand is updated to achieve the baseline pose update. Here, the specified dexterous hand is a dexterous hand other than the target dexterous hand.

[0095] Taking the left hand as an example, the target dexterous hand is the left hand, and the designated dexterous hand is the right hand. When updating the reference pose, the last 6 data paths of the reference pose remain unchanged, while the first 6 data paths are updated to the target angle template. The updated reference pose is the grasping pose, which can be used to control the dexterous hand to grasp and move the target object. Taking both hands as an example, the target dexterous hands are the left and right hands. When updating the reference pose, all 12 data paths of the reference pose are replaced with the target angle template, and the updated reference pose is the grasping pose.

[0096] The grasping pose in the preset pose parameter cmd is sent to the dexterous hand driving layer so that the dexterous hand can perform corresponding actions according to the updated 12-way angle.

[0097] After executing the updated 12 angles, the captured pose in the preset pose parameter cmd is rewritten back into the state cache hand_cmd_cache, overwriting the cached state of the previous frame, and serving as the base state for the next frame's action. In this way, the system always starts from the latest pose cache, making local replacements only based on hand type and gear position, thus achieving independent or synchronous control of the left and right dexterous hands.

[0098] The following reference Figure 5 The process of generating the target movement trajectory will be introduced. The generation process of the first movement trajectory, the second movement trajectory, and the third movement trajectory is the same. The target movement trajectory is any one of the first movement trajectory, the second movement trajectory, and the third movement trajectory.

[0099] Figure 5 A flowchart illustrating the steps for generating the target movement trajectory provided in an embodiment of this application is shown, as follows: Figure 5 As shown, the steps for generating the target movement trajectory include:

[0100] Step S1031: Determine the target movement amount based on the target grasping point pose and the target robotic arm pose.

[0101] The target grasping point pose, target robotic arm pose, and target movement amount are all determined based on the target movement trajectory. For example: if the target movement trajectory is the first movement trajectory, then the target grasping point pose is the first grasping point pose, the target robotic arm pose is the first robotic arm pose, and the target movement amount is the first movement amount; if the target movement trajectory is the second movement trajectory, then the target grasping point pose is the second grasping point pose, the target robotic arm pose is the second robotic arm pose, and the target movement amount is the second movement amount; if the target movement trajectory is the third movement trajectory, then the target grasping point pose is the preset handling pose, the target robotic arm pose is the current pose of the robotic arm, and the target movement amount is the third movement amount.

[0102] Specifically, the visual recognition module determines the center point pose of the target object, which is denoted as: , The grasping point pose is calculated based on the center point pose and physical dimensions of the target object. The grasping point pose includes a first grasping point pose and a second grasping point pose. The grasping point pose includes the left-hand grasping point pose and the right-hand grasping point pose. The left-hand grasping point pose is... The right hand's grasping position is ,in, , , and , , All are provided by an attitude adaptation algorithm. Simultaneously, the target robotic arm pose can be obtained. The target robotic arm includes the individual poses of the two robotic arms, namely, the left and right gripping point poses corresponding to the first and second gripping point poses. The left robotic arm pose is... The right robotic arm is in the following position: .

[0103] The target displacement can be calculated based on the target robotic arm's pose. This displacement includes the displacement of each of the left and right robotic arms. The displacement of the left robotic arm is... The movement of the right robotic arm is , and It directly describes the amount of spatial correction required for the arms to converge from the ready position to the final grasping point.

[0104] In one embodiment, when determining the center point pose of a target item, a first positioning code pose and a second positioning code pose can be determined respectively; the center point pose of the target item is determined based on the first positioning code pose and the second positioning code pose. The first positioning code pose may refer to the position of the first positioning code, and the second positioning code pose may refer to the pose of the second positioning code.

[0105] Step S1032: Generate the target movement path based on the target movement amount.

[0106] The upper limb control module is implemented based on ROS2 middleware and the MoveIt 2 plug-in planning framework. For external modules, the upper limb control module receives the grasping point pose of the target object from the controller through a unified action / service interface. Internally, the humanoid robot's geometry / inertia and joint upper and lower limits are described using the Unified Robot Description Format (URDF), the planning group is defined using the Semantic Robot Description Format (SRDF), and the inverse kinematics objective is solved using the Kinematics and Dynamics Library (KDL) to enable robot self-collision checks.

[0107] The upper limb control module supports both single-arm and dual-arm grasping tasks. When performing a single-arm grasping task, the robotic arm grasping type is set to single-arm grasping; when performing a dual-arm grasping task, the robotic arm grasping type is set to dual-arm grasping.

[0108] For different grasping tasks, the upper limb control module uses different methods to determine the movement path.

[0109] In one scenario, the robotic arm's grasping type is single-arm grasping. In this case, the target robotic arm is selected from the dual robotic arms based on the planar projection of the target object in the waist coordinate system. For example, using the humanoid robot's X-axis as the dividing line, if the center of the target object is to the left of the dividing line (y>0), the target robotic arm is determined to be the left arm; if the center of the target object is to the right of the dividing line (y<0), the target robotic arm is determined to be the right arm. Then, using a preset algorithm, the movement trajectory of the target robotic arm under the single-arm grasping type is generated based on the current pose of the target robotic arm and the pose of the grasping point.

[0110] The waist coordinate system is used to connect the base coordinate system with the kinematic calculation of the upper limbs / trunk, simplifying the posture and position calculation of multi-joint linkage. The X-axis of the waist coordinate system extends horizontally along the front of the robot, pointing in the direction of the robot's operation "front". The Z-axis of the waist coordinate system is perpendicular to the ground. The Y-axis of the waist coordinate system is derived by the right-hand rule and extends horizontally along the side of the robot.

[0111] In another scenario, the robotic arm grasping type is a dual-arm grasping type. In this case, based on the target grasping point pose and the target robotic arm pose in the waist coordinate system, a separate movement path is generated for each robotic arm. Each movement path satisfies reachability and collision avoidance constraints, and each movement path only includes multiple geometric waypoints, excluding velocity and acceleration. Then, the movement path with the largest number of geometric waypoints is determined as the reference movement path, and the movement trajectory of the two robotic arms in the dual-arm grasping type is determined based on the reference movement path.

[0112] It should be noted that there are prerequisite dependencies between movement control commands, object recognition commands, trajectory planning commands, and object grabbing commands. That is, the next command can only be executed after the previous command has been completed. For example, the trajectory planning command can only be executed after the object recognition command has been successfully executed, and the object grabbing command can only be executed after the trajectory planning command has been successfully executed.

[0113] Step S1033: Align the target movement path to obtain the target movement trajectory.

[0114] The target movement path includes two initial movement paths corresponding to the left and right robotic arms.

[0115] The following reference Figure 6 This section will detail the specific process of progress alignment.

[0116] Figure 6 A flowchart illustrating the progress alignment process steps provided in an embodiment of this application is shown, as follows: Figure 6 As shown, the progress alignment process includes:

[0117] Step S1041: The movement path with the largest number of geometric waypoints among the two initial movement paths is determined as the baseline movement path.

[0118] Determine the number of geometric waypoints for each initial movement path. Select the movement path with the largest number of geometric waypoints from the two initial movement paths as the base movement path. Select the initial movement paths other than the base movement path as the specified movement paths.

[0119] Step S1042: Using the baseline movement path, perform progress alignment processing on the specified movement path.

[0120] The first step is to determine the path arc length of each waypoint on each initial movement path.

[0121] For example, each initial movement path includes multiple waypoints. To project two initial movement paths with different discrete densities into the same progress domain, the path arc length is used as a metric. Path arc length of each waypoint Current joint position relative to the initial joint position The Euclidean distance between them .

[0122] The second step is to normalize the path arc length of each waypoint to obtain the progress array of the initial movement path.

[0123] The path arc length of each waypoint is normalized to obtain the progress array. The calculation method is as follows: ,in, Indicates the waypoint number. This indicates the number of waypoints for the left and right robotic arms, with 'i' used to distinguish between the left and right robotic arms.

[0124] Each robotic arm includes multiple sets of joint values, the number of which corresponds to the number of waypoints. Each set of joint values ​​includes 7 joint data points. Each initial movement path corresponds to a progress array, which includes a first progress array corresponding to the baseline movement path and a second progress array corresponding to the specified movement path.

[0125] The third step is to use the scale value in the first progress array as the standard progress, and then use the standard progress to perform linear interpolation on the joint value corresponding to the target progress interval in the joint space to obtain the movement path after progress alignment.

[0126] The target schedule interval can refer to the schedule interval determined based on the standard schedule in the second schedule array.

[0127] The standard schedule is selected using the principle of "more is better". For example, assuming the number of waypoints on the two initial movement paths are respectively... and ,from and The waypoint with the most occurrences is selected as the standard schedule. Then, the initial movement path corresponding to the standard schedule is determined as the base movement path, and another initial movement path is determined as the specified movement path.

[0128] Using each standard progress in the first progress array as a baseline, find the target progress interval corresponding to that standard progress position in the second progress array. In joint space, resample the joint values ​​corresponding to the target progress interval using linear interpolation. By traversing all standard progresses in the first progress array in this manner, a new waypoint sequence can be obtained, which is the specified movement path after progress alignment.

[0129] It should be noted that there is a one-to-one correspondence between joint values ​​and waypoints. When joint values ​​are aligned, the movement paths are also aligned.

[0130] Step S1043: Perform unified time parameterization processing on the initial movement path after progress alignment to generate the target movement trajectory after the left and right robotic arms are synchronized in time, so as to control the movement of the two arms.

[0131] Based on the initial movement path after progress alignment, a joint geometric trajectory is constructed. Under preset kinematic constraints, the joint geometric trajectory is subjected to time parameterization to generate the target movement trajectory.

[0132] For example: Since the joint values ​​in the first schedule array and the second schedule array correspond, each standard schedule corresponds to a pair of joint vectors. , The two-arm states are obtained by concatenating the joint vectors in a preset order. The joint geometric trajectory is formed by the two-arm states corresponding to all progresses. The joint geometric trajectory is as follows:

[0133] ;

[0134] In the above formula, The joint geometric trajectory consists only of the position sequence, with the time field left blank. Then, the joint geometric trajectory is input into the Time-Optimal Trajectory Generation (TOTG) algorithm for time tuning. Under the constraints of preset joint velocity and / or acceleration limits and scaling factors, a strictly increasing timestamp sequence is determined, thereby outputting a movement trajectory that satisfies preset kinematic constraints on a unified time axis. This movement trajectory includes the joint geometric trajectory, the timestamp sequence, and each velocity and acceleration.

[0135] In one embodiment, the humanoid robot object grasping and handling system further includes an image transmission module. The image transmission module adopts point-to-point video transmission technology, and the technology stack includes the ROS2 communication framework, depth camera driver, image processing library, and multimedia transmission framework.

[0136] The humanoid robot object grasping and handling system operates an image acquisition and encoding node on the humanoid robot's main control terminal. It utilizes OpenCV and GStreamer to perform real-time image compression and network transmission, sending the images to a host computer (controller). The host computer decodes and displays the images via a standard decoding interface or a custom display program. The image transmission module establishes a direct communication channel between the robot and the host computer (controller) through a local area network, enabling end-to-end transmission of the image stream without relying on a centralized server or cloud forwarding.

[0137] In one embodiment, the humanoid robot object grasping and handling system also includes an external communication module. The external communication module is based on the TCP / IP network communication mechanism, with a multi-instance TcpClient class as its core, and supports simultaneous connection to multiple server ports to realize multi-source command input and unified arbitration control.

[0138] For example, when a humanoid robot object grasping and handling system starts up, an independent communication channel is established for each external terminal. Each channel independently receives command data and parses it into a standardized instruction format. The parsing results are written to the shared command area via a thread-safe caching mechanism, and the command source identifier is recorded. The external terminal can be a host computer, smartphone, or tablet computer.

[0139] Meanwhile, to avoid conflicts caused by multiple external terminals controlling the humanoid robot simultaneously, the external communication module has a built-in command interlock mechanism. When a command issued by any external terminal enters the execution state, the external communication module temporarily locks the channels of other external terminals, allowing them to receive commands but not trigger execution. After the current task is completed or timeout, the lock is automatically released, thus ensuring that only one external command is executed by the humanoid robot's object grasping and handling system at any given time.

[0140] The humanoid robot object grasping and handling method provided in this application can, after the humanoid robot arrives at the target work area, not directly grasp the object, but use the relative deviation obtained from visual feedback to make micro-motion corrections to the humanoid robot, so that the object re-enters the visual center area, thereby compensating for the positioning error caused by bipedal walking; subsequently, a two-stage visual positioning mechanism is used to control the robotic arm to move to the grasping position, improving the end-effector positioning accuracy and reducing the impact of low-rigidity humanoid arm model errors; finally, in the dual-arm collaborative handling scenario, the trajectories of the left and right robotic arms are unified on the same time axis for interpolation and synchronous execution, avoiding problems such as squeezing, offset, or unstable gripping caused by asynchronous movements of the left and right robotic arms. Compared with the existing humanoid robot object grasping and handling methods, this method improves the grasping stability and grasping success rate of the humanoid robot in the production line material handling scenario.

[0141] Based on the same inventive concept, this application also provides a humanoid robot object grasping and handling device corresponding to the humanoid robot object grasping and handling method. Since the principle of the device in this application is similar to the humanoid robot object grasping and handling method described above in this application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0142] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of a humanoid robot object grasping and handling device provided in an embodiment of this application. Figure 7 As shown, the humanoid robot object grasping and handling device 400 includes:

[0143] The motion control module 401 is used to respond to the motion control command for the humanoid robot, determine the optimal path for moving from the initial pose to the target work area, and control the humanoid robot to move according to the optimal path;

[0144] The precise positioning module 402 is used to switch to the near-field fine-tuning mode based on visual feedback when the humanoid robot meets the preset coarse positioning completion conditions. In the near-field fine-tuning mode, the visual pose information of the target object is obtained according to the positioning code. Based on the visual pose information and the preset tolerance range, the micro-motion control quantity of the robot's waist is iteratively generated to adjust the humanoid robot's position until the precise positioning completion conditions are met.

[0145] The first trajectory generation module 403 is used to perform initial visual positioning of the robotic arm and the target object, obtain the first grasping point pose, generate a first movement trajectory for the two arms to move closer together based on the first grasping point pose and the first robotic arm pose, and control the robotic arm to execute the first movement trajectory to move to the grasping position.

[0146] The second trajectory generation module 404 is used to perform secondary visual positioning of the robotic arm and the target object to obtain the second grasping point pose. Based on the second grasping point pose and the second robotic arm pose, it generates a second movement trajectory for the two arms to move closer together and controls the robotic arm to execute the second movement trajectory to calibrate the grasping position.

[0147] The third trajectory generation module 405 is used to control the dexterous hand to complete the grasping of the target object. It generates a third movement trajectory of the two arms according to the preset handling posture and the current posture of the robotic arm, so as to control the robotic arm to move the target object to the preset handling posture according to the third movement trajectory.

[0148] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 8As shown, the electronic device 500 includes a processor 510, a memory 520, and a bus 530.

[0149] The memory 520 stores machine-readable instructions executable by the processor 510. When the electronic device 500 is running, the processor 510 and the memory 520 communicate via the bus 530. When the machine-readable instructions are executed by the processor 510, they can perform the operations described above. Figure 1 The steps of the humanoid robot object grasping and transporting method in the method embodiment shown are specifically implemented in the method embodiment and will not be repeated here.

[0150] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described actions. Figure 1 The steps of the humanoid robot object grasping and transporting method in the method embodiment shown are specifically implemented in the method embodiment and will not be repeated here.

[0151] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0152] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0153] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0154] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0155] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0156] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for grasping and transporting objects using a humanoid robot, characterized in that, include: In response to a movement control command for a humanoid robot, an optimal path is determined to move from an initial pose to a target work area, and the humanoid robot is controlled to move according to the optimal path; When the humanoid robot meets the preset coarse positioning completion conditions, it switches to the near-field fine-tuning mode based on visual feedback. In the near-field fine-tuning mode, the visual pose information of the target object is obtained according to the positioning code. Based on the visual pose information and the preset tolerance range, the micro-motion control quantity of the robot's waist is iteratively generated to adjust the humanoid robot's position until the precise positioning completion conditions are met. The robotic arm and the target object are initially located visually to obtain the first grasping point pose. Based on the first grasping point pose and the first robotic arm pose, a first movement trajectory for the two arms to move closer together is generated. The robotic arm is controlled to execute the first movement trajectory to move to the grasping position. The robotic arm and the target object are subjected to secondary visual positioning to obtain the second grasping point pose. Based on the second grasping point pose and the second robotic arm pose, a second movement trajectory for the two arms to move closer together is generated. The robotic arm is controlled to execute the second movement trajectory to calibrate the grasping position. The dexterous hand is controlled to grasp the target item. A third movement trajectory of the two arms is generated based on the preset handling posture and the current posture of the robotic arm, so as to control the robotic arm to move the target item to the preset handling posture according to the third movement trajectory.

2. The method according to claim 1, characterized in that, The preset tolerance range includes three dimensions. The step of iteratively generating micro-motion control quantities for the robot's waist based on the visual pose information and the preset tolerance range, and using these micro-motion control quantities to adjust the humanoid robot's position, includes: In each round of visual measurement, the visual pose information of the target object in the robot's waist coordinate system is obtained, and the three-dimensional deviation vector between the visual pose information and the preset alignment posture is determined. When the deviation vector of any dimension in the three-dimensional deviation vector exceeds the corresponding preset tolerance range, a micro-motion control command for the humanoid robot in that dimension is generated. After executing the micro-motion control command, the humanoid robot enters a stable waiting state and performs the next round of visual measurement until the deviation vector of any dimension of the three-dimensional deviation vector does not exceed the corresponding preset tolerance range.

3. The method according to claim 1, characterized in that, The target movement path is generated in the following way: The target movement is determined based on the target grasping point pose and the target robotic arm pose. Generate the target movement path based on the target movement amount; The target movement path is aligned with the progress to obtain the target movement trajectory; Wherein, the target movement trajectory is any one of the first movement trajectory, the second movement trajectory, and the third movement trajectory, and the target grasping point pose, the target robotic arm pose, and the target movement amount correspond to the target movement trajectory.

4. The method according to claim 3, characterized in that, The target movement path includes two initial movement paths corresponding to the left and right robotic arms. The step of performing progress alignment processing on the target movement path to obtain the target movement trajectory includes: Determine the number of geometric waypoints for each initial movement path, and determine the movement path with the largest number of geometric waypoints among the two initial movement paths as the base movement path. Determine the initial movement paths other than the base movement path as the specified movement paths. Using the baseline movement path, progress alignment processing is performed on the specified movement path; The initial movement path after progress alignment is processed with unified time parameterization to generate the target movement trajectory after the left and right robotic arms are synchronized in time.

5. The method according to claim 4, characterized in that, The step of performing progress alignment processing on the specified movement path using the reference movement path includes: For each initial movement path, determine the path arc length of each waypoint on that initial movement path; The path arc length of each waypoint is normalized to obtain the progress array of the initial movement path. The progress array includes a first progress array corresponding to the baseline movement path and a second progress array corresponding to the specified movement path. Using the elements in the first progress array as standard progress, linear interpolation is performed on the joint values ​​corresponding to the target progress interval in the joint space using the standard progress to obtain the movement path after progress alignment. The target progress interval is the progress interval determined in the second progress array based on the standard progress.

6. The method according to claim 1, characterized in that, The method further includes: Multiple markers are set on the two dexterous hands of the humanoid robot, including a first marker set at the web of the thumb, a second marker set on the thumb, and a third marker set at the base of the index finger. For two dexterous hands, planar fitting is performed based on multiple marked points on the dexterous hands; Using the first marker point as the origin, the direction pointing to the second marker point as the X-axis direction, and the plane normal vector as the Z-axis direction, the Y-axis direction is determined according to the right-hand rule to determine the hand posture; The current pose of the dexterous hand is determined based on the geometric center of the hand and the hand's posture.

7. The method according to claim 1, characterized in that, Both the first and second gripping point poses include the gripping point poses corresponding to the left and right robotic arms, and are determined in the following way: Visually locate the target object and determine the pose of the target object's center point; Based on the center point pose and the physical dimensions of the target object, the respective gripping point poses of the left and right robotic arms corresponding to the first gripping point pose and the second gripping point pose are determined.

8. The method according to claim 7, characterized in that, The positioning code includes a symmetrically distributed first positioning code and a second positioning code. The step of visually positioning the target item and determining the center point pose of the target item includes: Determine the pose of the first positioning code and the pose of the second positioning code respectively; The center point pose of the target item is determined based on the first positioning code pose and the second positioning code pose.

9. The method according to claim 1, characterized in that, Whether the humanoid robot meets the coarse positioning completion condition is determined by the following method: During the movement of the humanoid robot, the translational error and heading error between the humanoid robot and the target work area are obtained; When the translation error is less than or equal to a preset distance threshold, the heading error is less than or equal to a preset angle threshold, and the continuous holding time is greater than or equal to a preset time threshold, the humanoid robot is determined to meet the coarse positioning completion condition.

10. A humanoid robot object grasping and handling device, characterized in that, include: A motion control module is used to respond to motion control commands for a humanoid robot, determine the optimal path for moving from an initial pose to a target work area, and control the humanoid robot to move according to the optimal path; The precise positioning module is used to switch to a near-field fine-tuning mode based on visual feedback when the humanoid robot meets the preset coarse positioning completion conditions. In the near-field fine-tuning mode, the visual pose information of the target object is obtained according to the positioning code. Based on the visual pose information and the preset tolerance range, the micro-motion control quantity of the robot's waist is iteratively generated to adjust the humanoid robot's position until the precise positioning completion conditions are met. The first trajectory generation module is used to perform initial visual positioning of the robotic arm and the target object, obtain the first grasping point pose, generate a first movement trajectory for the two arms to move closer together based on the first grasping point pose and the first robotic arm pose, and control the robotic arm to execute the first movement trajectory to move to the grasping position. The second trajectory generation module is used to perform secondary visual positioning on the robotic arm and the target object to obtain the second grasping point pose. Based on the second grasping point pose and the second robotic arm pose, a second movement trajectory for the two arms to move closer together is generated, and the robotic arm is controlled to execute the second movement trajectory to calibrate the grasping position. The third trajectory generation module is used to control the dexterous hand to complete the grasping of the target object. It generates a third movement trajectory of the two arms according to the preset handling posture and the current posture of the robotic arm, so as to control the robotic arm to move the target object to the preset handling posture according to the third movement trajectory.

Images