A palm-based binocular vision target object pose prediction method and prediction system

Abstract

The application relates to a palm heart binocular vision-based target object pose prediction method and a prediction system, which comprises the following steps: acquiring an image frame sequence collected by a binocular camera fixed at the end effector of a six-degree-of-freedom robot mechanical arm in real time; judging whether the current image frame is the first frame or a tracking loss state; if yes, calling a visual semantic large model to perform semantic segmentation and geometric verification on the target object to determine an initial pose; if no, obtaining a dense depth image and a confidence weight through a lightweight binocular depth matching network, combining motion prior constraints to optimize the pose to obtain an optimized pose; based on the solved target pose, combining a fixed external parameter matrix of the binocular camera center relative to the end effector coordinate system and the real-time pose of the end effector in the world coordinate system, obtaining the absolute pose of the target object in the world coordinate system; inputting the absolute pose into a servo controller to generate a control instruction and guiding the six-degree-of-freedom robot to perform corresponding grabbing operations.

Description

Technical Field

[0001] This invention relates to the field of robot vision and control technology, and in particular to a method and system for predicting the pose of a target object based on palm-based binocular vision. Background Technology

[0002] In the field of autonomous robot operation, vision-based six-degree-of-freedom target pose estimation and servo grasping are key technologies for realizing tasks such as intelligent manufacturing and logistics sorting.

[0003] Most existing technical solutions employ monocular or binocular cameras fixed outside the scene for image acquisition, combined with traditional feature extraction or deep learning models to complete target recognition and pose calculation. However, in the palm vision configuration of a robotic arm, when the dexterous hand of the robotic arm approaches the target object, the dexterous fingers of the robotic arm will inevitably cause severe occlusion of the target object, making traditional methods based on feature points or pure point cloud registration prone to getting stuck in local optima or even failing outright. Furthermore, the external fixed camera is easily occluded by the robotic arm itself and the environment, making it difficult to adaptively acquire high-resolution near-field images and easily introducing cumulative drift errors, resulting in difficulty in accurately aligning the visual perception space with the robot's execution space. In addition, existing methods often rely on manually designed features or specific object templates in the first frame initialization stage, resulting in limited generalization ability and a high initialization failure rate when facing complex backgrounds, partial occlusion, and scenes with variable poses. Moreover, the lack of effective motion prior constraints and adaptive occlusion removal mechanisms under dynamic conditions leads to uneven pose trajectories and insufficient tracking stability, making it difficult to meet the stringent requirements of real-time performance and robustness for dynamic grasping. Summary of the Invention

[0004] This invention provides a target object pose prediction method and system based on palm-based binocular vision, aiming to solve at least one of the technical problems existing in the prior art.

[0005] The technical solution of this invention is a target object pose prediction method based on palm-based binocular vision, which includes: Acquire a sequence of image frames in real time from a binocular camera fixed to the palm position of the end effector of a six-degree-of-freedom robot arm; Determine whether the current image frame is the first frame or if tracking is lost. If the current image frame is the first frame or the tracking is lost, the first frame semantic initialization branch is executed, and the visual semantic big model is called to perform semantic segmentation and geometric verification of the target object to determine the initial pose. If the current image frame is a continuously tracked frame, then the continuous pose tracking branch is executed. The dense depth image and confidence weights are obtained through a lightweight binocular depth matching network, and the pose is optimized by combining motion prior constraints. Based on the target pose calculated by the semantic initialization branch of the first frame or the continuous pose tracking branch, and combined with the fixed extrinsic parameter matrix of the binocular camera center relative to the coordinate system of the end effector of the six-degree-of-freedom robot arm and the real-time pose of the end effector of the six-degree-of-freedom robot arm in the world coordinate system, the absolute pose of the target object in the world coordinate system is obtained through coordinate transformation. The absolute pose is input to the servo controller to generate control commands, and the six-degree-of-freedom robot is guided to perform the corresponding grasping operation based on the control commands.

[0006] According to some embodiments of the present invention, the step of executing the first frame semantic initialization branch and calling the visual semantic large model to perform semantic segmentation and geometric verification of the target object to determine the initial pose includes: Input the left eye image of the current image frame into the visual semantic big model, and output the two-dimensional semantic mask of the target object by fusing the text task instructions. By combining the depth map generated by the binocular depth network, the geometric consistency of the two-dimensional semantic mask is checked, and mis-segmented regions with depth continuity differences exceeding a set threshold at the edge of the two-dimensional semantic mask are removed to obtain the valid point cloud after verification. Based on the effective point cloud calculation and initialization of the target object's six-degree-of-freedom pose in the camera coordinate system, the initial pose of the target object is calculated using principal component analysis or bounding box algorithm.

[0007] According to some embodiments of the present invention, it further includes: During the task initiation phase, the six-degree-of-freedom robot arm is controlled to move above the work area, and the binocular camera at the palm position of the end effector of the six-degree-of-freedom robot arm is used to collect left and right eye images in the work area in real time at a preset frequency. The fixed extrinsic parameter matrix of the binocular camera center relative to the palm coordinate system of the end effector of the six-DOF robot arm, obtained through offline calibration, is pre-loaded. Based on the fixed extrinsic parameter matrix, the robot controller provides real-time feedback on the current joint state of the end effector of the six-degree-of-freedom robot arm via the bus based on the left and right eye images, and calculates the real-time pose of the end effector of the six-degree-of-freedom robot arm in the world coordinate system.

[0008] According to some embodiments of the present invention, the execution of the continuous pose tracking branch, which obtains dense depth images and confidence weights through a lightweight binocular depth matching network, and combines motion prior constraints to perform pose optimization to obtain an optimized pose, includes: Control the large visual semantic model to enter sleep mode; A lightweight stereo matching network is used to perform stereo matching between the left and right images of the current image frame to generate a dense depth map and confidence weights. The dense depth map and the confidence weight are used as geometric consistency constraints and input into the pose tracking optimizer based on the target 3D model to obtain the optimized pose.

[0009] According to some embodiments of the present invention, the step of inputting the dense depth map and the confidence weight as geometric consistency constraints into a pose tracking optimizer based on the target 3D model to obtain the optimized pose includes: The dense depth map and the confidence weight are used as geometric consistency constraints. The target pose of the previous image frame, the fixed extrinsic parameter matrix between the offline calibrated camera center and the end effector of the six-degree-of-freedom robot arm, and the end effector pose change matrix of adjacent frames obtained by robot forward kinematics are extracted. The upper limit of the maximum reasonable change range of the target pose is derived to limit the pose search radius of the current image frame and construct the motion prior constraint of the target pose. The target 3D model is virtually rendered according to the target pose of the previous image frame to generate a predicted depth map. The pixel-level residual between the dense depth map and the predicted depth map is calculated. Pixels with residuals exceeding the depth tolerance threshold are identified as occluded areas and the pixel optimization weight is reset to zero. A nonlinear cost function containing a robust kernel function is constructed, and an iterative optimization algorithm is used to solve for the optimal pose increment, thereby obtaining the optimized pose of the target object in the camera coordinate system in the current image frame.

[0010] According to some embodiments of the present invention, the geometric consistency check of the two-dimensional semantic mask is performed on the depth map generated by the binocular depth network, and mis-segmented regions with depth continuity differences exceeding a set threshold at the edges of the two-dimensional semantic mask are removed to obtain a valid point cloud after verification, including: The two-dimensional semantic mask is superimposed on the depth map of the first frame, and the local depth gradient of the pixels within the two-dimensional semantic mask is calculated. When the depth variance or normal vector of a certain sub-region suddenly exceeds the set threshold, it is determined that the spatial continuity of the sub-region is poor or there is a mis-segmented region, and the geometric consistency verification of the sub-region fails. Sub-regions that fail the geometric consistency check are removed from the pose initialization calculation to obtain the valid point cloud after verification.

[0011] According to some embodiments of the present invention, the target pose calculated based on the semantic initialization branch of the first frame or the continuous pose tracking branch, combined with the fixed extrinsic parameter matrix of the binocular camera center relative to the coordinate system of the end effector of the six-degree-of-freedom robot arm and the real-time pose of the end effector of the six-degree-of-freedom robot arm in the world coordinate system, is transformed to obtain the absolute pose of the target object in the world coordinate system, which is achieved by the following calculation formula: T_W_O = T_W_E * T_E_C * T_C_O; Where T_W_O is the absolute pose of the target object in the world coordinate system, T_W_E is the real-time pose of the end effector of the six-DOF robot arm in the world coordinate system, T_E_C is the fixed extrinsic parameter matrix of the binocular camera center relative to the coordinate system of the end effector of the six-DOF robot arm, and T_C_O is the target pose of the target object in the camera coordinate system of the current image frame.

[0012] According to some embodiments of the present invention, the step of inputting the absolute pose to the servo controller to generate control commands, and guiding the six-degree-of-freedom robot to perform the corresponding grasping operation based on the control commands, is implemented through a control law transfer function model, which is as follows: q_dot = λ * J_inv * e(t); Where q_dot is the joint velocity control command of the six-degree-of-freedom robot arm and dexterous hand, λ is the proportional control gain matrix; J_inv is the generalized Jacobian pseudo-inverse containing the fixed extrinsic parameter matrices of the end effector coordinate system of the six-degree-of-freedom robot arm and the binocular camera center relative to the end effector coordinate system of the six-degree-of-freedom robot arm; e(t) is the real-time error between the current pose of the target object and the desired grasping pose.

[0013] This invention also provides a target object pose prediction system based on palm-based binocular vision, used to execute the target object pose prediction method based on palm-based binocular vision described in the above embodiments, comprising: A binocular camera is installed in the palm of the end effector of a six-degree-of-freedom robot arm to acquire image frame sequences containing left and right eye images in the working area in real time. The image frame determination module is used to determine whether the current image frame is the first frame or a tracking loss state; The first frame semantic initialization module is used to execute the first frame semantic initialization branch when the current image frame is the first frame or when the tracking is lost. It calls the visual semantic big model to perform semantic segmentation and geometric verification of the target object to determine the initial pose. The continuous pose tracking module is used to execute the continuous pose tracking branch in continuous tracking frames. It obtains dense depth images and confidence weights through a lightweight binocular depth matching network, and combines motion prior constraints to optimize the pose and obtain the optimized pose. The coordinate transformation and pose output module is used to calculate the target pose based on the semantic initialization branch of the first frame or the continuous pose tracking branch, and combine the fixed extrinsic parameter matrix of the binocular camera center relative to the coordinate system of the end effector of the six-degree-of-freedom robot arm and the real-time pose of the end effector of the six-degree-of-freedom robot arm in the world coordinate system to obtain the absolute pose of the target object in the world coordinate system through coordinate transformation. The visual servo control module is used to input the absolute pose to the servo controller to generate control commands, and guide the six-degree-of-freedom robot to perform corresponding grasping operations based on the control commands; and, The control module is used to coordinate the decoupled operation of the first frame semantic initialization module and the continuous pose tracking module.

[0014] According to some embodiments of the present invention, the large visual semantic model includes a multimodal coding layer and a cross-attention decoding layer. The multimodal coding layer extracts features from text task instructions and features from the left eye image based on CLIP text encoder. The cross-attention decoding layer fuses text and left eye image features to output a two-dimensional semantic mask.

[0015] The present invention also relates to a computer device, including a memory and a processor, wherein the processor performs the above-described method when executing a computer program stored in the memory.

[0016] The present invention also relates to a computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement the above-described method.

[0017] The target object pose prediction method and prediction method based on palm-based binocular vision provided in this invention embodiment have at least one of the following advantages or beneficial effects: After acquiring the image frame sequence, it is first determined whether the current image frame is the first frame or in a tracking loss state. If so, the first frame semantic initialization branch is executed. The visual semantic large model completes high-precision initialization through semantic segmentation and geometric verification, effectively adapting to complex backgrounds, partial occlusion, and scenes with variable poses, reducing the initialization failure rate. If not, it is determined that the current image frame is in the continuous tracking stage, and the continuous pose tracking branch is executed. The lightweight binocular depth matching network is called to output dense depth and confidence weights, and the pose is optimized by combining motion prior constraints. The three-dimensional structural information of the object surface is fully utilized, and noise interference in low-confidence areas is automatically suppressed to ensure the smoothness and continuity of the pose trajectory, improve the robustness of pose estimation, and significantly enhance the system's anti-interference ability and tracking stability under dynamic conditions.

[0018] By using a fixed extrinsic parameter matrix between a binocular camera and the end effector of a six-degree-of-freedom robot arm, and combining the end pose fed back by the robot arm in real time, coordinate transformation is performed to obtain the absolute pose of the target object in the world coordinate system. This design utilizes the hand-eye calibration accuracy of rigid connection to automatically compensate for the perspective changes caused by robot movement, eliminate the coordinate alignment problem, and achieve accurate calculation of absolute pose.

[0019] Based on real-time updated six-degree-of-freedom absolute pose, the servo controller dynamically optimizes the robotic arm's motion trajectory and end effector posture to achieve optimal grasping of irregularly shaped objects in all postures. At the same time, it forms a visual servo closed loop of "perception-decision-execution", which corrects trajectory deviations in real time through continuous frame feedback, effectively overcoming model errors and load disturbances in open-loop control, and improving the grasping success rate and operation accuracy in dynamic scenarios.

[0020] Furthermore, additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0021] Figure 1 This is a flowchart of the target object pose prediction method based on palm-based binocular vision provided in the embodiments of the present invention. Figure 2 This is a detailed flowchart of step S300 in the target object pose prediction method based on palm binocular vision provided in the embodiments of the present invention. Figure 3 This is a detailed flowchart of a target object pose prediction method based on palm-based binocular vision provided in an embodiment of the present invention; Figure 4 This is a detailed flowchart of step S400 in the target object pose prediction method based on palm binocular vision provided in the embodiments of the present invention. Figure 5 This is a detailed flowchart of step S430 in the target object pose prediction method based on palm binocular vision provided in the embodiments of the present invention. Figure 6 This is a detailed flowchart of step S320 in the target object pose prediction method based on palm binocular vision provided in the embodiments of the present invention. Figure 7 This is a flowchart illustrating a specific implementation method of the target object pose prediction method based on palm-based binocular vision provided in this invention. Detailed Implementation

[0022] The following will provide a clear and complete description of the concept, specific structure, and technical effects of the present invention in conjunction with the embodiments and accompanying drawings, so as to fully understand the purpose, solution, and effects of the present invention.

[0023] It should be noted that, unless otherwise specified, when a feature is referred to as "fixed" or "connected" to another feature, it can be directly fixed or connected to the other feature, or indirectly fixed or connected to the other feature. The singular forms "a," "described," and "the" used herein are also intended to include the plural forms, unless the context clearly indicates otherwise. Furthermore, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this specification is for the purpose of describing particular embodiments only and not for limiting the invention. The term "and / or" as used herein includes any combination of one or more of the associated listed items.

[0024] It should be understood that although the terms first, second, third, etc., may be used to describe various elements in this disclosure, these elements should not be limited to these terms. These terms are used only to distinguish elements of the same type from one another. For example, a first element may also be referred to as a second element without departing from the scope of this disclosure, and similarly, a second element may also be referred to as a first element. Any and all instances or exemplary language (“e.g.,” “such as,” etc.) provided herein are intended only to better illustrate embodiments of the invention and, unless otherwise required, do not impose a limitation on the scope of the invention.

[0025] Reference Figures 1 to 7 As shown, the target object pose prediction method and prediction system based on palm-based binocular vision provided in the embodiments of the present invention will be further elaborated.

[0026] Reference Figure 1 As shown, Figure 1 This is a flowchart illustrating the overall process of a target object pose prediction method based on palm-based binocular vision provided in this invention. The method includes, but is not limited to, steps S100 to S600. Specifically, S100: Acquire a sequence of image frames in real time from a binocular camera fixed to the palm position of the end effector of a six-degree-of-freedom robot arm; S200: Determine whether the current image frame is the first frame or the tracking is lost; S300: If the current image frame is the first frame or the tracking is lost, execute the first frame semantic initialization branch, call the visual semantic big model to perform semantic segmentation and geometric verification of the target object to determine the initial pose; S400: If the current image frame is a continuously tracked frame, the continuous pose tracking branch is executed. The dense depth image and confidence weights are obtained through a lightweight binocular depth matching network, and the pose is optimized by combining motion prior constraints. S500: Based on the target pose calculated by the semantic initialization branch of the first frame or the continuous pose tracking branch, combined with the fixed extrinsic parameter matrix of the binocular camera center relative to the coordinate system of the end effector of the six-degree-of-freedom robot arm and the real-time pose of the end effector of the six-degree-of-freedom robot arm in the world coordinate system, the absolute pose of the target object in the world coordinate system is obtained through coordinate transformation. S600: Inputs the absolute pose to the servo controller to generate control commands, and guides the six-degree-of-freedom robot to perform the corresponding grasping operation based on the control commands.

[0027] In this embodiment of the invention, after acquiring the image frame sequence, it is determined whether the current image frame is the first frame or in a tracking loss state. If so, the first frame semantic initialization branch is executed. The visual semantic large model completes high-precision initialization through semantic segmentation and geometric verification, effectively adapting to complex backgrounds, partial occlusion, and scenes with varying poses, reducing the initialization failure rate. If not, it is determined that the current image frame is in the continuous tracking stage, and the continuous pose tracking branch is executed. The lightweight binocular depth matching network is called to output dense depth and confidence weights, and the pose is optimized by combining motion prior constraints. The three-dimensional structural information of the object surface is fully utilized, and noise interference in low-confidence areas is automatically suppressed to ensure the smoothness and continuity of the pose trajectory, improve the robustness of pose estimation, and significantly enhance the system's anti-interference ability and tracking stability under dynamic conditions.

[0028] By using a fixed extrinsic parameter matrix between a binocular camera and the end effector of a six-DOF robotic arm, and combining the end pose fed back by the robotic arm in real time, coordinate transformation is performed to convert the relative pose of the camera center into the absolute pose of the target object in the world coordinate system. This design utilizes the hand-eye calibration accuracy of rigid connection to automatically compensate for the perspective changes caused by robot movement, providing a unified and unambiguous spatial reference for downstream control, eliminating the problem of multi-sensor coordinate alignment, and achieving accurate calculation of absolute pose.

[0029] Based on real-time updated six-degree-of-freedom absolute pose, the servo controller dynamically optimizes the robotic arm's motion trajectory and end effector posture to achieve optimal grasping of irregularly shaped objects in all postures. At the same time, it forms a visual servo closed loop of "perception-decision-execution", which corrects trajectory deviations in real time through continuous frame feedback, effectively overcoming model errors and load disturbances in open-loop control, and improving the grasping success rate and operation accuracy in dynamic scenarios.

[0030] Furthermore, it is understood that the target object pose prediction method based on palm-based binocular vision provided in this embodiment of the invention avoids repeated inference of large models in consecutive frames through a decoupling mechanism between the semantic initialization of the first frame and pose tracking in consecutive frames. Tests show that the system frame rate jumps from the traditional <5 FPS to >60 FPS, and the overall computational overhead is reduced by more than 80%, perfectly adapting to resource-constrained edge computing platforms for robot control. The geometric consistency verification mechanism for semantics and depth in the first frame effectively filters out mis-segmented background regions that may be generated by large models, such as "semantically correct but geometrically suspended," eliminating the risk of pose initialization failure from the source and improving initialization robustness. Moreover, pose optimization based on semantic segmentation and geometric verification of the target object combined with motion prior constraints effectively reduces the solution space.

[0031] Reference Figure 2 and Figure 7 As shown, Figure 7 This is a flowchart illustrating a specific implementation method of the target object pose prediction method based on palm-based binocular vision provided in this invention. Figure 2 This is a detailed flowchart of step S300 in the target object pose prediction method based on palm-based binocular vision provided in this embodiment of the invention. Step S300 includes, but is not limited to, steps S310 to S330. Specifically, S310: Input the left eye image of the current image frame into the visual semantic large model, fuse the text task instructions, and output the two-dimensional semantic mask of the target object; S320: Combine the depth map generated by the binocular depth network to perform geometric consistency verification on the two-dimensional semantic mask, remove mis-segmented regions at the edge of the two-dimensional semantic mask whose depth continuity difference exceeds a set threshold, and obtain the verified valid point cloud. S330: Based on effective point cloud calculation, initialize the six-degree-of-freedom pose of the target object in the camera coordinate system, and calculate the initial pose of the target object through principal component analysis or bounding box algorithm.

[0032] In this embodiment of the invention, during the execution of the first-frame semantic initialization branch, which calls the visual semantic large model to perform semantic segmentation and geometric verification of the target object to determine the initial pose, the first-frame semantic initialization branch significantly improves the accuracy and robustness of the target object's initial pose estimation through the collaborative mechanism of the visual semantic large model and geometric verification. The left-eye image is input into the visual semantic large model and fused with text task instructions to output a two-dimensional semantic mask of the target object. This method leverages the powerful generalization ability of the large model to effectively adapt to scenarios with diverse target categories, complex backgrounds, and varying illumination, reducing reliance on manually designed features and pre-trained templates.

[0033] Based on this, the depth map generated by the binocular depth network is used to perform geometric consistency verification on the two-dimensional semantic mask. By detecting the difference in depth continuity at the edge of the mask and removing mis-segmented areas that exceed the set threshold, the valid point cloud after verification is obtained. This automatically filters out false edge detection pixels caused by background interference and object adhesion in semantic segmentation, avoids the pollution of pose calculation by erroneous 3D point clouds, and improves the quality and geometric reliability of point clouds.

[0034] Furthermore, based on the effective point cloud, the six-degree-of-freedom pose of the target object in the camera coordinate system is calculated, and the initial pose is determined by principal component analysis or bounding box algorithm. By making full use of the spatial distribution characteristics of the effective point cloud, the three-dimensional position and orientation of the object are quickly estimated by geometric dimensionality reduction or outer contour fitting. While ensuring computational efficiency, reliable pose priors are provided for subsequent continuous tracking branches, shortening the system startup response time and reducing the difficulty of re-initialization after tracking loss.

[0035] In this embodiment of the invention, the Visual Semantic Large Model (VLM) includes: Network structure: The large model contains multimodal encoding layers (such as CLIP-based Text / Vision Encoders) and cross-attention decoding layers.

[0036] Semantic output: The two-dimensional semantic mask of the target object is fused with the text task instruction output (denoted as M_semantic).

[0037] In one embodiment, when the first frame image (first frame) is acquired, the first frame semantic initialization branch is initiated, and the system's built-in visual semantic large model (such as Grounded-SAM) inputs the text prompt "orange square". The visual semantic large model accurately identifies and generates a two-dimensional pixel-level mask of the workpiece in the left eye image. To prevent the two-dimensional pixel-level mask from overflowing into the background or other interference objects, the system performs geometric consistency verification. Specifically, it extracts the initial depth information corresponding to the two-dimensional pixel-level mask region. If the difference between the depth value at the edge of the two-dimensional pixel-level mask and the depth continuity of the workpiece body exceeds a set threshold (for example, the set threshold is 2 cm), it is determined to be a background point, and the range of the two-dimensional pixel-level mask is automatically reduced. Subsequently, the axial direction and center position of the workpiece are extracted using principal component analysis (PCA) based on the verified effective point cloud, completing the coarse initialization of the six-degree-of-freedom pose of the first frame.

[0038] In one embodiment of the present invention, the specific calculation process of the initial pose of the target object is as follows: The relevant parameters are: horizontal focal length of the binocular camera fx = 652.4381 pixels, vertical focal length fy = 653.3026 pixels, X-axis coordinate of the camera principal point cx = 614.3621, Y-axis coordinate of the camera principal point cy = 331.6342, and binocular baseline length B = 0.06m; The visual semantic large model outputs the center pixel X coordinate of the target object's 2D semantic mask as u = 702 and the center pixel Y coordinate as v = 358. The binocular depth matching network outputs the disparity value corresponding to this point as d = 42 pixels. According to the binocular ranging formula Z = fx * B / d, the target depth Z = 652.4381 * 0.06 / 42 = 0.932m is calculated; where Z is the depth distance of the target object in the camera coordinate system, fx is the horizontal focal length of the binocular camera, B is the binocular baseline length, and d is the disparity value.

[0039] Combining the pinhole imaging model, the three-dimensional position coordinates of the target object in the camera coordinate system are calculated as follows: X = (u - cx) * Z / fx = (702 - 614.3621) * 0.932 / 652.4381 = 0.125m; Y = (v - cy) * Z / fy = (358 - 331.6342) * 0.932 / 653.3026 = 0.038m, where X is the X-axis coordinate of the target object in the camera coordinate system, Y is the Y-axis coordinate of the target object in the camera coordinate system, u and v are the X and Y coordinates of the center pixel of the two-dimensional semantic mask, cx and cy are the X and Y coordinates of the principal point of the camera, Z is the target depth, and fx and fy are the horizontal and vertical focal lengths of the camera, respectively. Thus, the position vector of the target object in the current image frame in the camera coordinate system is obtained as P_cam = [0.125, 0.038, ... [0.932], in meters, where P_cam is the three-dimensional translational position vector of the target object in the camera coordinate system. Then, the principal direction estimated by the principal component analysis of the effective point cloud is combined to construct the initial pose of the target object in the camera coordinate system.

[0040] Reference Figure 3 As shown, Figure 3 This is a detailed flowchart of a target object pose prediction method based on palm-based binocular vision provided in an embodiment of the present invention. The target object pose prediction method based on palm-based binocular vision also includes, but is not limited to, steps S10 to S30. Specifically, S10: During the task start-up phase, control the six-degree-of-freedom robot arm to move above the work area, and use the binocular camera at the palm position of the end effector of the six-degree-of-freedom robot arm to collect left and right eye images in the work area in real time at a preset frequency. S20: Preload the fixed extrinsic parameter matrix of the binocular camera center relative to the palm coordinate system of the end effector of the six-DOF robot arm, obtained from offline calibration; S30: Based on a fixed extrinsic parameter matrix, the robot controller provides real-time feedback on the current joint state of the end effector of the six-degree-of-freedom robot arm through the bus based on the left and right eye images, and calculates the real-time pose of the end effector of the six-degree-of-freedom robot arm in the world coordinate system.

[0041] Understandably, the target object pose prediction method based on palm-based binocular vision also includes: during the task initiation phase, controlling the six-DOF robot arm to move above the work area, and the palm-based binocular camera to acquire left and right eye images in the work area in real time at a preset frequency (e.g., 30Hz); by actively adjusting the camera angle through the robot arm, adaptively acquiring near-field high-resolution images of the target area, effectively avoiding the field-of-view occlusion and insufficient resolution problems of fixed external cameras, and improving the targeting and temporal continuity of image acquisition. At this time, the system pre-loads the fixed extrinsic parameter matrix T_E_C of the camera center relative to the palm coordinate system of the end effector of the six-DOF robot arm obtained through offline calibration, avoiding the cumulative error introduced by dynamic calibration, and ensuring the high accuracy and stability of the spatial transformation relationship between the camera and the end effector.

[0042] Simultaneously, based on the fixed extrinsic parameter matrix T_E_C, and through a bus, the robot controller calculates the real-time pose T_W_E of the end effector of the six-DOF robot arm in the world coordinate system by providing real-time feedback on the current joint states of the end effector based on the acquired left and right eye images. These two sets of hardware parameters constitute the geometric reference for subsequent absolute pose calculation, achieving millisecond-level synchronization between visual perception and robot kinematics. This provides a unified, drift-free spatial reference for subsequent target absolute pose calculation, ensuring the unity of visual perception and physical execution space, and significantly reducing hand-eye coordination errors.

[0043] Reference Figure 4 and Figure 7 As shown, Figure 4 This is a detailed flowchart of step S400 in the target object pose prediction method based on palm-based binocular vision provided in this embodiment of the invention. Step S400 includes, but is not limited to, steps S410 to S430. Specifically, S410: Controls the large visual semantic model to enter sleep mode; S420: Uses a lightweight stereo matching network to perform stereo matching between the left and right images of the current image frame, generating a dense depth map and confidence weights; S430: The dense depth map and confidence weights are used as geometric consistency constraints and input into the pose tracking optimizer based on the target 3D model to obtain the optimized pose.

[0044] In some embodiments of the present invention, a continuous pose tracking branch is executed, which obtains dense depth images and confidence weights through a lightweight binocular depth matching network, and combines motion prior constraints to perform pose optimization to obtain an optimized pose, specifically including: Starting from the second frame (i.e., if the current image frame is a continuously tracked frame), the visual semantic large model is controlled to enter a sleep mode to save computing power. This significantly releases computing resources, reduces system power consumption, and allows subsequent processing to focus on efficient pure geometric pose optimization, meeting the stringent real-time requirements of robot dynamic grasping. When executing the continuous pose tracking branch, a lightweight stereo matching network (including a multi-scale feature extraction module, a 3D cost volume construction module, and a disparity regression module) is used to perform fast stereo matching on the binocular images. While ensuring the frame rate, it generates a dense depth map D_k and its confidence weight W for the current image frame. Compared to sparse feature matching, the dense depth map provides a complete geometric description of the target surface, effectively dealing with low-texture or high-reflectivity areas, while the confidence weight can adaptively distinguish between reliable and unreliable depth pixels, suppressing noise interference.

[0045] Subsequently, the density depth and confidence weights are used as geometric consistency constraints input to the pose tracking optimizer based on the target 3D model. By registering and optimizing the observed point cloud with the model surface, high-precision iterative refinement of the pose is achieved. This technical solution significantly improves the system's computational efficiency and response speed while ensuring tracking continuity and pose accuracy.

[0046] Reference Figure 5 and Figure 7 As shown, Figure 5 This is a detailed flowchart of step S430 in the target object pose prediction method based on palm-based binocular vision provided in this embodiment of the invention. Step S430 includes, but is not limited to, steps S431 to S433. Specifically, S431: The dense depth map and confidence weights are used as geometric consistency constraints. The fixed extrinsic parameter matrix between the target pose of the previous image frame, the offline calibrated camera center and the end effector of the six-degree-of-freedom robot arm, and the end effector pose change matrix of adjacent frames obtained by robot forward kinematics are extracted. The upper limit of the maximum reasonable change range of the target pose is derived to limit the pose search radius of the current image frame and construct the motion prior constraint of the target pose. S432: Virtually render the target 3D model according to the target pose of the previous image frame to generate a predicted depth map, calculate the pixel-level residual between the dense depth map and the predicted depth map, and determine the pixels with residuals exceeding the depth tolerance threshold as occluded areas and reset the pixel optimization weight to zero. S433: Construct a nonlinear cost function containing a robust kernel function, and use an iterative optimization algorithm to solve for the optimal pose increment, thereby obtaining the optimized pose of the target object in the camera coordinate system in the current image frame.

[0047] In some embodiments of the present invention, the dense depth map and the confidence weight are used as geometric consistency constraints and input into a pose tracking optimizer based on the target 3D model to obtain an optimized pose, including: The dense depth map D_k and confidence weight W are used as geometric consistency constraints. Motion prior constraints are constructed by fusing robot kinematic information. The maximum reasonable range of target pose change is derived by using the end effector pose change matrix of adjacent frames and the fixed extrinsic parameter matrix. This effectively limits the pose search radius of the current frame, significantly reduces the optimization space and accelerates convergence, avoids getting trapped in local optima, and improves the stability and real-time performance of continuous tracking.

[0048] The target pose of the previous image frame is extracted, and the target 3D model is rendered into a predicted depth map according to the pose of the previous image frame. The predicted depth map D_pred is virtually rendered and compared with the observed dense depth map D_k at the pixel level. In the cost function of the nonlinear iterative optimization process, pixels with residuals exceeding the threshold are automatically identified as occluded regions and their optimization weights are set to zero. This effectively suppresses the influence of scene occlusion, dynamic interference and mismatched points on pose estimation, and eliminates interference from foreground occluders such as robotic arms.

[0049] In one embodiment, the target 3D model (such as a target 3D CAD model) is rendered according to the pose T_{k-1} of the previous image frame to generate a predicted depth map D_pred. The pixel-level residual e between the dense depth map D_k and the predicted depth map D_pred is calculated; when the residual e exceeds the depth tolerance threshold (indicating that the pixel is occluded by a dexterous hand finger), the optimization weight of that pixel is reset to 0. A nonlinear cost function containing a robust kernel function is constructed, and the optimal pose increment is iteratively solved using the LM (Levenberg-Marquardt) algorithm to obtain the current image frame pose T_k. The optimized pose T_C_O of the target object in the camera coordinate system in the current image frame is obtained through coordinate transformation.

[0050] By introducing residual comparison between the predicted depth map and the dense depth map, pixel-level adaptive occlusion culling is achieved, avoiding direct interference of depth noise on pose calculation when the six-DOF robot arm approaches the target object. Solving for the optimal pose increment through an iterative optimization algorithm reduces the sensitivity to outlier observations, significantly improving the accuracy and robustness of pose estimation under complex conditions while ensuring computational efficiency.

[0051] In one embodiment of the present invention, a priori constraints on the target pose are introduced. Based on the motion velocity of the dexterous hand of the six-DOF robot arm at the previous moment, the search radius of the target object's pose at the current moment is limited. When the dexterous hand fingers of the six-DOF robot arm gradually approach and occlude the workpiece handle, the known target 3D model is virtually rendered according to the predicted pose. In the anti-occlusion processing, the rendered predicted depth map is compared with the actual dense depth map, and the pixel-level residual between the dense depth map and the predicted depth map is calculated. If the dense depth map at a certain pixel is significantly smaller than the predicted depth map rendered by the target 3D model, the pixel with a residual exceeding the depth tolerance threshold (e.g., the depth tolerance threshold is 1 cm) is identified as an occlusion area, and this location is identified as a foreground occluder (i.e., the dexterous hand fingers). The pixel optimization weight is immediately reset to zero. This mechanism ensures that the pose tracker only uses the surface features of the unoccluded target object for iterative optimization, avoiding pose drift caused by finger interference.

[0052] It is understandable that the prior constraints for constructing the target pose are specifically as follows: using the real-time state of the joint feedback of the dexterous hand of the six-degree-of-freedom robot arm to solve the forward kinematics, the pose change matrix of the end effector between adjacent frames is obtained; combined with the fixed extrinsic parameter matrix T_E_C of the binocular camera center relative to the palm coordinate system of the end effector of the six-degree-of-freedom robot arm, the maximum search radius of the target pose is calculated, and the pose solution space of the dynamic constraint optimizer is determined.

[0053] If the current image frame is a continuously tracked frame, a continuous pose tracking branch is executed. Using the pose T_{k-1} of the previous image frame as a reference, in one embodiment, the motion increments of the end effector of the six-DOF robot arm are Δx = 0.012m, Δy = -0.004m, Δz = 0.003m, and Δyaw = 2.5 degrees. A predicted pose T_pred is generated based on the target 3D model. Here, T_{k-1} is the target object pose of the previous image frame, Δx, Δy, and Δz are the translation increments of the end effector in the three spatial dimensions, Δyaw is the yaw angle increment, and T_pred is the predicted pose generated by combining motion priors. The binocular depth network outputs the current frame depth residual Edepth = 0.007m, the point cloud ICP registration error Eicp = 0.0038, and the motion continuity constraint error Emotion = 0.0025.

[0054] Construct the pose optimization objective function E = λ1 * Edepth + λ2 * Eicp + λ3 * Emotion Where E is the current total error of pose optimization, Edepth is the current frame depth residual, Eicp is the point cloud ICP registration error, Emotion is the motion continuity constraint error, and λ1, λ2, and λ3 are the proportional control gain coefficients for the corresponding error terms. Taking the proportional control gain coefficients λ1 = 0.5, λ2 = 0.3, and λ3 = 0.2, the current total error E = 0.5 * 0.007 + 0.3 * 0.0038 + 0.2 * 0.0025 = 0.00514.

[0055] The Gauss-Newton optimization algorithm is used to iteratively update the target pose to minimize the error E. The final optimized pose of the target object in the current image frame in the camera coordinate system (target pose T_C_O) is obtained, with translations of x = 0.136m, y = 0.034m, z = 0.928m, and Euler angles of yaw = 2.1 degrees, pitch = -0.6 degrees, and roll = 0.4 degrees. Here, x, y, and z are the three-dimensional translation components in the camera coordinate system, and yaw, pitch, and roll are the yaw, pitch, and roll angles, respectively.

[0056] Reference Figure 6 As shown, Figure 6 This is a detailed flowchart of step S320 in the target object pose prediction method based on palm-based binocular vision provided in this embodiment of the invention. Step S320 includes, but is not limited to, steps S321 to S323. Specifically, S321: Overlay the two-dimensional semantic mask with the depth map of the first frame and calculate the local depth gradient of the pixels within the two-dimensional semantic mask; S322: When the depth variance or normal vector of a certain sub-region suddenly exceeds the set threshold, it is determined that the spatial continuity of the sub-region is poor or there is a mis-segmented region, and the geometric consistency verification of the sub-region fails. S323: Remove sub-regions that fail the geometric consistency check from the pose initialization calculation to obtain the valid point cloud after verification.

[0057] In one embodiment of the present invention, the local depth gradient within the two-dimensional semantic mask region is calculated by combining the first-frame depth map D generated by the binocular depth network. If the depth variance or normal vector abruptly exceeds a set threshold, the sub-region is determined to have been mis-segmented or has background overflow and is removed. Using the verified effective point cloud, the axis direction and center position of the target object are extracted by the bounding box algorithm or principal component analysis (PCA), completing the coarse initialization of the six-degree-of-freedom pose of the first frame. It should be noted that the principal component analysis algorithm or bounding box algorithm used in the present invention are conventional techniques for spatial feature extraction in the field. The first-frame large model semantic initialization mechanism based on palm-sized binocular vision integrates the two-dimensional semantic mask output by the visual semantic large model with the geometric consistency verification of the binocular depth matching network, providing a set of effective point clouds that filter out strong occlusion and background noise for the conventional principal component analysis algorithm. This solves the technical problem of easy divergence in target object pose initialization under complex close-range conditions from the source.

[0058] Understandably, overlaying a 2D semantic mask with the first frame's depth map and calculating the local depth gradient allows for quantitative verification of the semantic segmentation results from a 3D geometric perspective, compensating for the limitations of purely visual semantic models in spatial geometric understanding. By employing dual criteria of depth variance and normal vector mutation, sub-regions within the mask with poor spatial continuity or missegmentation, such as background adhesion, blurred edges, and depth holes, can be accurately identified and automatically removed from pose initialization calculations. This technique effectively filters out the mapping error from 2D noise to 3D space introduced by semantic segmentation, ensuring that the verified effective point cloud possesses high geometric consistency and spatial continuity. This significantly improves the accuracy and robustness of subsequent point cloud-based pose calculations, reducing the risk of initialization failure due to missegmentation.

[0059] In some embodiments of the present invention, the target pose calculated based on the semantic initialization branch of the first frame or the continuous pose tracking branch, combined with the fixed extrinsic parameter matrix of the binocular camera center relative to the coordinate system of the end effector of the six-degree-of-freedom robot arm and the real-time pose of the end effector of the six-degree-of-freedom robot arm in the world coordinate system, is obtained through coordinate transformation to obtain the absolute pose of the target object in the world coordinate system, which is achieved by the following calculation formula: T_W_O = T_W_E * T_E_C * T_C_O; Where T_W_O is the absolute pose of the target object in the world coordinate system, T_W_E is the real-time pose of the end effector of the six-DOF robot arm in the world coordinate system, T_E_C is the fixed extrinsic parameter matrix of the binocular camera center relative to the coordinate system of the end effector of the six-DOF robot arm, and T_C_O is the target pose of the target object in the camera coordinate system of the current image frame.

[0060] Coordinate transformation is achieved through cascaded matrix multiplication T_W_O = T_W_E * T_E_C * T_C_O. This technique effectively fuses the target's relative pose in the camera coordinate system, hand-eye calibration extrinsic parameters, and the robot's real-time end-effector pose in a rigid chain, forming a concise and clear closed-loop transformation chain. This facilitates millisecond-level real-time calculation by the embedded controller, significantly reducing the complexity and computational overhead of multi-coordinate system transformations. The target pose T_C_O (initial or optimized pose) calculated by the semantic initialization branch or continuous pose tracking branch of the first frame is used with a fixed extrinsic parameter matrix T_E_C calibrated offline with high precision to effectively avoid cumulative drift introduced by dynamic calibration. Combined with the real-time pose T_W_E of the six-DOF robot arm's end effector in the world coordinate system, which is fed back by real-time feedback from the robot's forward kinematics, the dynamic changes in the camera's perspective during the robot arm's movement are automatically compensated, ensuring strict synchronization between visual perception and the robot's motion state. The final output is the absolute pose T_W_O of the target object in the world coordinate system, which provides a globally unified and unambiguous spatial reference for downstream servo control, eliminates the coordinate alignment problem of relative pose in multi-task switching and multi-robot collaboration, and improves the absolute positioning accuracy and system reproducibility of grasping control.

[0061] In some embodiments of the present invention, the absolute pose is input to the servo controller to generate control commands, and the six-degree-of-freedom robot is guided to perform the corresponding grasping operation based on the control commands. This is achieved through a control law transfer function model, which is as follows: q_dot = λ * J_inv * e(t); Where q_dot is the joint velocity control command of the six-degree-of-freedom robot arm and dexterous hand, λ is the proportional control gain matrix; J_inv is the generalized Jacobian pseudo-inverse containing the fixed extrinsic parameter matrices of the end effector coordinate system of the six-degree-of-freedom robot arm and the binocular camera center relative to the end effector coordinate system of the six-degree-of-freedom robot arm; e(t) is the real-time error between the current pose of the target object and the desired grasping pose.

[0062] By using a control law transfer function model, the absolute pose of the target is mapped to joint velocity control commands in real time, constructing a closed-loop servo channel from visual perception to motion execution. This control law is driven by the real-time error e(t) between the current pose and the desired grasping pose. Utilizing the pseudo-inverse J_inv of the generalized Jacobian matrix, which includes fixed extrinsic parameters for hand and eye, the six-degree-of-freedom pose deviation in visual space is directly converted to the joint velocity space of the robotic arm and dexterous hand. This achieves a unified kinematic mapping between the visual coordinate system and the robot's joint space, avoiding the cumulative delay caused by repeated multi-coordinate system calculations in traditional hierarchical control. The proportional control gain matrix λ dynamically adjusts the system's response speed and tracking stability, ensuring rapid convergence while suppressing overshoot oscillations. This technique enables the six-degree-of-freedom robot to correct trajectory deviations in real time in dynamic environments, accurately guiding the end effector and dexterous hand to the optimal grasping posture, significantly improving the adaptability and positioning accuracy of grasping operations in unstructured scenarios.

[0063] In one embodiment of the present invention, the tracking optimizer outputs the real-time pose of the target object relative to the camera coordinate system, and calculates the absolute pose T_W_O of the target object in the world coordinate system according to the cascaded matrix T_W_O = T_W_E * T_E_C * T_C_O. By calculating the real-time error between the current pose of the target object and the desired grasping pose, and through proportional control of the gain matrix, the real-time error between the current pose of the target object and the desired grasping pose is converted into speed control commands for each joint of the six-degree-of-freedom robotic arm. Through this continuous pose feedback, the dexterous hand of the six-degree-of-freedom robotic arm can dynamically adjust its posture, improving the accuracy of close-range operations under strong occlusion, and ultimately completing a high-precision grasping task.

[0064] For example, in one specific embodiment, the comprehensive transformation matrix T_W_C (i.e., T_W_E * T_E_C) is obtained through forward kinematics and offline hand-eye calibration as follows: Line 1: [-0.01220, 0.69149, -0.72228, 0.72601]; Line 2: [ 0.99986, 0.00005, -0.01684, 0.29815]; Line 3: [-0.01161, -0.72238, -0.69140, 0.44635]; Line 4: [ 0.00000, 0.00000, 0.00000, 1.00000].

[0065] Where T_W_C is the comprehensive transformation matrix, T_W_E is the real-time pose matrix of the end effector of the six-DOF robot arm in the world coordinate system, and T_E_C is the fixed extrinsic parameter matrix of the binocular camera center relative to the coordinate system of the end effector.

[0066] Substituting the optimized translation vector of the target object in the camera coordinate system, P_opt = [0.136, 0.034, 0.928, 1]^T, into the matrix multiplication formula: X = -0.01220 * 0.136 + 0.69149 * 0.034 - 0.72228 * 0.928 + 0.72601 =0.078m; Y = 0.99986 * 0.136 + 0.00005 * 0.034 - 0.01684 * 0.928 + 0.29815 =0.419m; Z = -0.01161 * 0.136 - 0.72238 * 0.034 - 0.69140 * 0.928 + 0.44635 = -0.222m.

[0067] The calculated absolute position coordinates of the target object in the world coordinate system are X = 0.078m, Y = 0.419m, Z = -0.222m; Where P_opt is the homogeneous translation vector of the target object in the camera coordinate system in the current image frame, and X, Y, and Z are the absolute X-axis, Y-axis, and Z-axis coordinates of the target object in the world coordinate system, respectively. The system inputs this absolute pose T_W_O to the servo controller to generate control commands, which guide the six-DOF robot to perform the corresponding grasping operation based on the control law transfer function model q_dot = λ * J_inv * e(t). Where T_W_O is the absolute pose of the target object in the world coordinate system, q_dot is the joint velocity control command of the six-DOF robot arm and dexterous hand, λ is the proportional control gain matrix, J_inv is the physical parameter containing the pseudo-inverse of the generalized Jacobian matrix, and e(t) is the real-time error between the current pose of the target object and the desired grasping pose.

[0068] This invention also provides a target object pose prediction system based on palm-based binocular vision, used to execute the target object pose prediction method based on palm-based binocular vision described above, including: a binocular camera, an image frame judgment module, a first frame semantic initialization module, a continuous pose tracking module, a coordinate transformation and pose output module, a visual servo control module, and a controller.

[0069] Understandably, the binocular camera is positioned at the palm of the end effector of the six-DOF robot arm to acquire image frame sequences containing left and right eye images in real time within the work area. The image frame determination module determines whether the current image frame is the first frame or if tracking is lost. The first frame semantic initialization module executes the first frame semantic initialization branch when the current image frame is the first frame or tracking is lost, calling a large visual semantic model to perform semantic segmentation and geometric verification of the target object to determine the initial pose. The continuous pose tracking module executes the continuous pose tracking branch in continuous tracking frames, acquiring dense depth images and confidence weights through a lightweight binocular depth matching network, and combining this with motion prior constraints. The pose optimization module obtains the optimized pose; the coordinate transformation and pose output module is used to calculate the target pose based on the semantic initialization branch of the first frame or the continuous pose tracking branch, and combine the fixed extrinsic parameter matrix of the binocular camera center relative to the coordinate system of the end effector of the six-degree-of-freedom robot arm and the real-time pose of the end effector of the six-degree-of-freedom robot arm in the world coordinate system to obtain the absolute pose of the target object in the world coordinate system through coordinate transformation; the visual servo control module is used to input the absolute pose to the servo controller to generate control commands, and guide the six-degree-of-freedom robot to perform the corresponding grasping operation based on the control commands; and the control module is used to coordinate the decoupled operation of the semantic initialization module of the first frame and the continuous pose tracking module.

[0070] In this embodiment of the invention, the target object pose prediction system based on palm-based binocular vision adopts a modular architecture design. The control module coordinates the decoupled operation of the first-frame semantic initialization module and the continuous pose tracking module, enabling the large visual semantic model and the lightweight binocular depth network to work complementaryly in time: when the first frame or tracking is lost, the large model is activated to complete high-precision semantic-level initialization; during continuous tracking, the lightweight network is switched to ensure high frame rate real-time performance, effectively balancing computational resource consumption and pose estimation accuracy. The binocular camera is fixed at the palm position of the end effector of the six-DOF robot arm, forming a palm-based camera configuration. This allows the field of view to actively adjust with the robot arm, adaptively acquiring high-resolution near-field images and avoiding external field-of-view occlusion. The image frame judgment module achieves seamless switching between the two branches. The coordinate transformation and pose output module converts the camera's relative pose into the robot's real-time kinematics and transforms it into an absolute pose in the world coordinate system. The visual servo control module generates closed-loop control commands based on this absolute pose. The modules work together to form a complete closed loop from image acquisition and pose calculation to grasping execution, which significantly improves the overall robustness, real-time response capability and absolute positioning accuracy of the six-degree-of-freedom robot grasping system in unstructured dynamic scenes.

[0071] In some embodiments of the present invention, the visual semantic large model includes a multimodal coding layer and a cross-attention decoding layer. The multimodal coding layer extracts features from the text task instructions and the left eye image based on the CLIP text encoder. The cross-attention decoding layer fuses the text and left eye image features to output a two-dimensional semantic mask.

[0072] The visual semantic large-scale model adopts a CLIP-based multimodal encoding layer and cross-attention decoding layer architecture, fully leveraging the cross-modal alignment capability of the CLIP pre-trained model to achieve efficient association between text task instructions and left-eye image features in a shared semantic space. By parsing natural language instructions through a text encoder, the target object pose prediction system based on palm-based binocular vision can achieve target referential segmentation under open vocabulary conditions without requiring specific training or re-annotation for specific object categories, significantly enhancing its generalization ability for unseen objects and complex semantic descriptions. The cross-attention decoding layer dynamically fuses text and image features, accurately responding to the spatial and semantic constraints inherent in the instructions, and outputting a two-dimensional semantic mask that highly matches the target object's contour. This effectively suppresses similar interfering objects and background noise, improving the accuracy and completeness of segmentation boundaries, providing high-quality two-dimensional priors for subsequent geometric verification and pose initialization, and reducing dependence on large-scale scene annotation data.

[0073] It should be understood that the method steps in the embodiments of the present invention can be implemented or carried out by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable storage medium. The method can use standard programming techniques. Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with the computer system. However, if necessary, the program can be implemented in assembly or machine language. In any case, the language can be a compiled or interpreted language. Furthermore, for this purpose, the program can run on a programmed application-specific integrated circuit (ASIC).

[0074] Furthermore, the procedures described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by the context. The procedures described herein (or variations and / or combinations thereof) may be executed under the control of one or more computer systems configured with executable instructions, and may be implemented by hardware or a combination thereof as code (e.g., executable instructions, one or more computer programs, or one or more applications) that commonly executes on one or more processors. The computer program comprises a plurality of instructions executable by one or more processors.

[0075] Furthermore, the method can be implemented in any suitable type of computing platform, including but not limited to personal computers, minicomputers, mainframes, workstations, networked or distributed computing environments, standalone or integrated computer platforms, or in communication with charged particle tools or other imaging devices, etc. Aspects of the invention can be implemented as machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optical read and / or write storage medium, RAM, ROM, etc., such that it is readable by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the processes described herein. Furthermore, the machine-readable code, or portions thereof, can be transmitted via wired or wireless networks. The invention described herein includes these and other different types of non-transitory computer-readable storage media when such media comprises instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor. When programmed according to the methods and techniques described in the invention, the invention may also include the computer itself.

[0076] A computer program can be applied to input data to perform the functions described herein, thereby transforming the input data to generate output data stored in non-volatile memory. The output information can also be applied to one or more output devices, such as a display. In a preferred embodiment of the invention, the transformed data represents physical and tangible objects, including specific visual depictions of physical and tangible objects generated on the display.

[0077] The above description is merely a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention, as long as they achieve the technical effects of the present invention by the same means, should be included within the scope of protection of the present invention. Within the scope of protection of the present invention, the technical solutions and / or implementation methods can have various modifications and variations.

Claims

1. A target object pose prediction method based on palm-based binocular vision, characterized in that, include: Acquire a sequence of image frames in real time from a binocular camera fixed to the palm position of the end effector of a six-degree-of-freedom robot arm; Determine whether the current image frame is the first frame or if tracking is lost. If the current image frame is the first frame or the tracking is lost, the first frame semantic initialization branch is executed, and the visual semantic big model is called to perform semantic segmentation and geometric verification of the target object to determine the initial pose. If the current image frame is a continuously tracked frame, then the continuous pose tracking branch is executed. The dense depth image and confidence weights are obtained through a lightweight binocular depth matching network, and the pose is optimized by combining motion prior constraints. Based on the target pose calculated by the semantic initialization branch of the first frame or the continuous pose tracking branch, and combined with the fixed extrinsic parameter matrix of the binocular camera center relative to the coordinate system of the end effector of the six-degree-of-freedom robot arm and the real-time pose of the end effector of the six-degree-of-freedom robot arm in the world coordinate system, the absolute pose of the target object in the world coordinate system is obtained through coordinate transformation. The absolute pose is input to the servo controller to generate control commands, and the six-degree-of-freedom robot is guided to perform the corresponding grasping operation based on the control commands.

2. The target object pose prediction method based on palm-based binocular vision according to claim 1, characterized in that, The execution of the first frame semantic initialization branch, which calls the visual semantic large model to perform semantic segmentation and geometric verification of the target object to determine the initial pose, includes: Input the left eye image of the current image frame into the visual semantic big model, and output the two-dimensional semantic mask of the target object by fusing the text task instructions. By combining the depth map generated by the binocular depth network, the geometric consistency of the two-dimensional semantic mask is checked, and mis-segmented regions with depth continuity differences exceeding a set threshold at the edge of the two-dimensional semantic mask are removed to obtain the valid point cloud after verification. Based on the effective point cloud calculation and initialization of the target object's six-degree-of-freedom pose in the camera coordinate system, the initial pose of the target object is calculated using principal component analysis or bounding box algorithm.

3. The target object pose prediction method based on palm-based binocular vision according to claim 2, characterized in that, Also includes: During the task initiation phase, the six-degree-of-freedom robot arm is controlled to move above the work area, and the binocular camera at the palm position of the end effector of the six-degree-of-freedom robot arm is used to collect left and right eye images in the work area in real time at a preset frequency. The fixed extrinsic parameter matrix of the binocular camera center relative to the palm coordinate system of the end effector of the six-DOF robot arm, obtained through offline calibration, is pre-loaded. Based on the fixed extrinsic parameter matrix, the robot controller provides real-time feedback on the current joint state of the end effector of the six-degree-of-freedom robot arm via the bus based on the left and right eye images, and calculates the real-time pose of the end effector of the six-degree-of-freedom robot arm in the world coordinate system.

4. The target object pose prediction method based on palm-based binocular vision according to claim 3, characterized in that, The continuous pose tracking branch acquires dense depth images and confidence weights through a lightweight binocular depth matching network, and optimizes the pose by combining motion prior constraints, including: Control the large visual semantic model to enter sleep mode; A lightweight stereo matching network is used to perform stereo matching between the left and right images of the current image frame to generate a dense depth map and confidence weights. The dense depth map and the confidence weight are used as geometric consistency constraints and input into the pose tracking optimizer based on the target 3D model to obtain the optimized pose.

5. The target object pose prediction method based on palm-based binocular vision according to claim 4, characterized in that, The step of using the dense depth map and the confidence weight as geometric consistency constraints, and inputting them into the pose tracking optimizer based on the target 3D model to obtain the optimized pose includes: The dense depth map and the confidence weight are used as geometric consistency constraints. The target pose of the previous image frame, the fixed extrinsic parameter matrix between the offline calibrated camera center and the end effector of the six-degree-of-freedom robot arm, and the end effector pose change matrix of adjacent frames obtained by robot forward kinematics are extracted. The upper limit of the maximum reasonable change range of the target pose is derived to limit the pose search radius of the current image frame and construct the motion prior constraint of the target pose. The target 3D model is virtually rendered according to the target pose of the previous image frame to generate a predicted depth map. The pixel-level residual between the dense depth map and the predicted depth map is calculated. Pixels with residuals exceeding the depth tolerance threshold are identified as occluded areas and the pixel optimization weight is reset to zero. A nonlinear cost function containing a robust kernel function is constructed, and an iterative optimization algorithm is used to solve for the optimal pose increment, thereby obtaining the optimized pose of the target object in the camera coordinate system in the current image frame.

6. The target object pose prediction method based on palm-based binocular vision according to claim 2, characterized in that, The depth map generated by the binocular depth network is used to perform geometric consistency verification on the two-dimensional semantic mask, removing mis-segmented regions at the edges of the two-dimensional semantic mask where the depth continuity difference exceeds a set threshold, to obtain a valid point cloud after verification, including: The two-dimensional semantic mask is superimposed on the depth map of the first frame, and the local depth gradient of the pixels within the two-dimensional semantic mask is calculated. When the depth variance or normal vector of a certain sub-region suddenly exceeds the set threshold, it is determined that the spatial continuity of the sub-region is poor or there is a mis-segmented region, and the geometric consistency verification of the sub-region fails. Sub-regions that fail the geometric consistency check are removed from the pose initialization calculation to obtain the valid point cloud after verification.

7. The target object pose prediction method based on palm-based binocular vision according to claim 1, characterized in that, The target pose calculated based on the semantic initialization branch of the first frame or the continuous pose tracking branch, combined with the fixed extrinsic parameter matrix of the binocular camera center relative to the coordinate system of the end effector of the six-degree-of-freedom robot arm and the real-time pose of the end effector of the six-degree-of-freedom robot arm in the world coordinate system, is transformed to obtain the absolute pose of the target object in the world coordinate system, which is achieved by the following calculation formula: T_W_O = T_W_E * T_E_C * T_C_O; Where T_W_O is the absolute pose of the target object in the world coordinate system, T_W_E is the real-time pose of the end effector of the six-DOF robot arm in the world coordinate system, T_E_C is the fixed extrinsic parameter matrix of the binocular camera center relative to the coordinate system of the end effector of the six-DOF robot arm, and T_C_O is the target pose of the target object in the camera coordinate system of the current image frame.

8. The target object pose prediction method based on palm-based binocular vision according to claim 1, characterized in that, The absolute pose is input to the servo controller to generate control commands, and the six-degree-of-freedom robot is guided to perform the corresponding grasping operation based on the control commands. This is achieved through a control law transfer function model, which is as follows: q_dot = λ * J_inv * e(t); Where q_dot is the joint velocity control command of the six-degree-of-freedom robot arm and dexterous hand, λ is the proportional control gain matrix; J_inv is the generalized Jacobian pseudo-inverse containing the fixed extrinsic parameter matrices of the end effector coordinate system of the six-degree-of-freedom robot arm and the binocular camera center relative to the end effector coordinate system of the six-degree-of-freedom robot arm; e(t) is the real-time error between the current pose of the target object and the desired grasping pose.

9. A target object pose prediction system based on palm-based binocular vision, used to execute the target object pose prediction method based on palm-based binocular vision as described in any one of claims 1 to 8, characterized in that, include: A binocular camera is installed in the palm position of the end effector of a six-degree-of-freedom robot arm to acquire image frame sequences containing left and right eye images in the working area in real time. The image frame determination module is used to determine whether the current image frame is the first frame or a tracking loss state; The first frame semantic initialization module is used to execute the first frame semantic initialization branch when the current image frame is the first frame or when the tracking is lost. It calls the visual semantic big model to perform semantic segmentation and geometric verification of the target object to determine the initial pose. The continuous pose tracking module is used to execute the continuous pose tracking branch in continuous tracking frames. It obtains dense depth images and confidence weights through a lightweight binocular depth matching network, and combines motion prior constraints to optimize the pose and obtain the optimized pose. The coordinate transformation and pose output module is used to calculate the target pose based on the semantic initialization branch of the first frame or the continuous pose tracking branch, and combine the fixed extrinsic parameter matrix of the binocular camera center relative to the coordinate system of the end effector of the six-degree-of-freedom robot arm and the real-time pose of the end effector of the six-degree-of-freedom robot arm in the world coordinate system to obtain the absolute pose of the target object in the world coordinate system through coordinate transformation. The visual servo control module is used to input the absolute pose to the servo controller to generate control commands, and guide the six-degree-of-freedom robot to perform the corresponding grasping operation based on the control commands. as well as, The control module is used to coordinate the decoupled operation of the first frame semantic initialization module and the continuous pose tracking module.

10. The target object pose prediction system based on palm-sized binocular vision according to claim 9, characterized in that, The large visual semantic model includes a multimodal coding layer and a cross-attention decoding layer. The multimodal coding layer extracts features from the text task instructions and the left eye image based on the CLIP text encoder. The cross-attention decoding layer fuses the text and left eye image features to output a two-dimensional semantic mask.

Images