Robot precision grasping method and system based on visual positioning

Abstract

The application discloses a robot precision grabbing method and system based on visual positioning. The application overcomes the low precision, poor stability and insufficient flexibility of a single solution by combining three-dimensional visual geometry solving and AI perception through a two-stage framework of global coarse positioning and local precise grabbing. In the global stage, a visual recognition model is used to extract a two-dimensional mask and fuse depth data to generate a spatial point cloud. The geometric features of the point cloud are analyzed to determine the spatial arrangement form, and a collision avoidance pre-grabbing point and an initial approaching attitude are generated to achieve safe approaching. In the local stage, a local vision camera at the end acquires local images and depth data, extracts a local mask to calculate the main direction and obtain the rotation calibration angle of the gripper, and based on the local depth data, the spatial coordinates are solved and the gripper size bias is applied to generate the grabbing coordinates. Finally, the precise grabbing pose is synthesized to control grabbing. The two-stage cooperation improves the grabbing precision and reliability, and meets the requirements of industrial flexible grabbing.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and in particular to a method and system for precise robot grasping based on vision positioning. Background Technology

[0002] In smart manufacturing and flexible production scenarios, robot vision grasping is a core technology for achieving tasks such as material sorting, autonomous loading and unloading, and precision assembly. However, current mainstream single-vision solutions still have significant limitations under the high precision and high stability requirements of industrial mass production: traditional vision solutions based on two-dimensional images can only acquire planar features and lack the ability to effectively calculate the depth information and three-dimensional spatial posture of the target. Therefore, they are usually only suitable for simple grasping scenarios with regular workpieces or fixed postures. When faced with complex materials that are irregularly shaped, stacked, or have randomly changing postures, the positioning accuracy is seriously insufficient.

[0003] While traditional 3D vision solutions have achieved spatial positioning by introducing depth data, they suffer from high computational costs for point cloud processing, poor real-time performance, and are prone to missing depth point clouds and accumulated errors under complex lighting, occlusion, and surface reflection conditions in industrial settings. Furthermore, these solutions still have many shortcomings in terms of accuracy, versatility, and coordinate system adaptation in 3D spatial coordinate and attitude calculation, making it difficult to effectively meet the flexible grasping needs of various types of irregularly shaped materials.

[0004] In recent years, the industry has gradually introduced artificial intelligence visual algorithm models such as vision-language-action and vision-world-action. These methods have strong scene adaptability and flexibility in target recognition and feature perception, but they are highly dependent on large-scale labeled data. In non-standard and complex industrial production conditions, they are prone to problems such as inference drift, feature mismatch and unstable output. It is difficult to achieve the high reliability and high accuracy required for industrial mass production by relying on a single artificial intelligence algorithm.

[0005] In summary, traditional vision solutions struggle to balance accuracy, real-time performance, and flexibility, while single-person intelligent vision algorithms suffer from unstable output and low reliability in complex industrial environments. How to organically integrate the geometric calculation capabilities of traditional 3D vision with the perceptual flexibility of artificial intelligence algorithms, achieving high-precision and high-stability 3D pose calculation while avoiding inference uncertainties, has become a critical technological bottleneck that urgently needs to be overcome in the field of robot vision grasping. Summary of the Invention

[0006] The purpose of this application is to provide a visual positioning-based robot precision grasping method and system, which solves the problem that a single vision solution cannot simultaneously meet the requirements of high precision, high stability and flexible grasping.

[0007] To address the aforementioned technical problems, this application provides the following technical solutions: The first aspect of this application provides a method for precise grasping of a robot based on visual positioning, the method comprising: Acquire image data and depth data from a global vision camera, use a visual recognition model to extract a two-dimensional mask of the target object from the image data, and combine the two-dimensional mask with the depth data to map a spatial point cloud; The spatial geometric features of the spatial point cloud are analyzed to determine the spatial arrangement of the target object. Based on the spatial arrangement, a pre-grabbing spatial point and an initial approach posture of the end effector are adaptively generated to avoid collisions. The robotic arm is then controlled to move to the pre-grabbing spatial point. Local image data and local depth data are acquired by the end-effector local vision camera, the local mask of the target object is extracted, the main direction geometric features of the local mask are calculated, and the rotation calibration angle of the end gripper is obtained. Based on the local depth data corresponding to the local mask, the spatial coordinates of the target are calculated. Using the spatial coordinates as a reference, a position offset is applied along the gripping feed direction in combination with the size of the end gripper to generate the end gripping coordinates. Based on the rotation calibration angle and the end-effector grasping coordinates, a precise grasping pose is generated to control the robotic arm to grasp the target object.

[0008] A second aspect of this application provides a vision-based positioning-based robot precision grasping system, the system comprising: The spatial point cloud generation module is used to acquire image data and depth data from a global vision camera, extract a two-dimensional mask of the target object from the image data using a visual recognition model, and map the two-dimensional mask into a spatial point cloud by combining the depth data. The first control module is used to analyze the spatial geometric features of the spatial point cloud to determine the spatial arrangement of the target object, adaptively generate a pre-grabbing spatial point and an initial approach posture of the end effector to avoid collision based on the spatial arrangement, and control the robotic arm to move to the pre-grabbing spatial point. The rotation calibration angle calculation module is used to acquire local image data and local depth data through the end-effector local vision camera, extract the local mask of the target object, calculate the main direction geometric features of the local mask, and obtain the rotation calibration angle of the end gripper. The end-effector coordinate generation module is used to calculate the spatial coordinates of the target object based on the local depth data corresponding to the local mask, and to generate the end-effector coordinates by applying a position offset along the gripping feed direction in combination with the size of the end gripper, based on the spatial coordinates. The second control module is used to generate a precise grasping pose based on the rotation calibration angle and the end-grabbing coordinates, so as to control the robotic arm to grasp the target object.

[0009] Compared to existing technologies, the visual positioning-based robot precision grasping method and system provided in this application effectively integrates the geometric calculation capabilities of traditional 3D vision with the perceptual flexibility of artificial intelligence algorithms through a two-stage visual guidance framework that combines "global coarse positioning and local fine grasping." This overcomes the dual shortcomings of single vision solutions in complex industrial scenarios, such as low accuracy, poor real-time performance, and insufficient flexibility, as well as the unstable output and low reliability of single AI solutions. In the global stage, a visual recognition model is used to extract the two-dimensional mask of the target object from the image and combine it with depth data to map it into a spatial point cloud. By analyzing the spatial geometric features of the point cloud, the spatial placement of the target object is adaptively determined, thereby generating pre-grasping spatial points for collision avoidance and the initial approach posture of the end effector. This solves the problem that traditional methods cannot simultaneously consider obstacle avoidance safety and scene adaptability, and achieves flexible perception and safe approach to irregularly shaped, stacked, and randomly shaped materials. In the local phase, high-resolution local images and depth data are acquired through an end-effector local vision camera. Local masks are extracted, and their principal direction geometric features are calculated to obtain the rotation calibration angle of the end-effector gripper. Simultaneously, the precise spatial coordinates of the target object are calculated based on the local depth data, and gripper size offsets are applied to generate end-effector grasping coordinates. The joint calculation of 2D texture features and 3D spatial information achieves high-precision calibration of the target object's six-DOF pose, effectively avoiding the inference drift and feature mismatch risks of a single AI model in non-standard industrial conditions. The two phases work together to significantly enhance the system's reliability and versatility while ensuring grasping accuracy, meeting the stringent requirements of high-precision, high-stability flexible grasping in industrial mass production scenarios. Attached Figure Description

[0010] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily understood by reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of this application are illustrated by way of example and not limitation, with the same or corresponding reference numerals denoteing the same or corresponding parts, wherein: Figure 1 A flowchart illustrating a vision-based positioning method for precise robotic grasping is shown schematically. Figure 2 A schematic diagram of a vision-based robotic precision grasping system is shown. Detailed Implementation

[0011] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0012] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application shall have the ordinary meaning as understood by one of ordinary skill in the art to which this application pertains.

[0013] The methods described in the embodiments of this application will be explained in detail below.

[0014] This embodiment provides a vision-based robot precision grasping method. This method can be applied to tasks such as autonomous material sorting, precision autonomous loading and unloading, flexible assembly, and precise object grasping of humanoid robots, dual-arm robots, or multi-axis multi-joint robots in scenarios such as intelligent manufacturing, flexible production lines, warehousing and logistics sorting, and intelligent home services. In this embodiment, the algorithm processing tasks such as visual recognition, image segmentation, and geometric motion pose calculation can be flexibly deployed on cloud servers (such as GPU computing clusters), edge computing terminals mounted on the robot body (such as vehicle-mounted high-performance computing units, embedded GPUs / NPUs), or distributed collaborative computing platforms between the robot body and the cloud, depending on actual computing power and working conditions. These tasks can be executed independently by the robot terminal or collaboratively by the cloud server and the robot terminal through wired or wireless networks.

[0015] This embodiment is mainly used for autonomous object recognition, precise grasping, and positioning tasks of humanoid robots. The method integrates global and local 3D visual perception units deployed on different physical parts of the humanoid robot to estimate the high-precision 3D spatial pose of the object to be grasped relative to the robot body in real time. This allows for the control of a high-degree-of-freedom robotic arm and end effector to approach the target object safely without collision, ultimately achieving an autonomous closed-loop operation that grasps and stably places the specified object at a fixed target location.

[0016] To achieve coordination between global macro-level scene planning and local high-precision operation, the humanoid robot system in this embodiment mainly includes the following core hardware components: Head-mounted global vision perception unit: This includes an RGBD camera (preferably one) mounted on the robot's head. This head-mounted RGBD camera has a wide global field of view and is primarily responsible for acquiring large-field-of-view global color images and coarse depth data of the workspace, used for rapid intelligent recognition of target objects, global path planning, and obstacle avoidance navigation. Redundant degree-of-freedom actuators: including seven-degree-of-freedom robotic arms (preferably two in number) symmetrically arranged on both sides of the robot body in a dual-arm configuration. The kinematic redundancy design of the seven-degree-of-freedom robotic arms allows the robotic arms to approach target objects with highly flexible spatial postures, avoiding singularities and physical collisions in complex, non-standard industrial operating spaces. Wrist-based local visual perception unit: including binocular depth cameras (preferably two) mounted at the wrist joints of both robotic arms. The optical axis of the wrist-based binocular depth cameras is oriented towards the gripper's grasping space and dynamically changes its field of view as the robotic arm moves. It is specifically responsible for acquiring high-resolution local images and high-precision local depth data at close range after the robotic arm moves to the pre-grasping point, so as to achieve closed-loop fine pose calibration. End effector: includes end effector grippers (preferably two) connected to the end flanges of the two robotic arms respectively, which are used as end effectors to accurately perform clamping and releasing operations on the target object after receiving the final gripping posture control command.

[0017] Step 110: Obtain image data and depth data from the global vision camera, use a visual recognition model to extract the two-dimensional mask of the target object from the image data, and combine the two-dimensional mask with the depth data to map it into a spatial point cloud.

[0018] This step aims to achieve initial coarse localization of the target object using a global vision camera and to construct a high-precision world coordinate system point cloud.

[0019] The data acquired by a global vision camera typically includes two main categories: Image data (RGB / grayscale) It records two-dimensional planar information such as the texture, color, and outline of an object's surface, which forms the basis for subsequent identification of "what it is and where it is".

[0020] Deep data It records the physical distance (usually in millimeters) of each pixel in the scene from the camera plane. It is often represented as a grayscale image, where brighter pixels indicate closer proximity and darker pixels indicate greater distance. This forms the basis of three-dimensional spatial perception.

[0021] In a specific implementation, extracting the two-dimensional mask of the target object from image data using a visual recognition model can be achieved through the following four steps: (1) Receive text prompts for the target object.

[0022] The system receives a text prompt for the target object. This text prompt (such as inputting "red plastic cup") is fed into a text encoder (such as the CLIP model) and converted into a string of "feature number vectors" representing semantics.

[0023] (2) Use target detection algorithms to perform multimodal recognition and localization on text prompts and image data, so as to output the pixel area of ​​the target in the image data and determine the center point coordinates of the pixel area.

[0024] The system utilizes open-vocabulary detection models (such as the YOLO-world object detection algorithm) to perform multimodal fusion recognition and localization of text prompts and global image data, outputting the two-dimensional bounding box pixel region of the target object in the global image. Specifically, the open-vocabulary detection model extracts visual feature maps from the entire image and then compares these visual features with the "feature digital vectors" generated by the text encoder (through attention-based cross-fusion). When it finds that the visual features of a certain region highly match the text, it draws a rectangular box in that region and calculates the coordinates of the box's center point.

[0025] (3) Use the center point coordinates as input prompts and input them into the image segmentation algorithm to generate the original two-dimensional mask of the target object.

[0026] The coordinates of this center point are used as input cue points and fed into a large-scale image segmentation model (such as the SAM algorithm). The image segmentation algorithm uses these cue points as references to perform region growing and segmentation at the pixel level, outputting the original two-dimensional mask of the target object.

[0027] (4) Perform morphological processing, including dilation and erosion, on the original two-dimensional mask to obtain the final two-dimensional mask.

[0028] Finally, to eliminate burrs at the segmentation edges and fill in the holes, the original two-dimensional mask is subjected to morphological processing including dilation and erosion to obtain the final refined two-dimensional mask after denoising.

[0029] After obtaining the refined 2D mask, the system maps the 2D pixels within the mask to a 3D point cloud. Therefore, in a specific implementation, mapping the 2D mask combined with depth data to a spatial point cloud can be achieved through the following five steps: (1) Align the depth data of the global vision camera with the global image data, and query and extract the depth values ​​corresponding to each pixel in the two-dimensional mask from the aligned depth data.

[0030] Acquire the depth image from a global vision camera (such as an RGBD camera), and align the depth image with the image data pixel by pixel. In the aligned depth image, query and extract the depth value z corresponding to each valid pixel (x, y) in the 2D mask.

[0031] (2) Obtain the camera intrinsic parameters of the global vision camera, including the axial pixel focal length and the coordinates of the principal point of the image.

[0032] First, obtain the camera intrinsic parameter matrix K of the global vision camera: ; in, : These represent the pixel focal lengths of the camera along the horizontal and vertical axes, respectively. : These represent the pixel offsets of the principal point of the camera image sensor in the horizontal and vertical axes, respectively.

[0033] These represent the pixel offsets of the principal point of the camera image sensor along the horizontal and vertical axes, respectively.

[0034] (3) Using the pinhole camera inverse projection model, based on the axial pixel focal length, image principal point coordinates and depth value, the two-dimensional coordinates of each pixel in the two-dimensional mask are restored to the camera three-dimensional space points in the camera coordinate system.

[0035] Based on the horizontal pixel focal length in the intrinsic parameters Vertical pixel focal length and the horizontal principal point of the image Vertical principal point Using a pinhole camera inverse projection model, the coordinates of each pixel within a two-dimensional mask are determined. The corresponding alignment depth value z is then inversely projected to restore it to a 3D point in the local camera coordinate system. The specific calculation formula is as follows: ; The formula parameters are explained below: (x, y): Represents the row and column coordinates of the current pixel within the two-dimensional mask.

[0036] z: Represents the physical distance depth value obtained after the current pixel is aligned.

[0037] : Indicates the position coordinates of the point reconstructed by back projection in the current camera's 3D coordinate system.

[0038] (4) Using the coordinate transformation tree of the robot system, call the pre-stored camera calibration parameters of the global vision camera. The camera calibration parameters include the rotation matrix and translation vector.

[0039] (5) Using rotation matrix and translation vector, the camera's three-dimensional spatial points are transformed into the preset world coordinate system to construct a spatial point cloud.

[0040] Obtain the three-dimensional spatial points in the camera coordinate system Then, through the robot system's coordinate transformation tree (TF tree), the spatial rigid body transformation parameters (i.e., camera extrinsic pose) of the global vision camera relative to the world coordinate system at the current moment are called. The extrinsic transformation parameters consist of a rotation matrix R and a translation vector t, which are used to transform the points in the camera coordinate system... Transform to the preset world coordinate system (defined by front X, left Y, and top Z): ; Perform the above spatial transformation on all pixels corresponding to the mask to obtain all 3D points in the world coordinate system. The set of points is used to construct the global coarse localization point cloud of the target object.

[0041] Formula parameter description: : Represents the three-dimensional coordinates of a point in the target object's point cloud transformed to the world coordinate system. ; R: by to A 3×3 rotation matrix consisting of 9 elements represents the directional rotation transformation from the camera coordinate system to the world coordinate system; t: by The 3×1 translation vector represents the spatial translation displacement between the two coordinate systems.

[0042] Preferably, as a preliminary step for initialization in this embodiment, before executing the real-time grasping task, the camera calibration parameters of the global vision camera need to be calculated offline or online, and then preloaded into the coordinate transformation tree (TF tree) of the robot system as static geometric parameters.

[0043] The specific calibration process is as follows: The robot is controlled to move in N different joint poses (N being an integer greater than 3) to allow the global vision camera to present multiple different spatial poses. Simultaneously, the position of the robotic arm's end effector (or head-end link) relative to the basic coordinate system is recorded in each spatial pose. The first pose transformation data is recorded, and the RGB image data of the fixed calibration plate captured by the global vision camera is also recorded.

[0044] During the calibration process, the calibration plate is positioned relative to the basic coordinate system. The spatial pose remains unchanged. Based on this physical constraint, a set of homogeneous pose equations can be constructed for different times and attitudes: ; Since the calibration plate remains stationary, the transformation results at all times are equal, that is: ; By solving the above system of equations, the camera calibration parameters of the global vision camera relative to the robotic arm's end effector (or head joint) can be obtained. This parameter includes the static rotation matrix. and static translation vector And store them as static geometric constants in the coordinate transformation tree.

[0045] : Represents the pose matrix subscribed to in real time by the robot end effector (or head end effector Link) in the base coordinate system.

[0046] : Represents the calibration transformation parameters of the global vision camera relative to the end effector of the robotic arm.

[0047] : Represents the real-time pose of the calibration board in the camera coordinate system estimated by the global vision camera.

[0048] : Represents the spatial transformation matrix of the calibration plate coordinate system in the robot base coordinate system at time k.

[0049] Specifically, the camera calibration parameters obtained by solving the simultaneous equations Mathematically, this is represented as a 4×4 homogeneous pose transformation matrix, and its specific matrix structure is as follows: ; By performing matrix decomposition on the aligned pose transformation matrix, its top-left 3×3 orthogonal submatrix is ​​extracted and determined as the static rotation matrix. Extract its top-right 3×1 dimension column vector and determine it as the static translation vector. This leads to the static rotation matrix. and static translation vector The coordinate transformation tree is loaded into the robot system as static geometric parameters.

[0050] The formula parameters are explained below: : The calculated 4×4 homogeneous transformation matrix of the camera relative to the end of the robotic arm.

[0051] : A 3×3 static rotation matrix extracted from the homogeneous transformation matrix (representing the angular deviation between the camera and the axis of the robotic arm end).

[0052] : A 3×1 static translation vector extracted from the homogeneous transformation matrix (representing the spatial distance deviation between the camera and the physical center of the robotic arm's end effector).

[0053] : Represents a zero row vector with a dimension of 1×3.

[0054] Based on the above principles, the camera calibration parameters of the global vision camera are pre-calculated and loaded into the coordinate transformation tree in the following way, which can be implemented in the following four steps: The robot is controlled to move in a variety of different postures so that the global vision camera can present a variety of different spatial poses. Simultaneously, the first pose transformation data of the robotic arm end relative to the base coordinate system under each spatial pose, as well as the image data of the fixed calibration plate captured by the global vision camera, are recorded. Based on the image data, the second pose transformation data of the calibration board relative to the global vision camera is calculated; Based on the constraint that the spatial pose of the calibration plate remains unchanged relative to the base coordinate system during the calibration process, and by combining the first pose transformation data and the second pose transformation data under each pose, multiple sets of homogeneous pose equations at different times are constructed. The homogeneous equations of pose are solved simultaneously to calculate the camera calibration parameters of the global vision camera relative to the end effector of the robotic arm. The camera calibration parameters are composed of rotation matrices and translation vectors, and are then loaded as static geometric parameters into the coordinate transformation tree of the robot system.

[0055] Step 120: Analyze the spatial geometric features of the spatial point cloud to determine the spatial arrangement of the target object, adaptively generate pre-grabbing spatial points and initial approach posture of the end effector based on the spatial arrangement, and control the robotic arm to move to the pre-grabbing spatial points.

[0056] First, this step performs macroscopic geometric feature analysis on the spatial point cloud acquired by the global vision camera to calculate the longest geometric extension direction of the target object (i.e., the principal axis direction). This allows the robot to see and understand, like a human, whether the target object is physically upright or lying flat in space. Under this shape determination criterion, the system does not blindly drive the gripper directly to the target core for grasping. Instead, through mathematical vector constraints, it adaptively extrapolates a preset safety distance in the safe space surrounding the target object. This generates a safe transition point for the robot that provides the best secondary fine-tuning perspective for the hand-eye camera while completely avoiding the gripper's hard collision with the material or tabletop—the pre-grabbing spatial point. Simultaneously, the system uses the spatial normal vector pointing from this pre-grabbing point to the geometric center of the object to mathematically constrain and solve for the insertion orientation of the gripper when approaching the object—the initial approach posture of the end effector. This completes the spatial angle alignment of the gripper before it enters the charging state. Ultimately, the robotic arm, carrying the initial approach angle, first translates to this safe pre-grabbing point and stops there. Physically, this achieves a defensive coarse positioning transition action that moves from far to near, trading space for safety, thus laying a complete pose reference for the subsequent secondary local precise alignment calibration of the hand-eye camera.

[0057] Based on the above principles, in a specific implementation, step 120 can be implemented by the following steps: (1) Extract and project the boundary features of the spatial point cloud to determine the extension size of the target object in three mutually perpendicular geometric feature directions and the geometric center point of the spatial point cloud. The geometric feature direction with the largest size is defined as the principal axis direction of the target object.

[0058] To determine the physical boundaries of an object in a 3D point cloud, this method first projects the point cloud onto a horizontal plane (XY plane) and uses Principal Component Analysis (PCA) to calculate the two mutually perpendicular feature orientation vectors (principal direction vectors) of the object on the horizontal plane. and secondary direction vector Since the height direction (Z-axis) is naturally perpendicular to the horizontal plane, therefore, the principal direction... Secondary direction Together with the height direction Z, they constitute "three mutually perpendicular geometric feature directions".

[0059] By projecting the point cloud onto these three orthogonal directional axes and calculating the difference in their projection extreme values, the "extended dimensions" (i.e., length L, width W, and height H) of the target object in these three directions can be determined. The axis corresponding to the largest dimension is the "principal axis direction".

[0060] Extracting spatial point clouds And then perform centering (mean removal) processing: ; Coordinates of the point after centralization: ; Constructing a centralized matrix Then, a symmetric covariance matrix of the two-dimensional projection point set is constructed. : ; in: ; ; Perform eigenvalue decomposition on the covariance matrix to obtain mutually perpendicular orthogonal unit eigenvectors. (Main axis direction) and (Secondary axis direction): ; Projecting the mean point onto the feature vector, calculate the extended dimensions (length L, width W, height H) of the target object in these three orthogonal directions, as well as its three-dimensional geometric center point. : ; ; Based on the above size definition, the principal axis direction Dirmain of the target object is determined as follows: ; The parameters disclosed above include: : The two-dimensional position of the i-th sampling point in the point cloud on the horizontal projection plane.

[0061] The mean of all sampling points on the horizontal coordinate axis physically represents the center projection of the object onto the horizontal plane.

[0062] : The coordinates of the center point after eliminating the overall coordinate system offset (removing the mean).

[0063] : The covariance matrix characterizing the discrete features of the horizontal distribution of sampling points.

[0064] The two mutually perpendicular horizontal feature orientations obtained from the calculation correspond to the major axis and the minor axis of the object itself, respectively.

[0065] : Scalar values ​​of the projection of each point onto the length and width characteristic axes.

[0066] L, W, H: The calculated extension dimensions of the target object in three mutually perpendicular geometric directions (primary axis, secondary axis, and height axis).

[0067] : The geometric center point of the bounding box of the spatial point cloud, used as the reference origin for subsequent position offset calculations.

[0068] : Variance of the spatial coordinate distribution of edge image points in the horizontal and vertical directions.

[0069] The covariance of edge image points in the horizontal and vertical directions, its non-zero state directly reflects the rotation and skew of the object in the image.

[0070] (2) If the main axis direction is parallel to the height axis defined in the reference coordinate system of the current robot system, the target object is determined to be in an upright posture. The geometric center point is pushed outward by a preset safety distance along the radial direction perpendicular to the height axis to generate a pre-grabbing spatial point.

[0071] In the robot's world coordinate system, the "height axis" is the Z-axis. If the direction of the largest extension dimension (the principal axis direction) is calculated in the first step as the height H (i.e., H>[L,W]), since the measurement direction of H is itself along the Z-axis (vertical direction) of the world coordinate system, this physically means that "the principal axis direction of the object is parallel to the height axis".

[0072] At this point, the system determines that the object is in an "vertical posture." To prevent the robotic arm from directly knocking over the tall, slender, vertical object during its lateral horizontal approach, the gripper's waiting position must be shifted outward by a safe obstacle avoidance distance Δy from the object's geometric center along the horizontal radial plane perpendicular to the height axis. This adaptively generates pre-grabbing spatial positions for lateral obstacle avoidance. .

[0073] Under the constraint H>[L,W], the placement is determined to be vertical, and the coordinates of the pre-grabbed spatial points are generated. The calculation formula is:

[0074] Where H>[L,W]: determines the state condition where the height dimension is greater than the length and width (vertical state determination).

[0075] : The coordinates of the generated lateral pre-captured spatial points.

[0076] : The three axial components of the geometric center point C of the object's three-dimensional bounding box.

[0077] The preset lateral obstacle avoidance offset distance in the radial plane perpendicular to the height axis (specifically represented by the Y axis in this case) ensures that the gripper will not rub against and push over objects during horizontal approach when it opens by subtracting this value (extrapolation).

[0078] (3) If the main axis direction is not parallel to the height axis, the target object is determined to be in a lying position. The geometric center point is shifted by a preset safe distance along the height axis to generate a pre-grabbing spatial point.

[0079] If the direction of the largest extension dimension (the principal axis direction) is the length L or the width W (i.e., L, W>H), since both the major axis and the minor axis are distributed in the horizontal projection plane (XY plane) and are perpendicular to the vertical Z-axis (height axis) (i.e., not parallel), this physically means that "the principal axis direction of the object is not parallel to (intersecting or perpendicular to) the height axis".

[0080] At this point, the system determines that the object is in a "lying position." To avoid the gripper rubbing against the placement platform when the robotic arm approaches horizontally, the system decides to adopt a top-down vertical approach strategy. This is achieved by positioning the geometric center point above the height axis (Z-axis). Move upwards by a safe obstacle avoidance distance This allows for the adaptive generation of pre-grabbing spatial points for vertical obstacle avoidance. Under the constraint L, W>H, the placement pattern is determined to be a lying posture, and the coordinates of the pre-grabbed spatial points are generated. The calculation formula is: ; Where L,W>H: spatial constraints that determine if the length or width is greater than the height (flat state determination).

[0081] : Preset safe obstacle avoidance distance along the height axis upwards. By adding this value (upward translation), the end effector of the robotic arm can hover at a safe height directly above the lying object, waiting for final alignment and grasping.

[0082] (4) Construct an orientation vector from the pre-captured spatial point to the geometric center point, and use the angle between the orientation vector and the preset reference direction vector in the reference coordinate system and the normal axis to constrain and solve the initial approach posture of the end.

[0083] In order for the robotic arm's end effector gripper to point at and approach the object in the perfect orientation, it is necessary to calculate the safe transition waiting point (pre-grasp spatial position). Pointing to the center point of the target Three-dimensional spatial "orientation vector" Then, normalize the vector to obtain the unit normal vector. .

[0084] The default reference direction vector of this system is the forward axis of the world coordinate system. By solving Rotate to The spatial angle required for overlap and the normal axis of rotation perpendicular to their plane of rotation This allows for the construction of a perfect spatial rotation constraint, which in turn enables the calculation of the quaternion of the initial approach posture at the end point. .

[0085] Construct and normalize the orientation vector from the pre-fetched points to the center point of the bounding box: ; ; Calculate the unit normal vector With reference direction vector The angle between and orthogonal rotation normal axis : ; Based on the rotation axis and angle, construct quaternion rotation constraints and solve for the initial approach posture quaternion of the end effector. : ; Combining the above equations, we obtain the final attitude quaternion. The only complete calculation formula: ; in, : The approximation vector from the pre-grab point to the center of the target.

[0086] Normalized 3D spatial orientation normal vector.

[0087] The spatial angle required for the end effector of the robotic arm to rotate from the world forward reference axis to align with the target orientation vector.

[0088] The spatial normal rotation axis obtained by the cross product of two vectors defines the axis around which the end effector of the robotic arm rotates.

[0089] The calculated initial approach quaternion of the end effector. This quaternion directly constrains and locks the orientation of the robotic arm's end effector at the pre-grasp point, ensuring that its central axis is precisely aligned with the center of the object.

[0090] Step 130: Acquire local image data and local depth data through the end-effector local vision camera, extract the local mask of the target object, calculate the principal direction geometric features of the local mask, and obtain the rotation calibration angle of the end gripper.

[0091] Once the robotic arm, carrying a coarse positioning posture, has stabilized and stopped at the pre-grasping spatial point, the physical distance between it and the target object has been significantly reduced. At this point, the infrared or visible light sensor of the binocular depth camera will be at the optimal imaging distance (i.e., within the effective working depth of field of baseline convergence). The system sends a synchronization trigger pulse signal to control the left and right miniature image sensors of the binocular depth camera to simultaneously acquire left and right eye images of the target object from two spatial perspectives with a fixed small baseline difference at the same microsecond time stamp. These two high-resolution two-dimensional pixel matrices together constitute the local image data for subsequent local semantic feature extraction. Therefore, the local image data acquired by the local vision camera is a local binocular image.

[0092] Simultaneously, after acquiring local image data containing geometric differences between the two views, the depth engine or vision computing module within the robot's control system immediately initiates the stereo vision matching and disparity calculation process to generate the local depth data required for 3D spatial reconstruction. In the specific calculation process, the system first performs distortion correction and epipolar correction on the left and right eye images, simplifying the complex two-dimensional pixel search into a one-dimensional matching of a single horizontal row of pixels. Next, using a semi-global matching (SGM) algorithm or a deep learning-based cost volume stereo matching algorithm, the system searches for the highest-matching corresponding feature pixel on the corresponding epipolar line of the right eye image, using the target object pixel in the left eye image as a reference, thereby calculating the pixel displacement difference in the horizontal direction between the two images, i.e., the disparity. Finally, the system calculates the disparity based on the physical baseline length of the binocular cameras. and the standard pixel focal length in the lens internal parameters Using the geometric trigonometric parallax mapping formula The entire image's two-dimensional disparity map is converted pixel-by-pixel into a depth matrix representing the physical distance from each point on the target object's surface to the optical axis of the binocular camera. The value of each pixel in this depth matrix represents the local depth value of the corresponding point on the material's surface, thus forming high-density, low-noise local depth data. This high-precision local depth data not only accurately pinpoints the microscopic three-dimensional undulations of the target object's local surface but also achieves pixel-level geometric alignment with the left-eye image, providing a bias-free underlying data benchmark for subsequent high-precision local mask extraction and final calibration of the gripper's end rotation angle.

[0093] In a specific implementation, extracting the local mask of the target object, calculating the principal direction geometric features of the local mask, and obtaining the rotation calibration angle of the end gripper can be achieved through the following steps: (1) Receive local prompt information for the target object, and control the end local vision camera to acquire local binocular images and local depth images of the target object at the pre-capture spatial point, and perform alignment operation.

[0094] When the robot reaches the pre-grabbing spatial point Then, the robotic arm's wrist camera is in a close-range approach state. At this time, the robot control system first receives "local cue information" (such as "blue metal bolt" or other prompts) for the target object to be grasped from the main control cloud, manual input, or upper-level task chain. Subsequently, it controls the binocular depth camera on the end effector to perform local exposure, simultaneously acquiring local binocular images of the object and corresponding high-precision depth images, and performs high-precision pixel registration and alignment between the two, so that the row and column coordinates of the same pixel correspond to a unique physical depth value.

[0095] (2) Use a visual recognition model to match local prompt information and local binocular images, calculate the location prompt points of the corresponding target in the local binocular images, and input the location prompt points as guiding prompts into the image segmentation algorithm to segment and extract the local mask of the target.

[0096] After acquiring local images using the robot's wrist-mounted binocular depth camera, the robot system initiates high-precision local semantic feature-level matching. The system retrieves the previously received "local prompt information" (in this embodiment, specifically manifested as text prompts / Prompts stored in the target.txt configuration file, such as "M8 bolt").

[0097] The system controls a visual recognition model (preferably using the YOLO-world object detection algorithm in this embodiment) to perform multimodal fusion calculations on local cue information and local binocular images. This visual recognition model accurately retrieves, matches, and delineates the two-dimensional bounding box of the target object corresponding to the textual semantics in the local image. Subsequently, the system calculates the mean geometric center of the two-dimensional bounding box and directly determines the center pixel coordinates of this bounding box as the "location cue point".

[0098] Next, the system uses the calculated location cue point as a positive pixel guide seed input into the image segmentation algorithm (in this embodiment, the SAM image segmentation algorithm is preferred). The image segmentation algorithm uses this location cue point as its core focus, performing regional pixel feature clustering and boundary growth within a multi-scale image space. This achieves pixel-level precise separation between the object's projection surface and the background worktable surface, thereby segmenting and extracting a highly reliable local mask that most realistically reflects the object's physical geometric edge projection.

[0099] This step utilizes a cascaded multimodal detection network (YOLO-world) and a large interactive image model (SAM) to achieve adaptive end-to-end extraction of "text prompt words → 2D bounding box center → location prompt points → refined local mask" under zero-sample conditions. This eliminates the need for complex classical image operators and hard-coded threshold adjustments, greatly enhancing the precise positioning flexibility and anti-interference capability of humanoid robots when facing irregularly shaped and non-standard workpieces.

[0100] It should be noted that the specific implementation principle and algorithm processing flow of using local cue information to perform target detection, cue point extraction, pixel-level image segmentation, and morphological processing including dilation and erosion on the local binocular image to obtain the local mask in this step are exactly the same as the method of extracting the global two-dimensional mask of the target object in step 110 above. The technical details and the corresponding morphological denoising process will not be repeated here.

[0101] (3) Extract the edge pixels within the local mask to form an edge pixel set.

[0102] To significantly improve the computational speed of geometric solution without sacrificing pose accuracy, this embodiment does not directly use all solid pixels within the local mask. Instead, it utilizes a first- or second-order differential contour operator to extract only the outermost closed edge pixels of the local mask, thereby constructing a high-density set of edge pixels.

[0103] Suppose that the set of edge pixels contains N sampled edge pixels, and their respective two-dimensional pixel coordinates on the image are represented as follows: ; in, is the i-th closed boundary feature pixel that has been extracted in the local binocular image.

[0104] N is the total number of effective pixels that constitute the outer envelope edge of the object's two-dimensional plane.

[0105] (4) Calculate the mean center of the edge pixel set, and construct the covariance matrix of the two-dimensional geometric distribution based on the coordinate deviation of each edge pixel relative to the mean center.

[0106] Calculate the mean center (i.e., centroid) of the entire closed envelope curve in the image coordinate system. Subsequently, a two-dimensional deviation distribution matrix is ​​constructed by performing a mean shift (mean-neutralization) operation on all edge sampling points. This eliminates the influence of original image translation on rotational features. Based on this, a symmetric covariance matrix characterizing the two-dimensional geometric discrete distribution features of the object's contour is constructed. .

[0107] Calculate the two-dimensional mean center (geometric centroid) of the edge pixel set: ; By using the mean center to eliminate global translation, a 2×2 covariance matrix is ​​constructed to describe the rotational diffusion characteristics of the image. : ; in: ; ; The mean value of each discrete coordinate of the local mask edge curve represents the origin of rotational symmetry of the object in the two-dimensional image.

[0108] : A 2×2 real symmetric matrix representing the discrete state and skew angle of local boundary pixels.

[0109] : Variance of the spatial coordinate distribution of edge image points in the horizontal and vertical directions.

[0110] The covariance of edge image points in the horizontal and vertical directions, its non-zero state directly reflects the rotation and skew of the object in the image.

[0111] (5) Perform eigenvalue decomposition on the covariance matrix and define the eigenvector corresponding to the largest eigenvalue as the main direction geometric feature of the local mask.

[0112] For the symmetric covariance matrix describing the divergence characteristics of local boundaries Perform eigenvalue decomposition and solve the following multidimensional eigenvalue equation:

[0113] Since the eigenvector corresponding to the largest eigenvalue represents the direction of the most discrete and elongated distribution of the point set (i.e., the length direction of the object's two-dimensional circumscribed matrix), the calculated largest eigenvalue... The corresponding orthogonal eigenunit vector Defined as the main direction geometric feature of the local mask.

[0114] Solving the simultaneous characteristic equations yields two geometric eigenvalues ​​and their corresponding eigenvectors: ; Extract the corresponding largest eigenvalue from the decomposition results. First eigenvector : ; : Scalar eigenvalues ​​derived from covariance decomposition.

[0115] : The first eigenvector corresponding to the direction of maximum variance, which mathematically represents the physical principal direction of the narrowest length (length direction) of the local mask on the image plane.

[0116] The smallest eigenvalue represents the width direction of the rectangle.

[0117] : The dimensionless component values ​​of the first feature vector on the physical horizontal and vertical axes of the image.

[0118] (6) Calculate the orientation angle of the main direction geometric features in the image coordinate system using inverse trigonometric functions, so as to serve as the rotation calibration angle of the end gripper.

[0119] To calculate the angle that the gripper ultimately needs to compensate for in its rotational degree of freedom, the first feature vector is extracted using a two-parameter arctangent function. The corresponding slope is used to calculate the initial absolute deflection angle of the major axis of the target object relative to the reference axis of the image coordinate system. .

[0120] To ensure that the two straight gripping fingers of the gripper are parallel and aligned with the physical surface of the target object, alignment must be performed in a direction perpendicular to the long axis. Therefore, the absolute deflection angle... Subtract the right angle radian value (i.e. The final rotation calibration angle of the gripper is calculated. .

[0121] Calculating the absolute direction deflection using the two-parameter arctangent function : ; Subtracting the right-angle radian offset, the final rotation calibration angle is calculated. : ; : The angle of the target's long axis relative to the horizontal axis of the current wrist camera image's physical coordinate reference.

[0122] : Differential calibration angle value for controlling the end effector to perform self-rotation correction (the system uses this angle to adaptively determine whether the camera and gripper perform clockwise or counterclockwise rotation compensation, so that the gripper finger surfaces achieve 100% zero tilt surface contact when closing).

[0123] Step 140: Calculate the spatial coordinates of the target object based on the local depth data corresponding to the local mask. Using the spatial coordinates as a reference, apply a position offset along the gripping feed direction in combination with the size of the end gripper to generate the end gripping coordinates.

[0124] This step utilizes the precise local mask and local depth data extracted in step 130 above to calculate the centroid coordinates of the target object in three-dimensional space, and introduces actuator physical size compensation to generate the final anti-collision grab coordinates.

[0125] In practice, the system uses the local mask of the target object output by the image segmentation algorithm in the previous step as a pixel index mask, mapping it to pixel-aligned local depth data (depth matrix) to "extract" the set of local depth values ​​belonging only to the surface region of the target object. Next, the system obtains the intrinsic parameters of the end-point local vision camera (including horizontal focal length, vertical focal length, and principal pixel coordinates), and uses a pinhole camera inverse projection model to map the two-dimensional coordinates of each pixel within the local mask. Combined with its corresponding true depth value Batch reconstruction of three-dimensional spatial points in the local camera coordinate system Subsequently, the system queries the robot's current coordinate transformation tree (TF Tree), retrieves the real-time extrinsic parameter matrix of the end-effector local camera relative to the world coordinate system, and uniformly transforms the high-precision 3D point set to the global reference world coordinate system. Based on this, the system calculates the spatial geometric center of all transformed 3D points, that is, it solves for the 3D centroid coordinates of the target object through a 3D mean filtering algorithm, and determines the centroid coordinates as the spatial coordinates of the target object.

[0126] After obtaining the spatial coordinates of the target object, the system executes an adaptive position offset and gripping coordinate generation pipeline based on the physical dimensions of the gripper. Since the calculated spatial coordinates are located on the physical surface or center of the target object, if the robotic arm directly uses these coordinates as the gripper's closing center, the tip or outer side of the open end-effector is highly likely to collide irreversibly with the target object during the final feeding approach stage, leading to material damage or gripping failure. To eliminate such collisions at the physical control level, the system introduces a dynamic position offset mechanism: the system first obtains the current axis of the end effector corresponding to the initial approach posture (e.g., the gripper coordinate system's...). The axis is determined as the unique gripping feed direction; simultaneously, the system reads the pre-configured end-effector size parameters (specifically, the physical length of the gripper fingers in this embodiment) from the register. Next, the system moves in the opposite direction of the gripping feed direction (i.e., the direction in which the robotic arm retracts and moves away from the material) to measure the physical length dimension. The mapping is expressed as a three-dimensional spatial position offset. Finally, the system uses the spatial coordinates of the target object as the three-dimensional origin and applies the position offset in the opposite direction of the gripping feed direction to perform a homogeneous translation transformation (i.e., subtractive displacement compensation along the current feed axis), thereby synthesizing the final end-effector gripping coordinates. This design ensures that when the subsequent control system drives the robotic arm to these end-effector gripping coordinates, the gripper's "palm (gripping center)" perfectly encloses the target object's spatial centroid, while the claw tips on both sides are safely suspended above the target object before closing. This achieves a collision-free, highly robust, and precise gripping pose closed loop at the algorithm's underlying level.

[0127] Based on the above principles, in a specific implementation, calculating the spatial coordinates of the target object based on the local depth data corresponding to the local mask can be achieved through the following steps: (1) Query and extract the local depth value of each pixel in the local mask in the aligned local depth image.

[0128] To calculate the highly noise-resistant and accurate object center position, this method performs 3D reconstruction on each pixel within the local mask individually. Let the local mask contain M pixel sampling points, and its 2D pixel coordinate set be represented as... The system sequentially queries and extracts each pixel from the pixel-aligned local depth image. Corresponding local depth value .

[0129] (2) Combining the intrinsic parameters of the local vision camera at the end, the two-dimensional coordinates of each pixel in the local mask and the corresponding local depth value are restored to three-dimensional points in the local camera coordinate system using the inverse projection model. The three-dimensional points are then transformed to the world coordinate system using the coordinate transformation tree to calculate the three-dimensional mean center, so as to generate the spatial coordinates of the target object.

[0130] This step utilizes the camera's geometric imaging principle to map each discrete pixel within the mask into three-dimensional spatial coordinates.

[0131] First, obtain the camera intrinsic parameter matrix K of the wrist camera: ; Using a pinhole camera back projection model, the coordinates of each pixel are... and its depth Restored to a 3D spatial point in the local wrist camera coordinate system : ; After calculating the 3D coordinates of each pixel within the 2D mask in the camera coordinate system using the inverse projection model of the wrist camera, the calculated 3D spatial points of the camera are then transformed using a coordinate transformation tree. Transform to the world coordinate system to obtain the corresponding three-dimensional coordinate points in the world coordinate system. The specific formula for transforming the coordinates of a single point is as follows: ; In this embodiment, to achieve highly robust and accurate centroid calculation, each pixel sampling point within the local mask is substituted into the above formula for coordinate transformation. To mathematically distinguish the calculation results for each point, the calculation result for the first pixel within the local mask is... The world coordinates of each pixel after transformation are represented as follows: ,in , This represents the total number of pixel sampling points within the local mask. Therefore, by calculating the 3D mean center of the transformed 3D point cloud set, the sampling labels are eliminated. To calculate and generate the final, unique spatial coordinates of the target object. The specific formula for calculating its mean centroid is as follows: ; mean center here The spatial geometric centroid that physically represents the target object is the final spatial coordinate of the target object obtained in this step. : Represents the three-dimensional spatial coordinate vector of the pixel within the two-dimensional mask of the target object in the wrist (local) camera coordinate system; : These represent the x-coordinate, y-coordinate, and depth value (distance along the camera's optical axis) of the pixel in the wrist camera coordinate system, respectively. : Represents the spatial coordinate vector of a 3D point in the wrist camera coordinate system after transformation to the preset world coordinate system; : These represent the x-coordinate, y-coordinate, and altitude coordinates of the point in the world coordinate system, respectively; : This represents the wrist camera coordinate system relative to the world coordinate system, provided in real-time by the coordinate transformation tree. The rotation matrix parameter is used to characterize the spatial attitude deflection relationship between two coordinate systems; : This represents the wrist camera coordinate system relative to the world coordinate system, provided in real-time by the coordinate transformation tree. Translation vector parameters are used to characterize the spatial displacement deviation between the origins of two coordinate systems; : Represents the traversal index number of the pixel sampling point within the local mask ( ); : Indicates the total number of pixel sampling points contained within the currently extracted local mask of the target object; : indicates the first in the local mask Transform each pixel into a 3D spatial coordinate vector in the world coordinate system; : respectively represent the first The component coordinates of each pixel in the world coordinate system; : This represents the final and unique spatial centroid coordinate vector of the target object after performing coordinate transformation on all pixels within the mask and removing noise.

[0132] Based on the above principle, using spatial coordinates as a reference and combining the dimensions of the end effector gripper with a position offset applied along the gripping feed direction to generate end effector gripping coordinates, this can be specifically achieved through the following steps: (1) Determine the end effector feed axis corresponding to the initial approach posture of the end effector as the gripping feed direction; (2) Obtain the physical length dimension of the end gripper and map the physical length dimension into a position offset in the opposite direction of the gripping feed direction.

[0133] The robot needs to control the gripper to enclose the physical center of mass of the object. However, there is a fixed length between the fingertips of the grippers and their control reference point (the center of the wrist flange). Therefore, the system determines the axis (end-effector feed axis) corresponding to the "initial end-effector approach posture" (i.e., flange approach orientation) calculated in step 120 as the gripping feed direction. The physical length of the end-effector gripper along the feed axis is then obtained. To prevent the wrist from directly impacting the object, the physical length must be adjusted in the opposite direction of the feed direction (i.e., the yield direction). Mapped to the recoil position offset vector in the world coordinate system: ; in, It is the unit vector of the feed axis in the world coordinate system corresponding to the initial approach posture at the end point.

[0134] (3) Using the spatial coordinates as the three-dimensional origin, apply a position offset along the gripping feed direction to perform coordinate translation transformation to generate the end gripping coordinates.

[0135] Using the target object's physical center of mass that was just calculated Using the origin as the reference point for translation calculation, a three-dimensional homogeneous coordinate translation is performed by superimposing position offsets in the opposite direction, thereby dynamically synthesizing the end-effector gripping coordinates that the robotic arm's end-effector wrist should actually move to. : ; The robotic arm feeds horizontally along the X-axis (forward axis) in the world coordinate system, at which point the feed axis unit vector... The offset directly acts on the X-axis component. Therefore, the final summation formula for the above homogeneous coordinate translation transformation is: ; Through this translation coordinate Combined with the previously calculated rotation calibration angle, the robot ultimately generates an extremely precise, fully adaptive 6-DoF final grasping pose. The controller sends a command, and the robotic arm smoothly moves to that point and closes its gripper, perfectly executing the grasping motion.

[0136] The physical geometric length parameter of the end effector gripper along the feed axis.

[0137] : The three-dimensional backoff offset vector calculated in the world coordinate system.

[0138] The calculated end-effector coordinates used to control the movement of the robotic arm's wrist (this point ensures that when the robotic arm's wrist reaches this position, the two fingertips of the gripper perfectly enclose the object's physical center of mass on both sides). ).

[0139] Step 150: Generate a precise grasping pose based on the rotation calibration angle and end-effector grasping coordinates to control the robotic arm to grasp the target object.

[0140] The system performs a unified spatial geometric cross-modal mapping between the "end-effector grasping coordinates" (i.e., 3D translation control quantities) with the "final grasping posture parameters" (i.e., 3D rotation control quantities) after secondary local fine-tuning calibration by the hand-eye camera, synthesizing them into a precise grasping pose with six degrees of freedom (6-DoF) constraints. In physical essence, this action is to simultaneously lock the "spatial endpoint trajectory" and the "penetration engagement angle" leading to the grasping target by the end effector of the robotic arm.

[0141] Through this step, the robot system no longer needs to adjust its position and angle step by step. Instead, it can directly deliver the synthesized, precise grasping pose to the underlying motion controller (such as an inverse kinematics engine), which calculates the target control pulses for each joint motor of the robotic arm during its movement in one go. Finally, the system sends drive commands through the bus to control the robotic arm to smoothly and seamlessly move the end effector gripper to the target position without any physical collisions. It then precisely drives the gripper to close, transforming the complex grasping task, which was originally prone to failure due to changing environments and disordered material placement, into a highly robust and safe automated standard process. This perfectly achieves the technical goal of fully autonomous, collision-free, and precise grasping.

[0142] Based on the above principles, step 150 can be implemented in the following steps: (1) The calculated end-grabbing coordinates are determined as the spatial translation control parameters in the precise grabbing pose.

[0143] After the control system achieves a translation endpoint that does not trigger physical collisions and high-precision fine-tuning of the rotation angle, this embodiment executes the final six-degree-of-freedom (6-DoF) control parameter cascading and fusion pipeline. The system first performs parameter extraction, converting the end-effector coordinates calculated in the previous step... It is directly mapped to the spatial translation control parameters in the robot's spatial motion equations.

[0144] (2) The rotation calibration angle is used as the angle compensation amount. The rotation amount of the initial approach posture of the end is superimposed along the feed axis corresponding to the initial approach posture of the end, so as to correct the initial approach posture of the end to the final grasping posture parameters.

[0145] In practical implementation, the system first obtains the initial approach attitude of the end effector calculated in the coarse positioning stage. In this embodiment, the initial attitude is oriented with the flange pointing to the quaternion. Mathematical representation is then performed. Subsequently, the system acquires the two-dimensional plane rotation calibration angle calculated by the hand-eye camera at the pre-grasping point. This angle is used as the angle compensation amount for performing angle fine-tuning corrections throughout the entire space.

[0146] In order for the finger surfaces of the end effector to be precisely aligned with the edge contour of the target object, the system must drive the end effector to precisely rotate along the feed axis corresponding to the current initial approach posture of the end effector (i.e., the current center axis of the end effector). Angle. Within the quaternion control space, this fine-tuning motion of rotation along a specific axis is achieved by constructing a unit vector about the feed axis. And the corner is Incremental calibration quaternion To achieve this, the incremental calibration quaternion The specific mathematical formula for its construction is as follows: ; In constructing the incremental calibration quaternion Subsequently, the system performs a superposition operation of rotations within the quaternion space. Since the continuous composition of three-dimensional rotations mathematically corresponds to non-commutative quaternion multiplication, the system will use the initial approach attitude quaternion of the end effector. With incremental calibration quaternions Perform quaternion cross product (denoted as ) This allows the rotation calibration constraint to be seamlessly superimposed onto the initial pointing attitude, thereby generating the final grasping attitude quaternion that integrates both "pointing alignment" and "rotation calibration" constraints. The quaternion rotational superposition formula it strictly follows is as follows: ; Finally, the system uses the Rodriguez transform or quaternion-to-matrix mapping to transform the calculated final grasping posture quaternion. Convert to the corresponding 3D orthogonal rotation transformation matrix The rotation matrix is ​​then determined as the final grasping posture parameter, which is used to participate in the final synthesis of the subsequent six-degree-of-freedom homogeneous matrix.

[0147] The parameters are described below: : Represents the initial approach attitude quaternion calculated in the preceding steps, before the rotation fine-tuning correction has been performed; : This refers to the rotational calibration angle that the hand-eye camera finely identifies at the pre-grasping point, used to eliminate the angular deviation between the gripper finger surface and the target object contour; : Represents the unit vector of the feed axis along which the end effector is currently approaching the target, and serves as the rotation axis vector for spin fine-tuning in quaternion space; : Represents a unit vector based on the feed axis With rotation calibration angle The constructed incremental calibration quaternion; : Represents Hamiltonian operator multiplication (quaternion multiplication) in quaternion space, used mathematically to characterize the continuous composition and superposition of two three-dimensional spatial rotations; : Represents the final gripping posture quaternion generated after the rotation amount is superimposed and corrected, which is used to constrain the gripper to the most perfect orientation at the moment of gripping; : Represents the quaternion of the final grasping posture The converted The spatial orthogonal rotation transformation matrix is ​​the final grasping posture parameter.

[0148] (3) Use the spatial translation control parameters as translation vectors and the final grasping posture parameters as rotation transformation matrices to perform homogeneous matrix synthesis operation, and determine the synthesized homogeneous transformation matrix as the precise grasping posture.

[0149] After obtaining the spatial translation control parameters and the final grasping posture parameters, the control system performs the highest-level synthesis of spatial kinematics to eliminate the decoupling uncertainty between position control and posture control, thereby sending a unified six-degree-of-freedom (6-DoF) pose control word to the robot joint controller.

[0150] In the specific synthesis process, the system utilizes projective geometry and the principle of homogeneous transformation to construct a... Special Euclidean group of dimensions Homogeneous transformation matrix. The system will use the previously calculated matrix... Grab the coordinate vector at the end of the dimension As a spatial translation control parameter, it is nested and filled in the upper right corner of the homogeneous transformation matrix. Within the sub-block, the physical landing point of the robot's wrist center in Cartesian space is used to absolutely constrain it; simultaneously, the system will determine the previous step from the final grasping posture quaternion. The converted Spatial orthogonal rotation transformation matrix As the final grasping posture parameters, they are nested and filled in the upper left corner of the homogeneous transformation matrix. Within the sub-block, the axial spatial orientation is used to precisely constrain the moment the grippers close. The bottom row of the matrix is ​​filled with the standard homogeneous projection zero row vector. With a scale factor of 1, through the matrix mapping and nested combination of these multimodal parameters, the system dynamically synthesizes a homogeneous transformation matrix representing the six-degree-of-freedom spatial state, and directly determines this matrix as the precise grasping pose. The strict formula for homogeneous matrix composition that it follows is as follows: ; The parameters are described below: : Represents the quaternion of the final grasping posture The converted The spatial orthogonal rotation transformation matrix is ​​the final grasping posture parameter.

[0151] : Represents the end-grabbing coordinate vector with the physical length offset of the grippers ( (in column vector form), used as translation sub-blocks of a homogeneous matrix to constrain position; : These represent the three spatial translation control components of the end-capture coordinates in the world coordinate system, where The physical relief compensation parameter applied along the feed axis; :express The homogeneous projection of the zero row vector of dimension, i.e. This is used to meet the matrix dimension alignment requirements of homogeneous coordinate transformation; : Indicates the final synthetic construction The homogeneous transformation matrix in the 3D space, which is the precise grasping pose finally generated by the scheme of this invention, highly unifies the 3D translation and 3D rotation in the grasping task, and serves as the direct input command for the inverse kinematics solution of the robotic arm joints. ; These represent the final grasping posture rotation matrices. The nine three-dimensional spatial direction cosine components.

[0152] The system uses the synthesized homogeneous transformation matrix Once the precise grasping pose is determined, it is directly published to the robot's hardware control bus as the target endpoint parameter for the final motion planning. The robot's underlying controller then calls... The system also runs the inverse kinematics (IK) solution in real time in the background, converting it into target operating pulses for the robot's joint motors on each axis, guiding the coordinated movement of the seven joint motors. This control mechanism physically ensures that when the robotic arm's wrist center translates without collision, the motion is achieved. When the anti-collision space position is represented, the two finger surfaces of the gripper rotate precisely and synchronously in the axial direction to the specified position. The corresponding alignment posture. Finally, the fingertips of the two grippers perfectly close in a state of safely enveloping the physical center of mass of the target object, achieving a high-precision, fully adaptive, autonomous and stable grasping effect.

[0153] This application employs a two-stage visual guidance framework combining "global coarse localization and local fine grasping," effectively integrating the geometric calculation capabilities of traditional 3D vision with the perceptual flexibility of artificial intelligence algorithms. This overcomes the dual shortcomings of single-vision solutions in complex industrial scenarios, such as low accuracy, poor real-time performance, and insufficient flexibility, as well as the unstable output and low reliability of single-AI solutions. In the global stage, a visual recognition model extracts a 2D mask of the target object from the image and maps it to a spatial point cloud using depth data. By analyzing the spatial geometric features of the point cloud, the spatial arrangement of the target object is adaptively determined, thereby generating pre-grabbing spatial points for collision avoidance and the initial approach posture of the end effector. This solves the problem that traditional methods cannot simultaneously address obstacle avoidance safety and scene adaptability, achieving flexible perception and safe approach to irregularly shaped, stacked, and randomly oriented materials. In the local phase, high-resolution local images and depth data are acquired through an end-effector local vision camera. Local masks are extracted, and their principal direction geometric features are calculated to obtain the rotation calibration angle of the end-effector gripper. Simultaneously, the precise spatial coordinates of the target object are calculated based on the local depth data, and gripper size offsets are applied to generate end-effector grasping coordinates. The joint calculation of 2D texture features and 3D spatial information achieves high-precision calibration of the target object's six-DOF pose, effectively avoiding the inference drift and feature mismatch risks of a single AI model in non-standard industrial conditions. The two phases work together to significantly enhance the system's reliability and versatility while ensuring grasping accuracy, meeting the stringent requirements of high-precision, high-stability flexible grasping in industrial mass production scenarios.

[0154] Based on the same inventive concept, as an implementation of the above-mentioned visual positioning-based robot precision grasping method, this application also provides a visual positioning-based robot precision grasping system. Figure 2 This is a structural diagram of the system in an embodiment of this application. See also... Figure 2 As shown, the system includes: The spatial point cloud generation module is used to acquire image data and depth data from a global vision camera, extract a two-dimensional mask of the target object from the image data using a visual recognition model, and map the two-dimensional mask into a spatial point cloud by combining the depth data. The first control module is used to analyze the spatial geometric features of the spatial point cloud to determine the spatial arrangement of the target object, adaptively generate a pre-grabbing spatial point and an initial approach posture of the end effector to avoid collision based on the spatial arrangement, and control the robotic arm to move to the pre-grabbing spatial point. The rotation calibration angle calculation module is used to acquire local image data and local depth data through the end-effector local vision camera, extract the local mask of the target object, calculate the main direction geometric features of the local mask, and obtain the rotation calibration angle of the end gripper. The end-effector coordinate generation module is used to calculate the spatial coordinates of the target object based on the local depth data corresponding to the local mask, and to generate the end-effector coordinates by applying a position offset along the gripping feed direction in combination with the size of the end gripper, based on the spatial coordinates. The second control module is used to generate a precise grasping pose based on the rotation calibration angle and the end-grabbing coordinates, so as to control the robotic arm to grasp the target object.

[0155] This application employs a two-stage visual guidance framework combining "global coarse localization and local fine grasping," effectively integrating the geometric calculation capabilities of traditional 3D vision with the perceptual flexibility of artificial intelligence algorithms. This overcomes the dual shortcomings of single-vision solutions in complex industrial scenarios, such as low accuracy, poor real-time performance, and insufficient flexibility, as well as the unstable output and low reliability of single-AI solutions. In the global stage, a visual recognition model extracts a 2D mask of the target object from the image and maps it to a spatial point cloud using depth data. By analyzing the spatial geometric features of the point cloud, the spatial arrangement of the target object is adaptively determined, thereby generating pre-grabbing spatial points for collision avoidance and the initial approach posture of the end effector. This solves the problem that traditional methods cannot simultaneously address obstacle avoidance safety and scene adaptability, achieving flexible perception and safe approach to irregularly shaped, stacked, and randomly oriented materials. In the local phase, high-resolution local images and depth data are acquired through an end-effector local vision camera. Local masks are extracted, and their principal direction geometric features are calculated to obtain the rotation calibration angle of the end-effector gripper. Simultaneously, the precise spatial coordinates of the target object are calculated based on the local depth data, and gripper size offsets are applied to generate end-effector grasping coordinates. The joint calculation of 2D texture features and 3D spatial information achieves high-precision calibration of the target object's six-DOF pose, effectively avoiding the inference drift and feature mismatch risks of a single AI model in non-standard industrial conditions. The two phases work together to significantly enhance the system's reliability and versatility while ensuring grasping accuracy, meeting the stringent requirements of high-precision, high-stability flexible grasping in industrial mass production scenarios.

[0156] This application also provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, such as a program for a vision-based robot precision grasping method. When the processor executes the computer program, it implements all the steps of the aforementioned vision-based robot precision grasping method, such as S110 to S150 and their sub-steps. Alternatively, when the processor executes the computer program, it implements the functions of each module / unit in the aforementioned system embodiments.

[0157] For example, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device.

[0158] The electronic device may be a computing device such as a server or cloud server deployed in a vision-based robot precision grasping system. The electronic device may include, but is not limited to, processors and memory. Those skilled in the art will understand that the above components are merely examples of electronic devices and do not constitute a limitation on the electronic device. It may include more or fewer components than described above, or combine certain components, or different components. For example, the electronic device may also include data acquisition cards and output cards for connecting sensors and actuators, network communication devices, buses, etc.

[0159] The processor can be a central processing unit, or other general-purpose processors, digital signal processors, application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (OPGs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the electronic device, connecting various parts of the entire electronic device via various interfaces and lines.

[0160] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function, etc.; the data storage area may store data created according to system usage, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart memory card, secure digital card, flash memory card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0161] Wherein, if the modules / units integrated in the electronic device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0162] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for precise grasping of a robot based on vision localization, characterized in that, The method includes: Acquire image data and depth data from a global vision camera, use a visual recognition model to extract a two-dimensional mask of the target object from the image data, and combine the two-dimensional mask with the depth data to map a spatial point cloud; The spatial geometric features of the spatial point cloud are analyzed to determine the spatial arrangement of the target object. Based on the spatial arrangement, a pre-grabbing spatial point and an initial approach posture of the end effector are adaptively generated to avoid collisions. The robotic arm is then controlled to move to the pre-grabbing spatial point. Local image data and local depth data are acquired by the end-effector local vision camera, the local mask of the target object is extracted, the main direction geometric features of the local mask are calculated, and the rotation calibration angle of the end gripper is obtained. Based on the local depth data corresponding to the local mask, the spatial coordinates of the target are calculated. Using the spatial coordinates as a reference, a position offset is applied along the gripping feed direction in combination with the size of the end gripper to generate the end gripping coordinates. Based on the rotation calibration angle and the end-effector grasping coordinates, a precise grasping pose is generated to control the robotic arm to grasp the target object.

2. The method according to claim 1, characterized in that, The extraction of a two-dimensional mask of the target object from the image data using a visual recognition model includes: Receive text prompts for the target object; The target detection algorithm is used to perform multimodal recognition and localization on the text prompt information and the image data, so as to output the pixel region of the target object in the image data and determine the center point coordinates of the pixel region; The coordinates of the center point are used as input prompts and input into the image segmentation algorithm to generate the original two-dimensional mask of the target object. The original two-dimensional mask is subjected to morphological processing including dilation and erosion to obtain the final two-dimensional mask.

3. The method according to claim 1, characterized in that, The step of mapping the two-dimensional mask to a spatial point cloud by combining the depth data includes: Align the depth data of the global vision camera with the global image data, and then query and extract the depth values ​​corresponding to each pixel in the two-dimensional mask from the aligned depth data. Obtain the camera intrinsic parameters of the global vision camera, including the axial pixel focal length and the coordinates of the principal point of the image; Using the pinhole camera inverse projection model, based on the axial pixel focal length, the image principal point coordinates, and the depth value, the two-dimensional coordinates of each pixel in the two-dimensional mask are restored to the camera's three-dimensional spatial points in the camera coordinate system. The robot system uses a coordinate transformation tree to call up the pre-stored camera calibration parameters of the global vision camera, which include a rotation matrix and a translation vector. Using the rotation matrix and the translation vector, the camera's three-dimensional spatial points are transformed into a preset world coordinate system to construct the spatial point cloud.

4. The method according to claim 3, characterized in that, The camera calibration parameters of the global vision camera are pre-calculated and loaded into the coordinate transformation tree in the following ways: The robot is controlled to move in a variety of different postures so that the global vision camera can present a variety of different spatial poses. Simultaneously, the first pose transformation data of the end of the robotic arm relative to the base coordinate system and the image data of the fixed calibration plate captured by the global vision camera are recorded in each of the spatial poses. Based on the image data, the second pose transformation data of the calibration board relative to the global vision camera is calculated; Based on the constraint that the spatial pose of the calibration plate relative to the base coordinate system remains unchanged during the calibration process, and by combining the first pose transformation data and the second pose transformation data under each pose, multiple sets of homogeneous pose equations at different times are constructed. The homogeneous pose equations are solved simultaneously to calculate the camera calibration parameters of the global vision camera relative to the end effector of the robotic arm. The camera calibration parameters are composed of rotation matrices and translation vectors. The camera calibration parameters are then loaded as static geometric parameters into the coordinate transformation tree of the robot system.

5. The method according to claim 1, characterized in that, The process of analyzing the spatial geometric features of the spatial point cloud to determine the spatial arrangement of the target object, and adaptively generating pre-grabbing spatial points and initial approach postures for collision avoidance based on the spatial arrangement, includes: Boundary features are extracted and projected onto the spatial point cloud to determine the extension dimensions of the target object in three mutually perpendicular geometric feature directions and the geometric center point of the spatial point cloud. The geometric feature direction with the largest dimension is defined as the principal axis direction of the target object. If the main axis direction is parallel to the height axis defined in the reference coordinate system of the current robot system, the target object is determined to be in an upright posture. The geometric center point is pushed outward by a preset safety distance along the radial direction perpendicular to the height axis to generate the pre-grabbing spatial point. If the main axis direction is not parallel to the height axis, the target object is determined to be in a lying position. The geometric center point is translated by a preset safe distance along the upper part of the height axis to generate the pre-grabbing spatial point. A direction vector is constructed from the pre-captured spatial point to the geometric center point. The initial approach posture of the end point is constrained and solved by using the angle between the direction vector and the preset reference direction vector in the reference coordinate system and the normal axis.

6. The method according to claim 1, characterized in that, The process of extracting the local mask of the target object, calculating the principal direction geometric features of the local mask, and obtaining the rotation calibration angle of the end gripper includes: Receive local cue information for the target object, and control the end local vision camera to acquire local binocular images and local depth images of the target object at the pre-capture spatial point, and perform alignment operation; The visual recognition model is used to match the local prompt information and the local binocular image. The location prompt point corresponding to the target object is calculated in the local binocular image. The location prompt point is used as a guiding prompt and input into the image segmentation algorithm to segment and extract the local mask of the target object. Extract the edge pixels within the local mask to form an edge pixel set; Calculate the mean center of the edge pixel set, and construct the covariance matrix of the two-dimensional geometric distribution based on the coordinate deviation of each edge pixel relative to the mean center; The covariance matrix is ​​decomposed into eigenvalues, and the eigenvector corresponding to the largest eigenvalue is defined as the main direction geometric feature of the local mask. The orientation angle of the main direction geometric feature in the image coordinate system is calculated using inverse trigonometric functions, and used as the rotation calibration angle of the end gripper.

7. The method according to claim 1, characterized in that, The calculation of the spatial coordinates of the target object based on the local depth data corresponding to the local mask includes: Query and extract the local depth value of each pixel within the local mask in the aligned local depth image; By combining the intrinsic parameters of the local vision camera at the end, the two-dimensional coordinates and corresponding local depth values ​​of each pixel in the local mask are restored to three-dimensional points in the local camera coordinate system using the inverse projection model. The coordinate transformation tree is then used to transform the three-dimensional points to the world coordinate system for three-dimensional mean center calculation to generate the spatial coordinates of the target object.

8. The method according to claim 7, characterized in that, The step of generating end-effector coordinates by applying a position offset along the gripping feed direction based on the spatial coordinates and in combination with the size of the end-effector includes: The end effector feed axis corresponding to the initial approach posture of the end effector is determined as the gripping feed direction; Obtain the physical length dimension of the end gripper, and map the physical length dimension into a position offset in the opposite direction of the gripping feed direction; Using the spatial coordinates as the three-dimensional origin, the position offset is applied along the gripping feed direction to perform coordinate translation transformation, thereby generating the end-grabbing coordinates.

9. The method according to claim 1, characterized in that, The step of generating a precise grasping pose based on the rotation calibration angle and the end-effector grasping coordinates includes: The calculated end-grabbing coordinates are determined as the spatial translation control parameters in the precise grasping pose; The rotation calibration angle is used as the angle compensation amount. The rotation amount of the initial approach posture of the end is superimposed along the feed axis corresponding to the initial approach posture of the end, so as to correct the initial approach posture of the end to the final grasping posture parameters. The spatial translation control parameters are used as translation vectors, and the final grasping posture parameters are used as rotation transformation matrices. Homogeneous matrix synthesis is performed, and the synthesized homogeneous transformation matrix is ​​determined as the precise grasping pose.

10. A vision-based positioning-based robot precision grasping system, characterized in that, The system includes: The spatial point cloud generation module is used to acquire image data and depth data from a global vision camera, extract a two-dimensional mask of the target object from the image data using a visual recognition model, and map the two-dimensional mask into a spatial point cloud by combining the depth data. The first control module is used to analyze the spatial geometric features of the spatial point cloud to determine the spatial arrangement of the target object, adaptively generate a pre-grabbing spatial point and an initial approach posture of the end effector to avoid collision based on the spatial arrangement, and control the robotic arm to move to the pre-grabbing spatial point. The rotation calibration angle calculation module is used to acquire local image data and local depth data through the end-effector local vision camera, extract the local mask of the target object, calculate the main direction geometric features of the local mask, and obtain the rotation calibration angle of the end gripper. The end-effector coordinate generation module is used to calculate the spatial coordinates of the target object based on the local depth data corresponding to the local mask, and to generate the end-effector coordinates by applying a position offset along the gripping feed direction in combination with the size of the end gripper, based on the spatial coordinates. The second control module is used to generate a precise grasping pose based on the rotation calibration angle and the end-grabbing coordinates, so as to control the robotic arm to grasp the target object.

Images