Method and apparatus for determining operational pose of robot end effector

Abstract

The application relates to the technical field of robot positioning, and particularly discloses a robot end effector operation pose determination method and device. According to the application, the mobile navigation correction and the operation pose correction can be synchronously completed through a side camera of a machine body, the camera does not need to be carried on the end of the mechanical arm, the environment markers can be identified in real time during the movement, and the navigation cumulative error can be corrected, so that the robot positioning accuracy is ensured, the same marker is used to calculate the parking deviation matrix to complete the overall compensation of the operation pose, the mechanical arm does not need to take a second photo for calibration, the work beat is compressed, the pose determination efficiency is improved, and the mechanical arm work efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of robot positioning technology, and in particular to a method and apparatus for determining the operating pose of a robot end effector. Background Technology

[0002] In scenarios such as semiconductor wafer handling, 3C electronics tray loading and unloading, and warehousing logistics, composite robots (AGV mobile chassis + robotic arm) need to perform high-precision grasping or placement operations after moving to the correct position. Existing technologies typically utilize two sets of calibration and correction schemes: one for robot navigation and positioning markers, and another for robotic arm grasping and guiding markers, to correct navigation and positioning as well as determine the robotic arm's posture. Composite robots rely on laser SLAM, ground markers, and reflectors for navigation and positioning. Since navigation and positioning accuracy is typically ±5-10mm, while robotic arm grasping accuracy requirements are typically ±1-2mm, there is a significant accuracy gap between the two. After the robot arrives at the workstation, it must re-photograph the markers on the worktable using a side-mounted camera at the robotic arm's end effector or on the arm, and perform hand-eye calibration or posture compensation calculations before executing the grasping action. This results in redundant operation cycles, low posture determination efficiency, and ultimately, low robotic arm operation efficiency.

[0003] Therefore, how to improve the efficiency of pose determination while ensuring pose accuracy, and thus improve the working efficiency of the robotic arm, has become an urgent problem to be solved. Summary of the Invention

[0004] This application provides a method and apparatus for determining the operating pose of a robot end effector, so as to improve the pose determination efficiency while ensuring pose accuracy, thereby improving the working efficiency of the robotic arm.

[0005] In a first aspect, this application provides a method for determining the operating pose of a robot end effector, the method comprising: During robot movement, the poses of markers in the moving environment are detected by a side-mounted camera, and the robot's pose is corrected based on the poses of the markers to obtain the precise pose of the robot body. When the precise pose of the robot body is within the range of the target pick-and-place station, the pose of the target marker corresponding to the target pick-and-place station is acquired based on the side-mounted camera, and the docking deviation matrix of the robot is determined based on the pose of the target marker. The pose of the object to be operated on and the first pose transformation relationship are obtained. Based on the docking deviation matrix, the pose of the object to be operated on and the first pose transformation relationship, the operating pose of the end effector is determined. The first pose transformation relationship is the pose transformation relationship of the end effector from the base coordinate system to the camera coordinate system.

[0006] Secondly, this application also provides an operating pose determination device for a robot end effector, the device comprising: The robot pose correction module is used to detect the pose of markers in the moving environment based on a side-mounted camera during robot movement, and to correct the robot pose based on the pose of the markers to obtain the precise pose of the robot body. The docking deviation acquisition module is used to acquire the target marker pose corresponding to the target pick-and-place station based on the side-mounted camera when the precise pose of the robot body is within the range of the target pick-and-place station, and to determine the docking deviation matrix of the robot based on the target marker pose. The operation pose acquisition module is used to acquire the pose of the operation object and the first pose transformation relationship. Based on the docking deviation matrix, the pose of the operation object and the first pose transformation relationship, the operation pose of the end effector is determined. The first pose transformation relationship is the pose transformation relationship of the end effector from the base coordinate system to the camera coordinate system.

[0007] Thirdly, this application also provides a computer device, the computer device including a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and, when executing the computer program, implement the above-described method for determining the operating pose of a robot end effector.

[0008] Fourthly, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the above-described method for determining the operational pose of a robot end effector.

[0009] This application discloses a method and apparatus for determining the operating pose of a robot end effector. During robot movement, the poses of markers in the moving environment are detected by a side-mounted camera, and the robot's pose is corrected based on the marker poses to obtain a precise robot body pose. When the precise robot body pose is within the range of a target pick-and-place station, the pose of the target marker corresponding to the target pick-and-place station is acquired based on the side-mounted camera, and the robot's docking deviation matrix is ​​determined based on the target marker pose. The pose of the manipulated object and a first pose transformation relationship are acquired. Based on the docking deviation matrix, the pose of the manipulated object, and the first pose transformation relationship, the operating pose of the end effector is determined, wherein the first pose transformation relationship is the pose transformation relationship of the end effector from the base coordinate system to the camera coordinate system. This application can complete the mobile navigation correction and operation posture correction by using a side-mounted camera on the robot body, without the need to mount a camera at the end of the robotic arm. It can identify environmental markers in real time during movement and correct the accumulated navigation error, ensuring the robot's positioning accuracy. It can use the same marker to calculate the docking deviation matrix to complete the operation posture correction, without the need for secondary photo calibration of the robotic arm, thus reducing the operation cycle and improving the posture determination efficiency, thereby improving the operation efficiency of the robotic arm. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic flowchart of a method for determining the operating pose of a robot end effector according to the first embodiment of this application; Figure 2 This is a schematic diagram showing the positional relationship between markers and a side-mounted camera in a method for determining the operational pose of a robot end effector according to an embodiment of this application; Figure 3 This is a schematic flowchart of a method for determining the operating pose of a robot end effector according to the second embodiment of this application; Figure 4 A schematic block diagram of an operating pose determination device for a robot end effector provided for embodiments of this application; Figure 5 A schematic block diagram of the structure of a computer device provided for an embodiment of this application. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0014] It should be understood that the terminology used in this application specification is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this application specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0015] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0016] The robot end effector operation pose determination method provided in this embodiment of the invention can be applied in an operation pose determination system. It can simultaneously complete the movement navigation correction and operation pose correction by using a side-mounted camera on the robot body, without the need to mount a camera at the end of the robot arm. During the movement, environmental markers are identified in real time and navigation cumulative errors are corrected, ensuring the robot's positioning accuracy. The docking deviation matrix is ​​calculated using the same marker to complete the overall operation pose compensation, eliminating the need for secondary photo calibration of the robot arm, compressing the operation cycle, improving the pose determination efficiency, and thus improving the robot arm's operation efficiency.

[0017] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0018] Please see Figure 1 , Figure 1 This is a schematic flowchart illustrating a method for determining the operational pose of a robot end effector, as provided in an embodiment of this application. Figure 1 As shown, the method for determining the operating pose of the robot end effector specifically includes steps S101 to S104.

[0019] S101. During the robot's movement, the pose of markers in the moving environment is detected by a side-mounted camera, and the robot's pose is corrected based on the pose of the markers to obtain the precise pose of the robot body. In one embodiment, the robot may experience positioning errors during long-distance movement. Therefore, it is necessary to correct the robot's pose during movement to obtain an accurate pose.

[0020] The mobile environment includes multiple robot pick-and-place stations. As can be understood, a robot pick-and-place station refers to a machine, shelf, or similar structure where robots perform pick-and-place operations on objects.

[0021] like Figure 2 As shown, a camera is positioned on the side of the robot's body, i.e., a side-mounted camera. Markers are placed laterally along the robot's planned travel path, typically mounted on the side of the robot's pick-and-place station, such as on a wall, shelf column, or dedicated bracket. The height is the same as the mounting height of the side-mounted camera to ensure that the camera can fully capture the image of the marker. These markers can be AprilTags, ArUco Markers, etc., and each marker is assigned a unique identifier.

[0022] The side-mounted camera's optical axis faces to the side, working in conjunction with markers placed at the same height on the side or wall of the robot's pick-and-place station.

[0023] After the side-mounted camera is installed, its intrinsic parameters are calibrated to obtain the intrinsic parameter matrix and distortion coefficients. It is understandable that camera lenses exhibit radial and tangential distortion, causing pixel stretching and shift; simultaneously, the camera's focal length and imaging center lack a unified reference. Intrinsic parameter calibration is used to solve for the camera's inherent optical parameters (intrinsic parameter matrix) and distortion coefficients, eliminating systematic positioning errors introduced by the lens.

[0024] Specifically, while keeping the side-mounted camera in a fixed position, the calibration board is moved within the camera's field of view, changing its posture through various means, including switching between near and far, tilting left and right, pitching up and down, and rotating, to ensure that the calibration board appears completely in the upper left, upper right, center, lower left, and lower right areas of the image. A predetermined number of images are acquired, and after image preprocessing, the Zhang Zhengyou calibration method is used to solve for the initial intrinsic parameter matrix and initial distortion coefficients. The initial intrinsic parameter matrix and initial distortion coefficients are then optimized using CERES (Cyrus) nonlinear optimization to obtain the calibrated intrinsic parameter matrix and distortion coefficients.

[0025] In one embodiment, the marker pose includes the marker's current pose in the camera coordinate system and the marker's calibration pose in the world coordinate system, and the calibration pose is stored in a preset map database. The map database is pre-built and stores the marker's identifier, the marker's calibration pose in the camera coordinate system, the marker's calibration pose in the world coordinate system, and the pose of the manipulated object.

[0026] In one embodiment, the robot moves along a globally planned path, and a side-mounted camera continuously performs visual detection. During the robot's movement, it continuously acquires images of markers, and after noise reduction, distortion correction, and contour filtering, identifies the markers within the image and obtains the corner coordinates of the markers in the image. Then, based on the intrinsic parameter matrix and distortion coefficients of the side-mounted camera, a pose transformation is performed to obtain the current pose of the markers in the camera coordinate system.

[0027] Based on the unique identifier (such as ID) of the marker, extract the marker's pose in the world coordinate system from the map database.

[0028] The robot's body pose is corrected based on the current pose and the calibration pose to eliminate navigation drift caused by body slippage and road bumps, thereby obtaining the robot's precise body pose.

[0029] In one embodiment, such as Figure 3 As shown, during robot movement, the process of detecting the poses of markers in the moving environment using a side-mounted camera and correcting the robot's pose based on these marker poses to obtain a precise robot pose includes: S201. During the robot's movement, images of markers in the moving environment are acquired based on the side-mounted camera, and the corner coordinates of the markers in the marker images are identified based on the marker images, wherein the marker images contain markers corresponding to preset pick-up and place positions; S202. Based on the corner coordinates, the intrinsic parameter matrix of the side-mounted camera, and the distortion coefficient, perform pose transformation to obtain the current pose of the marker in the camera coordinate system; S203. Extract the coordinate pose of the marker in the world coordinate system from the map database; S204. Perform pose reasoning on the current pose and the calibration pose to obtain the robot's body pose in the world coordinate system; S205. Based on the body pose, correct the current estimated pose of the robot to obtain the precise body pose.

[0030] In one embodiment, image preprocessing of the marker image may include grayscale conversion, Gaussian filtering for noise reduction, adaptive threshold binarization, etc., to eliminate interference from ambient lighting, reflections, and ground debris. After preprocessing, the marker contours in the image are detected, and the corner coordinates of the markers are extracted.

[0031] Based on the pre-calibrated intrinsic parameter matrix and distortion coefficients of the side-mounted camera, the rotation and translation vectors of the image coordinate system relative to the camera coordinate system are calculated using either Efficient Perspective-n-Point (EPnP) or Infinitesimal Plane-based Pose Estimation (IPPE). The obtained rotation vectors are converted into rotation matrices and combined with the translation vectors to construct a coordinate transformation matrix. The corner coordinates are then transformed using the coordinate transformation matrix to obtain the current pose of the marker in the camera coordinate system. .

[0032] For example, when the marker is a QR code, all closed contours in the image are traversed, and candidate QR code regions are selected by considering contour area, quadrilateral aspect ratio, and quadrilateral interior angle constraints. The boundaries of the quadrilateral strips of the QR code are fitted, and the intersection points of the four sides are obtained to get the integer pixel coordinates of the four QR code corner points. A sub-pixel iterative algorithm can be further employed, using the minimum gray-level gradient in the neighborhood of the corner point as a constraint to improve the accuracy of the corner coordinates. The original pixel coordinates of the sub-pixel corner points are inversely distorted using distortion coefficients to eliminate pixel offsets caused by radial and tangential distortion of the lens, resulting in distortion-free standard 2D pixel coordinates. The offline preset physical side length of the QR code is read to generate fixed 3D coordinates of the four corner points in the QR code coordinate system. The rotation and translation vectors of the image coordinate system relative to the camera coordinate system are calculated using the EPnP or IPPE algorithm. The obtained rotation vector is converted into a rotation matrix and combined with the translation vector to construct a coordinate transformation matrix. The corner coordinates are transformed according to the coordinate transformation matrix to obtain the current pose of the QR code in the camera coordinate system.

[0033] In one embodiment, the second pose transformation relationship between the pre-calibrated camera coordinate system and the body coordinate system is obtained. According to the preset coordinate transformation formula The robot's body pose in the world coordinate system is obtained through reasoning. .in, This indicates the positioning pose of the marker in the world coordinate system.

[0034] In one embodiment, the current estimated pose output by the robot navigation system is corrected based on the obtained body pose to eliminate accumulated positioning errors and obtain the accurate body pose.

[0035] Further, the step of correcting the robot's current estimated pose based on the body pose to obtain the precise body pose includes: acquiring the current estimated pose and estimated covariance matrix output by the robot's navigation system; obtaining the relative distance between the side-mounted camera and the marker based on the current pose of the marker in the camera coordinate system, and determining the noise covariance matrix of the visual observation based on the relative distance; determining the Kalman gain based on the estimated covariance matrix and the noise covariance matrix; and performing Kalman filtering based on the Kalman gain, the body pose, and the current estimated pose to obtain the precise body pose.

[0036] In one embodiment, during robot movement, the navigation system continuously outputs the robot's current estimated pose in the world coordinate system and the estimated covariance matrix representing the uncertainty of odometry / SLAM (Simultaneous Localization and Mapping) positioning.

[0037] Based on the current pose of the marker in the camera coordinate system, the translation component of the pose is extracted to calculate the spatial relative distance between the side-mounted camera and the marker. Using this spatial relative distance, and pre-defined formulas for calculating the standard deviation of the position and angle components, the standard deviations of the position observation noise and angle observation noise are obtained, respectively. A noise covariance matrix for visual observation is then constructed based on these standard deviations.

[0038] The estimated covariance matrix and the noise covariance matrix are substituted into the Kalman gain calculation formula to complete the matrix operation and solve for the Kalman gain. It can be understood that this gain can adaptively balance the reliability of the navigation estimated pose and the fuselage pose obtained by visual calculation of the markers. The larger the navigation cumulative error and the higher the estimated covariance value, the more the Kalman gain is biased towards the visual observation results.

[0039] The fuselage pose calculated based on the markers is used as the observation value. Combined with the solved Kalman gain and the current estimated pose, a Kalman filter update operation is performed to synchronously correct the three-axis translation and three-axis rotation components of the current estimated pose. After the filter update operation converges, the corrected accurate fuselage pose is obtained, eliminating the cumulative positioning drift caused by odometry and long-distance SLAM driving.

[0040] Furthermore, the step of correcting the robot's current estimated pose based on the body pose to obtain the precise body pose further includes: acquiring the robot's current estimated pose; determining the visual weight of the body pose and the prediction weight of the current estimated pose based on the relative distance between the side-mounted camera and the marker; and performing a weighted fusion of the body pose and the current estimated pose based on the visual weight and the prediction weight to obtain the precise body pose.

[0041] In one embodiment, during robot movement, the navigation system continuously outputs the robot's current estimated pose in the world coordinate system.

[0042] Based on the current pose of the marker in the camera coordinate system, the translation component in the pose is extracted to calculate the spatial relative distance between the side-mounted camera and the marker. Visual weights are determined according to a preset mapping relationship between distance and weights. It is understood that the greater the distance, the greater the visual observation noise, corresponding to a smaller visual weight and a larger prediction weight; conversely, the closer the distance, the higher the visual observation accuracy, corresponding to a larger visual weight and a smaller prediction weight. The sum of the visual weight and the prediction weight is always equal to 1.

[0043] The three-axis translation parameters and quaternion rotation parameters of the fuselage pose, as well as the three-axis translation parameters and quaternion rotation parameters of the current estimated pose output by navigation, are extracted separately. Linear weighting is applied to the translation components, and spherical interpolation is used to weight and fuse the two sets of rotation quaternions.

[0044] The weighted and fused translation values ​​and the interpolated rotation quaternions are combined to form the complete fuselage pose, which is the accurate fuselage pose that eliminates the cumulative drift error.

[0045] In the above embodiments, environmental marker images are acquired in real time by a side-mounted camera during the robot's movement. The real-time camera pose of the markers is calculated based on the marker corner points and the in-camera distortion coefficients. The robot's pose is then determined by combining this with the world marker poses pre-stored in the database. This pose is used to correct the estimated pose output by the navigation system, resulting in a high-precision robot pose. This allows for flexible correction of accumulated navigation errors throughout the robot's entire movement path, continuously suppressing positioning drift caused by long-distance travel. No downtime is required, and the operation cycle is not significantly impacted, effectively improving the positioning accuracy of the robot during long-distance transport.

[0046] S102. When the precise pose of the robot body is within the range of the target pick-up and place station, the pose of the target marker corresponding to the target pick-up and place station is obtained based on the side-mounted camera, and the docking deviation matrix of the robot is determined based on the pose of the target marker. In one embodiment, the target marker pose includes the current pose of the target marker in the camera coordinate system. and the positioning pose of the target marker in the camera coordinate system The calibration pose is stored in the map database.

[0047] In one embodiment, the three-dimensional distance between the machine body and the target pick-and-place station is calculated based on the machine body's precise pose; when the translation error is within a preset range and the attitude angle deviation is less than a set angle threshold, it is determined that the machine body's precise pose is within the range of the target pick-and-place station.

[0048] When the robot's precise pose is within the range of the target pick-and-place station, the robot stops moving. The side-mounted camera acquires an image of the marker corresponding to the target pick-and-place station, and performs distortion correction, corner extraction, and PnP optimization calculations to obtain the current pose of the marker corresponding to the target pick-and-place station in the camera coordinate system. .

[0049] Based on the target marker identification, retrieve the target marker's positioning pose in the camera coordinate system from the map database. Based on the current pose , mark positioning Calculate the robot docking deviation matrix.

[0050] In one embodiment, acquiring the target marker pose corresponding to the target pick-and-place station based on the side-mounted camera, and determining the robot's docking deviation matrix based on the target marker pose, includes: acquiring a target marker image based on the side-mounted camera and identifying the corner coordinates of the target marker in the target marker image, wherein the target marker image contains a target marker corresponding to the target pick-and-place station; obtaining the current pose based on the corner coordinates, the intrinsic parameter matrix of the side-mounted camera, and the distortion coefficient; extracting the target marker's calibration pose in the camera coordinate system from a map database; and determining the docking deviation matrix based on the current pose and the calibration pose.

[0051] In one embodiment, when the robot reaches the target pick-and-place station, it stops moving, and a side-mounted camera is aimed at the fixed marker at the target pick-and-place station to acquire an original image of the target marker containing the complete marking area. Preprocessing of the target marker image may include Gaussian filtering to remove dust and lighting noise, adaptive binarization to segment the black-and-white coded region of the marker, and distinguishing the foreground of the marker from the tooling, ground background, etc., to provide a clean image for contour extraction.

[0052] The algorithm iterates through all closed contours in the binary image, filtering target marker contours based on contour area thresholds, quadrilateral aspect ratios, and interior angle constraints. It then performs a minimum quadrilateral fit on the marker contours, solving for the intersection points of the four boundaries to obtain the coarse integer pixel coordinates of the four marker corner points. Alternatively, a sub-pixel iterative algorithm can be used, with the gray-level gradient of the corner point's neighborhood as a constraint, to iteratively optimize and improve the accuracy of the corner coordinates. Finally, it combines camera intrinsic parameters and distortion coefficients to determine the current pose of the target marker in the camera coordinate system.

[0053] Specifically, the camera intrinsic parameter matrix and distortion coefficients stored in the offline calibration are read, and inverse distortion mapping is performed on the corner coordinates to obtain the distortion-free standard 2D point coordinates. The preset physical side length of the marker is read, and the 3D coordinates of the four corner points in the marker's local coordinate system are generated. The EPnP algorithm is used to solve the perspective n-point problem, outputting the rotation and translation vectors of the marker relative to the camera. The obtained rotation vectors are converted into rotation matrices, which are then combined with the translation vectors to construct a coordinate transformation matrix. Based on the coordinate transformation matrix, the corner coordinates are transformed to obtain the current pose of the marker in the camera coordinate system.

[0054] In one embodiment, the map database is accessed using the identifier of the target marker as a retrieval index. The calibration pose of the target marker in the camera coordinate system, obtained through pre-teaching and calculation during the calibration phase, is read. It is understood that the calibration pose represents the fixed pose of the marker relative to the side-mounted camera when the robot is docked at the theoretical standard position of the workstation.

[0055] Based on current pose , mark positioning Calculate the robot docking deviation matrix The formula for calculating the docking deviation matrix is:

[0056] Understandably, if the robot is completely docked at the standard teaching position, the docking deviation matrix is ​​a 4th-order identity matrix with no compensation offset; if the robot has forward / backward, left / right, or vertical offsets or tilts around three axes, the translation and rotation components in the docking deviation matrix simultaneously record all 6-dimensional offsets for subsequent grasping operation pose correction.

[0057] In the above embodiments, the same set of side-mounted cameras and markers are reused in the navigation phase to solve the workstation docking deviation. There is no need to deploy an additional arm-mounted camera or move the robotic arm to the shooting point for secondary photography. This saves time on secondary imaging and pose calculation, effectively shortens the robot's work cycle, and improves work efficiency while ensuring positioning accuracy.

[0058] S103. Obtain the pose of the object to be operated and the first pose transformation relationship. Based on the docking deviation matrix, the pose of the object to be operated and the first pose transformation relationship, determine the operating pose of the end effector. The first pose transformation relationship is the pose transformation relationship of the end effector from the base coordinate system to the camera coordinate system.

[0059] In one embodiment, the end effector can be a robotic arm gripper, and the object being manipulated can be an object grasped by the robotic arm gripper, such as a semiconductor wafer, a workpiece, a toolbox, etc.

[0060] Operator pose This describes the pose of the manipulated object in the camera coordinate system during robot teaching. The first pose transformation relationship between the base coordinate system and the camera coordinate system is also shown. Obtained through calibration during the calibration phase.

[0061] Understandably, the docking deviation matrix represents the deviation of the robot's current docking position relative to the standard taught position. This deviation is simultaneously superimposed on the grasping pose, causing the end effector to be unable to accurately perform the grasping operation on the object. Therefore, it is necessary to compensate and correct the grasping pose.

[0062] Specifically, according to the docking deviation matrix Position of the operation object Transform to the current camera coordinate system, that is, after offset compensation, the actual pose of the manipulated object relative to the camera is: Then, the actual pose after deviation compensation is transformed into the robot arm's base coordinate system to obtain the pose that the robot arm's end effector can execute. The final pose determination formula is as follows:

[0063] Understandably, if the robot has no docking deviation, that is... If the matrix is ​​an identity matrix, then the actual pose of the manipulated object relative to the camera after bias compensation is equal to the original pose of the manipulated object, i.e. .

[0064] In one embodiment, the solved pose is parsed into kinematically recognizable joint motion commands for the robotic arm and sent to the robotic arm servo drive unit to control the robotic arm gripper to move to the pose to complete the grasping or placement of the object.

[0065] Furthermore, before the robot moves by detecting the poses of markers in the moving environment using a side-mounted camera and correcting the robot's pose based on the marker poses to obtain the precise pose of the robot body, the method further includes: controlling the robot to move to at least one preset calibration pose, reading the robot's body pose, and acquiring an image using a side-mounted camera, the image containing a calibration plate at a fixed position; obtaining a second pose transformation relationship between the camera coordinate system and the body coordinate system based on the image and the robot's body pose; obtaining a third pose transformation relationship between the base coordinate system and the body coordinate system of the end effector, and obtaining a first pose transformation relationship between the base coordinate system and the camera coordinate system based on the second pose transformation relationship and the third pose transformation relationship.

[0066] In one embodiment, the calibration board is placed in a flat area visible to the side-mounted camera, keeping its spatial position and orientation completely fixed. The robot is then controlled to move sequentially to each preset calibration pose according to the planned path. After the robot stabilizes at each pose, the robot's current pose in the world coordinate system is read from the navigation system. The system also uses a side-mounted camera to capture multiple sets of images, including the calibration plate, in different poses.

[0067] Understandably, the calibration plate remains stationary, and its pose in the world coordinate system... It is a constant.

[0068] The corner points of the calibration board in the image are detected, and the pixel coordinates of the corner points are extracted. Based on the physical dimensions of the calibration board, a correspondence is established between the image corner points and the 3D model of the calibration board. The EPnP algorithm is then used to solve the pose of the calibration board in the camera coordinate system. According to the robot's body pose Fixed world pose with calibration plate The pose transformation relationship between the computer body and the camera. For any set of images of the robot in different poses Transform the equation to obtain Substituting each set of collected data into the coordinate transfer identity yields a set of constraint equations. Multiple pose equations are then combined to construct a hyperequation system. This system is solved using least-squares iteration, specifically employing a hand-eye calibration standard solution algorithm. Least-squares optimization is then applied to multiple sets of constraints to solve for the transformation matrix. This refers to the second pose transformation relationship between the camera coordinate system and the body coordinate system.

[0069] In one embodiment, the 3D assembly drawings of the robot are retrieved, and the three-axis translational offset and three-axis rotational Euler angles of the robotic arm base mounting reference relative to the origin of the robot body coordinate system are read. Based on the rigid body homogeneous transformation construction rules, the pose transformation matrix from the robot body to the robotic arm base is directly generated. The pose transformation relationship between the fuselage coordinate system and the base coordinate system is represented by the pose transformation relationship from the fuselage coordinate system to the base coordinate system. The third pose transformation relationship between the base coordinate system and the fuselage coordinate system is then: .

[0070] Alternatively, three non-collinear high-precision feature reference points can be marked on the fuselage and robotic arm base. Using a laser rangefinder and a coordinate measuring machine, the three-dimensional coordinates of the three reference points in the fuselage coordinate system and the robotic arm base coordinate system can be measured respectively. The transformation matrix from the fuselage to the robotic arm base can be fitted and solved by the point set matching least squares algorithm. .

[0071] Based on the second pose transformation relationship between the camera coordinate system and the body coordinate system And the third pose transformation relationship is The transformation yields the first pose transformation relationship from the base coordinate system to the camera coordinate system. .

[0072] Furthermore, before the robot moves by detecting the poses of markers in the moving environment using a side-mounted camera and correcting the robot's pose based on the marker poses to obtain the precise pose of the robot body, the method further includes: controlling the robot to stop at each preset teaching position and recording the robot body pose; detecting the markers corresponding to each preset teaching position using the side-mounted camera and obtaining the marker poses of the markers in the camera coordinate system; obtaining the marker poses of the markers in the world coordinate system based on the robot body pose, the marker poses, and the second pose transformation relationship between the camera coordinate system and the robot body coordinate system; obtaining the pose of the manipulated object in the camera coordinate system as the pose of the manipulated object; and associating and storing the marker identifier, the marker pose in the camera coordinate system, the marker pose in the world coordinate system, and the pose of the manipulated object as a quadruple to obtain a map database.

[0073] In one embodiment, the robot is controlled to move to the teachable position, and its forward, backward, left, right, and rotational postures are fine-tuned to bring it to a stop at the theoretical standard reference position of the workstation. The global pose of the robot body output by the robot navigation module is then read to obtain the robot's body pose in the world coordinate system. .

[0074] The robot remains stationary while a side-mounted camera captures images of complete markers corresponding to the teachable position. The closed contours of the images are extracted, and markers specific to the teachable position are selected based on area and quadrilateral aspect ratio. The unique identifier of each marker is decoded. The marker boundaries are fitted, and the coarse pixel intersections at the four corners are calculated. High-precision ordered 2D corner coordinates are obtained through Shi-Tomasi sub-pixel iterative optimization. The intrinsic parameter matrix and distortion coefficients of the side-mounted camera are retrieved, and inverse distortion mapping is performed on the corner coordinates to eliminate pixel shifts caused by lens distortion. Based on the standard physical dimensions of the markers, the 3D coordinates of the marker corners are generated. The EPnP algorithm is used to solve for the initial rotation and translation vectors of the markers relative to the camera. Then, CERES nonlinear optimization is used to minimize the corner reprojection error. The optimized rotation vectors are converted into rotation matrices, which are combined with the translation vectors to construct a coordinate transformation matrix. The corner coordinates are transformed using the coordinate transformation matrix to obtain the marker's pose in the camera coordinate system. .

[0075] Obtain the second pose transformation relationship between the side-mounted camera and the camera body. According to the fuselage position , mark positioning Relationship between the pose transformation of the robotic arm base and the body Determine the positioning pose of the marker in the world coordinate system. , .

[0076] In one embodiment, a human operator uses a robotic arm teach pendant to drag the robotic arm gripper to the standard gripping point at the workstation, aligns it with the preset gripping reference for the object being manipulated, locks the robotic arm posture, and reads the standard gripping posture of the robotic arm end effector relative to the robotic arm base. Grasp the pose Transform from the robot arm base coordinate system to the side-mounted camera coordinate system to obtain the pose of the manipulated object in the camera coordinate system. , .in, This represents the pose transformation relationship from the fuselage coordinate system to the base coordinate system.

[0077] The system integrates the marker identifier, the marker's calibration pose in the camera coordinate system, the marker's calibration pose in the world coordinate system, and the pose of the manipulated object for each preset pick-up and drop-off position. These are then stored as a quadruple and added to a preset map database. This quadruple-based binding and storage constructs an integrated map database, enabling synchronized data management.

[0078] In the above embodiments, a side-mounted camera is used to complete all visual acquisition work, eliminating the need for an arm-mounted camera for workstation calibration and teaching. This simplifies on-site teaching procedures and reduces the difficulty of manual teaching. A single set of markers simultaneously performs both long-distance navigation correction and workstation grasping deviation compensation functions, eliminating the need to deploy two separate sets of markers for navigation and grasping, significantly reducing the workload of robot setup and subsequent maintenance costs.

[0079] Please see Figure 4 , Figure 4 This is a schematic block diagram of an embodiment of the present application providing an operating pose determination device for a robot end effector. This device is used to execute the aforementioned operating pose determination method for a robot end effector. The operating pose determination device can be configured on a server.

[0080] like Figure 4 As shown, the robot end effector's pose determination device 300 includes: The robot pose correction module 301 is used to detect the pose of markers in the moving environment based on a side-mounted camera during robot movement, and to correct the robot pose based on the pose of the markers to obtain the precise pose of the robot body. The docking deviation acquisition module 302 is used to acquire the target marker pose corresponding to the target pick-up and place station based on the side-mounted camera when the precise pose of the robot body is within the range of the target pick-up and place station, and to determine the docking deviation matrix of the robot based on the target marker pose. The operation pose acquisition module 303 is used to acquire the pose of the operation object and the first pose transformation relationship. Based on the docking deviation matrix, the pose of the operation object and the first pose transformation relationship, the operation pose of the end effector is determined. The first pose transformation relationship is the pose transformation relationship of the end effector from the base coordinate system to the camera coordinate system.

[0081] Furthermore, the marker pose includes the current pose of the marker in the camera coordinate system and the calibration pose of the marker in the world coordinate system, and the calibration pose is stored in a preset map database; The fuselage pose correction module 301 includes: The marker recognition unit is used to acquire marker images in the moving environment based on the side-mounted camera during robot movement, and to identify the corner coordinates of the markers in the marker images based on the marker images, wherein the marker images contain markers corresponding to preset pick-up and place positions; The current pose acquisition unit is used to perform pose transformation based on the corner coordinates, the intrinsic parameter matrix of the side-mounted camera, and the distortion coefficients to obtain the current pose of the marker in the camera coordinate system. A marker pose extraction unit is used to extract the marker pose in the world coordinate system from the map database. The body pose acquisition unit is used to perform pose reasoning on the current pose and the calibration pose to obtain the body pose of the robot in the world coordinate system. The pose estimation correction unit is used to correct the current estimated pose of the robot based on the body pose to obtain the accurate pose of the body.

[0082] Furthermore, the estimated pose correction unit includes: The pose estimation acquisition subunit is used to acquire the current estimated pose and estimated covariance matrix output by the robot's navigation system; The covariance matrix determination subunit is used to obtain the relative distance between the side-mounted camera and the marker based on the current pose of the marker in the camera coordinate system, and to determine the noise covariance matrix of the visual observation based on the relative distance. A Kalman gain determination subunit is used to determine the Kalman gain based on the estimated covariance matrix and the noise covariance matrix; The precise pose determination subunit is used to perform Kalman filtering based on the Kalman gain, the fuselage pose, and the current estimated pose to obtain the precise fuselage pose.

[0083] Furthermore, the estimated pose correction unit further includes: The pose estimation acquisition subunit is used to acquire the current estimated pose of the robot; The weight determination subunit is used to determine the visual weight of the body pose and the prediction weight of the current estimated pose based on the relative distance between the side-mounted camera and the marker. The precise pose determination subunit is used to perform weighted fusion of the fuselage pose and the current estimated pose based on the visual weights and the prediction weights to obtain the precise pose of the fuselage.

[0084] Furthermore, the target marker pose includes the current pose of the target marker in the camera coordinate system and the calibration pose of the target marker in the camera coordinate system, wherein the calibration pose is stored in the map database; The docking deviation acquisition module 302 includes: The target marker recognition unit is used to acquire target marker images based on the side-mounted camera and identify the corner coordinates of the target markers in the target marker images, wherein the target marker images contain target markers corresponding to the target pick-and-place station; The current pose acquisition unit is used to obtain the current pose based on the corner coordinates, the intrinsic parameter matrix of the side-mounted camera, and the distortion coefficients. The calibration pose extraction unit is used to extract the calibration pose of the target marker in the camera coordinate system from the map database; The deviation matrix determination unit is used to determine the docking deviation matrix based on the current pose and the calibration pose.

[0085] Furthermore, the robot end effector's operation pose determination device 300 further includes a pose transformation relationship acquisition module, which includes: The image acquisition unit is used to control the robot to move to at least one preset calibration pose, read the robot body pose, and acquire images through a side-mounted camera. The images contain a calibration plate at a fixed position. The second pose transformation relationship acquisition unit is used to obtain the second pose transformation relationship between the camera coordinate system and the body coordinate system based on the image and the robot body pose. The first pose transformation relationship acquisition unit is used to obtain the third pose transformation relationship between the base coordinate system and the body coordinate system of the end effector, and to obtain the first pose transformation relationship between the base coordinate system and the camera coordinate system based on the second pose transformation relationship and the third pose transformation relationship.

[0086] Furthermore, the robot end effector's pose determination device 300 also includes a map database construction module, which includes: The robot body posture recording unit is used to control the robot to stop at each preset teaching position and record the robot body posture; The calibration pose acquisition unit is used to detect the markers corresponding to each of the preset teaching positions through the side-mounted camera, and obtain the calibration pose of the markers in the camera coordinate system; based on the body pose, the calibration pose, and the second pose transformation relationship between the camera coordinate system and the body coordinate system, it obtains the calibration pose of the markers in the world coordinate system. The object pose acquisition unit is used to acquire the pose of the object in the camera coordinate system as the object pose. The map database acquisition unit is used to associate and store the identifier of the marker, the calibration pose of the marker in the camera coordinate system, the calibration pose of the marker in the world coordinate system, and the pose of the operation object as a quadruple to obtain the map database.

[0087] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the above-described apparatus and modules can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0088] The aforementioned device can be implemented as a computer program, which can be used in, for example... Figure 5 It runs on the computer device shown.

[0089] Please see Figure 5 , Figure 5 This is a schematic block diagram illustrating the structure of a computer device according to an embodiment of this application. The computer device may be a server.

[0090] See Figure 5 The computer device includes a processor, memory, and network interface connected via a system bus, wherein the memory may include non-volatile storage media and internal memory.

[0091] Non-volatile storage media can store operating systems and computer programs. These computer programs include program instructions that, when executed, cause the processor to perform any method for determining the pose of a robot end effector.

[0092] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0093] Internal memory provides an environment for the execution of computer programs in non-volatile storage media. When the computer program is executed by the processor, it enables the processor to perform any method for determining the pose of a robot end effector.

[0094] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0095] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0096] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps: During robot movement, the poses of markers in the moving environment are detected by a side-mounted camera, and the robot's pose is corrected based on the poses of the markers to obtain the precise pose of the robot body. When the precise pose of the robot body is within the range of the target pick-and-place station, the pose of the target marker corresponding to the target pick-and-place station is acquired based on the side-mounted camera, and the docking deviation matrix of the robot is determined based on the pose of the target marker. The pose of the object to be operated on and the first pose transformation relationship are obtained. Based on the docking deviation matrix, the pose of the object to be operated on and the first pose transformation relationship, the operating pose of the end effector is determined. The first pose transformation relationship is the pose transformation relationship of the end effector from the base coordinate system to the camera coordinate system.

[0097] The embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, and the processor executing the program instructions to implement any of the robot end effector operation pose determination methods provided in the embodiments of this application.

[0098] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the computer device.

[0099] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0100] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for determining the operating pose of a robot end effector, characterized in that, include: During robot movement, the poses of markers in the moving environment are detected by a side-mounted camera, and the robot's pose is corrected based on the poses of the markers to obtain the precise pose of the robot body. When the precise pose of the robot body is within the range of the target pick-and-place station, the pose of the target marker corresponding to the target pick-and-place station is acquired based on the side-mounted camera, and the docking deviation matrix of the robot is determined based on the pose of the target marker. The pose of the object to be operated on and the first pose transformation relationship are obtained. Based on the docking deviation matrix, the pose of the object to be operated on and the first pose transformation relationship, the operating pose of the end effector is determined. The first pose transformation relationship is the pose transformation relationship of the end effector from the base coordinate system to the camera coordinate system.

2. The method for determining the operating pose of a robot end effector according to claim 1, characterized in that, The marker pose includes the current pose of the marker in the camera coordinate system and the calibration pose of the marker in the world coordinate system, and the calibration pose is stored in a preset map database. During robot movement, the method involves detecting the poses of markers in the moving environment using a side-mounted camera, and then correcting the robot's pose based on these marker poses to obtain a precise robot pose. This includes: During the robot's movement, images of markers in the moving environment are acquired based on the side-mounted camera, and the corner coordinates of the markers in the marker images are identified based on the marker images, wherein the marker images contain markers corresponding to preset pick-up and place positions; Based on the corner coordinates, the intrinsic parameter matrix of the side-mounted camera, and the distortion coefficient, a pose transformation is performed to obtain the current pose of the marker in the camera coordinate system. Extract the coordinate pose of the marker in the world coordinate system from the map database; Perform pose inference on the current pose and the calibrated pose to obtain the robot's body pose in the world coordinate system; The robot's current estimated pose is corrected based on the body pose to obtain the precise body pose.

3. The method for determining the operating pose of a robot end effector according to claim 2, characterized in that, The step of correcting the robot's current estimated pose based on the fuselage pose to obtain the precise fuselage pose includes: Obtain the current estimated pose and estimated covariance matrix output by the robot's navigation system; Based on the current pose of the marker in the camera coordinate system, the relative distance between the side-mounted camera and the marker is obtained, and based on the relative distance, the noise covariance matrix of the visual observation is determined. The Kalman gain is determined based on the estimated covariance matrix and the noise covariance matrix. The precise fuselage pose is obtained by performing Kalman filtering based on the Kalman gain, the fuselage pose, and the current estimated pose.

4. The method for determining the operating pose of a robot end effector according to claim 2, characterized in that, The step of correcting the robot's current estimated pose based on the fuselage pose to obtain the precise fuselage pose further includes: Obtain the robot's current estimated pose; Based on the relative distance between the side-mounted camera and the marker, the visual weight of the body pose and the prediction weight of the current estimated pose are determined. Based on the visual weights and the prediction weights, the fuselage pose and the current estimated pose are weighted and fused to obtain the precise fuselage pose.

5. The method for determining the operating pose of a robot end effector according to claim 1, characterized in that, The target marker pose includes the current pose of the target marker in the camera coordinate system and the target marker pose in the camera coordinate system, wherein the target marker pose is stored in the map database; The step of acquiring the pose of the target marker corresponding to the target pick-and-place station based on the side-mounted camera, and determining the robot's docking deviation matrix based on the target marker pose, includes: The target marker image is acquired based on the side-mounted camera, and the corner coordinates of the target marker in the target marker image are identified, wherein the target marker image contains the target marker corresponding to the target pick-and-place station; The current pose is obtained based on the corner coordinates, the intrinsic parameter matrix of the side-mounted camera, and the distortion coefficients. Extract the target marker's pose in the camera coordinate system from the map database; Based on the current pose and the calibration pose, the docking deviation matrix is ​​determined.

6. The method for determining the operating pose of a robot end effector according to claim 1, characterized in that, Before the step of detecting the poses of markers in the moving environment based on a side-mounted camera during robot movement, and correcting the robot's pose based on the marker poses to obtain the precise pose of the robot body, the method further includes: The robot is controlled to move to at least one preset calibration pose, the robot body pose is read, and images are acquired through a side-mounted camera. The images contain a calibration plate at a fixed position. Based on the image and the robot's body pose, a second pose transformation relationship between the camera coordinate system and the body coordinate system is obtained; Obtain the third pose transformation relationship between the base coordinate system and the body coordinate system of the end effector, and obtain the first pose transformation relationship between the base coordinate system and the camera coordinate system based on the second pose transformation relationship and the third pose transformation relationship.

7. The method for determining the operating pose of a robot end effector according to any one of claims 1 to 6, characterized in that, Before the step of detecting the poses of markers in the moving environment based on a side-mounted camera during robot movement, and correcting the robot's pose based on the marker poses to obtain the precise pose of the robot body, the method further includes: Control the robot to stop at each preset teaching position and record the robot's body posture; The markers corresponding to each of the preset teaching positions are detected by the side-mounted camera to obtain the calibration pose of the markers in the camera coordinate system. Based on the fuselage pose, the calibration pose, and the second pose transformation relationship between the camera coordinate system and the fuselage coordinate system, the calibration pose of the marker in the world coordinate system is obtained. The pose of the operation object in the camera coordinate system is obtained and used as the pose of the operation object; The map database is obtained by associating and storing the identifier of the marker, the calibration pose of the marker in the camera coordinate system, the calibration pose of the marker in the world coordinate system, and the pose of the manipulated object as a quadruple.

8. A device for determining the operating pose of a robot end effector, characterized in that, include: The robot pose correction module is used to detect the pose of markers in the moving environment based on a side-mounted camera during robot movement, and to correct the robot pose based on the pose of the markers to obtain the precise pose of the robot body. The docking deviation acquisition module is used to acquire the target marker pose corresponding to the target pick-and-place station based on the side-mounted camera when the precise pose of the robot body is within the range of the target pick-and-place station, and to determine the docking deviation matrix of the robot based on the target marker pose. The operation pose acquisition module is used to acquire the pose of the operation object and the first pose transformation relationship. Based on the docking deviation matrix, the pose of the operation object and the first pose transformation relationship, the operation pose of the end effector is determined. The first pose transformation relationship is the pose transformation relationship of the end effector from the base coordinate system to the camera coordinate system.

9. A computer device, characterized in that, The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and, when executing the computer program, implement the method for determining the operating pose of the robot end effector as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to implement the method for determining the operating pose of a robot end effector as described in any one of claims 1 to 7.

Images