A robot control method and related products

Abstract

This application discloses a robot control method and related products. The method includes: determining the initial pose data of the object to be operated by the robot end effector in the camera coordinate system; acquiring the transformation matrix from the robot base coordinate system to the camera coordinate system, and the tool matrix from the robot end effector to the tool coordinate system; adjusting the initial pose data according to the execution task corresponding to the object to be operated by the robot end effector based on the transformation matrix and the tool matrix, to obtain the target pose data of the robot end effector in the base coordinate system; generating control commands based on the target pose data to perform motion control on the robot and complete the execution task. This application does not simply perform coordinate transformation, but adjusts the pose data according to the execution task corresponding to the object to be operated by the robot end effector, enabling dynamic adjustment of the target pose data generation strategy according to different operational requirements.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a robot control method and related products. Background Technology

[0002] In industrial assembly applications, simply acquiring the pose information of the target object in the camera coordinate system is often insufficient to directly convert it into motion commands that the robot can execute. Currently, robot control faces two major technical bottlenecks: First, the tool center point (TCP) offset of the robot's end effector lacks accurate modeling, leading to coordinate mapping errors between visual perception and physical operation; second, in dual-arm collaborative assembly tasks, the data flow between visual pose data and the robot motion planning module is not smooth, severely affecting the accuracy and efficiency of multi-machine collaborative operations. Summary of the Invention

[0003] To address the aforementioned issues, this application provides a robot control method and related products.

[0004] The embodiments of this application disclose the following technical solutions: The first aspect of this application provides a robot control method, the method comprising: Determine the initial pose data of the object to be manipulated by the robot end effector in the camera coordinate system; Obtain the transformation matrix from the robot's base coordinate system to the camera coordinate system, and the tool matrix from the robot's end effector to the tool coordinate system; Based on the transformation matrix and the tool matrix, the initial pose data is adjusted according to the task to be performed on the object to be operated by the robot end effector to obtain the target pose data of the robot end effector in the base coordinate system. Based on the target pose data, control commands are generated to control the robot's motion and complete the task.

[0005] A second aspect of this application provides a robot control device, the robot control device comprising: The initial pose determination module is used to determine the initial pose data of the object to be operated by the robot end effector in the camera coordinate system. The acquisition module is used to acquire the transformation matrix from the robot's base coordinate system to the camera coordinate system, and the tool matrix from the robot's end effector to the tool coordinate system; The target pose determination module is used to adjust the initial pose data according to the transformation matrix and the tool matrix, and according to the execution task corresponding to the object to be operated by the robot end, to obtain the target pose data of the robot end in the base coordinate system. The execution module is used to generate control commands based on the target pose data, perform motion control on the robot, and complete the execution task.

[0006] A third aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the robot control method provided in the first aspect.

[0007] The fourth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the robot control method provided in the first aspect.

[0008] Compared with the prior art, this application has the following beneficial effects: This application includes determining the initial pose data of the object to be manipulated by the robot end effector in the camera coordinate system; obtaining the transformation matrix from the robot base coordinate system to the camera coordinate system, and the tool matrix from the robot end effector to the tool coordinate system; adjusting the initial pose data according to the execution task corresponding to the object to be manipulated by the robot end effector based on the transformation matrix and the tool matrix to obtain the target pose data of the robot end effector in the base coordinate system; generating control commands based on the target pose data to perform motion control on the robot and complete the execution task.

[0009] To overcome the problem of "lack of accurate modeling of the tool center point (TCP) offset of the robot end effector," this application explicitly introduces a tool matrix for coordinate compensation when calculating the target pose data. Instead of simply driving the robot's end flange center to align with the visual coordinates, it accurately calculates the offset of the tool center point relative to the flange. By jointly adjusting the initial pose data based on the transformation matrix and the tool matrix, this application can accurately map the visual observations in the camera coordinate system to the actual working pose in the base coordinate system, effectively eliminating the positioning error caused by the lack of TCP offset modeling and significantly improving the grasping and placement accuracy in industrial assembly scenarios.

[0010] To overcome the problem of "poor data flow between visual pose data and robot motion planning module in dual-arm collaborative assembly tasks", this application can quickly convert unstructured visual pose information into structured robot motion control instructions, ensuring that different robots can respond based on a unified coordinate reference and task logic, solving the data flow discontinuity problem, and thus greatly improving the response speed and overall assembly efficiency of multi-machine collaborative operations.

[0011] This application does not simply perform coordinate transformation, but adjusts the pose data according to the task to be performed on the object to be operated at the robot end. It can dynamically adjust the generation strategy of target pose data according to different job requirements, so that the robot can not only "see" the object, but also "move accurately" and "do the right thing" according to specific process requirements. This effectively solves the problem that traditional methods that rely solely on fixed coordinate transformation are difficult to cope with complex assembly processes. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A flowchart illustrating a robot control method provided in this application embodiment; Figure 2 This is a structural diagram of a robot control device provided in an embodiment of this application. Detailed Implementation

[0014] As described earlier, in industrial assembly, the camera pose of the target object is difficult to directly translate into robot motion commands. The bottlenecks of existing technologies are: on the one hand, inaccurate end-effector center point (TCP) offset modeling leads to coordinate mapping errors; on the other hand, the data flow between vision data and motion planning is not smooth, which restricts the accuracy and efficiency of multi-machine collaborative assembly.

[0015] In view of the above problems, this application provides a robot control generation method and related products. The method includes: determining the initial pose data of the object to be operated by the robot end effector in the camera coordinate system; obtaining the transformation matrix from the robot base coordinate system to the camera coordinate system, and the tool matrix from the robot end effector to the tool coordinate system; adjusting the initial pose data according to the execution task corresponding to the object to be operated by the robot end effector based on the transformation matrix and the tool matrix, to obtain the target pose data of the robot end effector in the base coordinate system; generating control commands based on the target pose data to perform motion control on the robot and complete the execution task.

[0016] To address the issue of inaccurate modeling of the tool center point (TCP) offset in robot end-effectors, this application explicitly introduces a tool matrix for coordinate compensation, accurately mapping visual observations to the actual working pose in the base coordinate system. This effectively eliminates positioning errors and improves grasping and placement accuracy. Simultaneously, addressing the pain point of poor data flow between vision and motion planning in multi-robot collaboration, this application can quickly convert unstructured visual information into structured motion commands, ensuring multi-robot responses based on a unified reference, significantly improving collaborative assembly efficiency. Furthermore, this application can dynamically adjust the pose generation strategy according to specific tasks, enabling the robot to accurately handle complex assembly process requirements.

[0017] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0018] Figure 1 A flowchart of a robot control method provided in an embodiment of this application is shown below. Figure 1 As shown, a robot control method includes: S101: Determine the initial pose data of the object to be operated by the robot end in the camera coordinate system.

[0019] This step aims to obtain the precise pose of the target object in the camera coordinate system through standardized visual perception services, providing a data foundation for subsequent robot motion planning. Specifically, the initial pose data determination method in this embodiment includes, but is not limited to, the following steps: To ensure the accuracy and compatibility of data interaction between the robot and the vision system, this application defines a unified service interface protocol (e.g., named PoseEstimate.srv). This protocol clearly defines the fields for vision service requests and responses: Request fields: These must include at least the target part identifier (part_name), the symmetry correction flag (enable_symmetry_correction), and the image resolution parameters (width / height). Specifically, part_name is used to index a specific part category and its corresponding CAD reference point; enable_symmetry_correction indicates whether symmetry axis rotation blur correction is enabled for a specific part; and width / height is used to constrain the resolution of image acquisition or preprocessing.

[0020] Response fields: These should include at least the initial pose data of the target object in the camera coordinate system (t_cam_obj, usually represented as a 4×4 homogeneous transformation matrix), pose quality assessment information (pose_score), target detection box information (bbox), and service execution status (success / message).

[0021] After receiving the requested field, retrieve the part data corresponding to the requested field, for example: The system retrieves the corresponding part data from a pre-stored parts database based on the part_name field in the request. The part data includes at least the part's 3D reference points (CAD model feature points used for PnP calculation) and, in specific cases, the part's symmetry axis constraint data (e.g., the symmetry axis direction of a cylindrical or rotating part). Simultaneously, the width / height parameters are passed to the robot's acquisition device as the standard resolution for image acquisition or scaling.

[0022] Target detection is performed on the image data collected by the robot to obtain multiple candidate boxes for target parts and the confidence level of each candidate box.

[0023] To acquire high-quality image data and ensure the timeliness and consistency of camera intrinsic parameters, this application proposes a three-level steady-state acquisition strategy for images and intrinsic parameters. The specific implementation process is as follows: A frame of RGB image and depth image is acquired through the robot's acquisition device (such as a camera) driver interface (e.g., encapsulated as the Capture.srv service), and the current image timestamp is recorded.

[0024] Three-level steady-state strategy for obtaining internal references: First priority: Attempt to read camera intrinsic data (CameraInfo) and intrinsic timestamps from a locally stored configuration file (such as camera.json); Secondary rollback: If file reading fails, the camera software development kit (SDK) is invoked to read the current camera's intrinsic parameter data in real time; Three-level fallback: If the SDK call still fails, the system's default fallback approximation parameters will be enabled to ensure uninterrupted service.

[0025] The difference between the acquired intrinsic timestamp and the image timestamp (i.e., time delay) is calculated. If the difference exceeds a preset threshold (e.g., 50ms), it indicates that the intrinsic parameters may no longer match the current frame image (e.g., due to zooming or movement), and the system will discard the data of this frame and trigger re-acquisition; if the difference does not exceed the threshold, it is determined that the image and the intrinsic parameters are consistent in the same frame, and the RGB image, Depth image, timestamp, and intrinsic parameter data are saved and output.

[0026] The collected and verified image data is input into a pre-trained target detection neural network (such as the YOLO series, Faster R-CNN, etc.). The model performs inference on the image and outputs multiple candidate bounding boxes (bboxes) for target parts, as well as the confidence score for each candidate box.

[0027] The multiple candidate boxes for target parts are filtered based on confidence level to determine the selected box for the target part, for example: Multiple candidate boxes are filtered based on a preset confidence threshold, and low-confidence interference items are eliminated to determine the final target part selection box.

[0028] Based on the target part selection box, the image data, and the part data, the initial pose data of the object to be manipulated by the robot end effector in the camera coordinate system is obtained, for example: Feature points are obtained by extracting features from the selected bounding box of the target part.

[0029] For example, extract 2D key points or feature points (such as corner points, edge points, or feature descriptors extracted by deep learning) within the selected area of ​​the target part.

[0030] Based on the feature points, the three-dimensional reference points of the part, and the intrinsic parameter data of the image data, the pose data to be corrected is obtained.

[0031] For example, by using the intrinsic parameters of image data, the projection relationship between 2D feature points and the three-dimensional reference points (CAD model points) of the part can be established. The preliminary pose of the object relative to the camera, i.e. the pose data to be corrected, can be solved by the Perspective-n-Point (PnP) algorithm.

[0032] The rotational branches of the pose data to be corrected are corrected using the symmetry axis constraint data of the part, so as to obtain the initial pose data of the object to be operated by the robot end in the camera coordinate system.

[0033] For example, check the `enable_symmetry_correction` flag in the request field. If the flag is true, then correct the rotation branch of the pose data to be corrected based on the symmetry axis constraint data of the part. For example, for parts with rotational symmetry, eliminate the ambiguity of rotation angles caused by symmetry and output a unique and stable rotation matrix.

[0034] S101 outputs the initial pose data (t_cam_obj) of the object to be manipulated at the robot's end effector in the camera coordinate system. In specific implementations, this data can be output together with the quality assessment information (pose_score) and the detection box (bbox) for direct use in subsequent robot coordinate transformation and motion control steps.

[0035] S102: Obtain the transformation matrix from the robot base coordinate system to the camera coordinate system, and the tool matrix from the robot end effector to the tool coordinate system; This step aims to establish a complete coordinate transformation link from the visual perception space to the robot's physical operating space. After receiving the initial pose data of the target object in the camera coordinate system (denoted as T_cam_obj) output in step S101, the robot control system needs to further load the pre-stored calibration configuration file to parse out the key extrinsic parameter matrices required for subsequent coordinate transformations. The specific implementation process includes the following steps: Access the pre-stored calibration configuration file (e.g., named tcp_calibration.json). This file is serialized and stored in a standardized JSON format, designed to enable persistent management, cross-platform migration, and version tracking of calibration data. The file contains at least the following two core data fields: The hand-eye calibration result domain stores the fixed transformation relationship between the camera coordinate system and the robot's base coordinate system. In the "eye-to-hand" mounting architecture, this transformation matrix remains constant after calibration.

[0036] The tool calibration results field (tools) stores the tool center point (TCP) extrinsic parameters of various end effectors (such as grippers, suction cups, welding torches, etc.) in key-value pairs. Each tool entry contains a tool matrix unique to that tool, that is, the transformation relationship between the tool coordinate system and the robot end flange coordinate system.

[0037] Read the transformation matrix (denoted as T_base_cam) from the robot base coordinate system to the camera coordinate system from the hand_eye field of the calibration configuration file.

[0038] If the vision system uses a depth camera to directly output a 6-DOF pose, T_base_cam is represented as a 4×4 homogeneous transformation matrix, which includes a 3×3 rotation matrix and a 3×1 translation vector.

[0039] In practice, the read data can be validated (e.g., check if the matrix is ​​orthogonal and if the determinant is 1) to prevent abnormal coordinate mapping caused by file corruption or data tampering.

[0040] Based on the target part identifier or tool identifier specified in the current task (e.g., tool_name obtained from the request field of S101), an index match is performed in the tools field of the calibration configuration file to obtain the tool matrix (denoted as T_end_tool) corresponding to the tool actually mounted on the end effector of the robot.

[0041] The system supports rapid switching of different end effectors via software configuration without requiring re-physical teaching. For example, when a robot moves from a "grasping station" to an "assembly station" and needs to change its end effector, the TCP bias parameters can be automatically updated simply by reading the T_end_tool under the corresponding tool name. If multiple historical calibration versions of the same tool exist in the calibration configuration file (e.g., distinguished by version_id), the tool matrix marked as currently effective or with the latest version number will be loaded by default; if no matching tool identifier is found, an exception handling mechanism will be triggered (such as an error message or loading the default factory calibration value).

[0042] The homogeneous transformation matrices (T_base_cam and T_end_tool) output in this step can be used as the core parameters of the coordinate transformation chain in the subsequent step S103 to eliminate the spatial deviation between the visual observation coordinate system and the robot's physical execution coordinate system.

[0043] S103: Based on the transformation matrix and the tool matrix, adjust the initial pose data according to the task to be performed on the object to be operated by the robot end effector to obtain the target pose data of the robot end effector in the base coordinate system.

[0044] This step is the core link connecting visual perception and robot motion control. Based on the task (such as grasping or placing) corresponding to the object to be manipulated at the robot's end effector, the pose data in the visual coordinate system is mapped to the executable target pose in the robot's base coordinate system using the calibration matrix obtained in step S102. The specific implementation process includes the following two task modes: If the task is to grasp an object, the target pose that the robot's end effector (flange) should reach needs to be calculated so that the tool center point (TCP) is accurately aligned with the object. The specific steps are as follows: Using the homogeneous transformation formula, based on the transformation matrix and the tool matrix, the initial pose data is adjusted to obtain the target transformation matrix of the robot end effector in the base coordinate system. Example: Using the homogeneous transformation formula, combined with the transformation matrix T_base_cam from the robot base coordinate system to the camera coordinate system, the initial pose data T_cam_obj of the object in the camera coordinate system, and the tool matrix T_end_tool from the robot end effector to the tool coordinate system, the target transformation matrix T_base_end_target that the end effector should reach in the robot base coordinate system is calculated.

[0045] The homogeneous transformation formula is as follows: T_base_end_target = T_base_cam × T_cam_obj × (T_end_tool) -1 ; Where T_base_end_target represents the target transformation matrix of the robot's end effector in the base coordinate system; T_base_cam represents the transformation matrix from the robot's base coordinate system to the camera coordinate system; T_cam_obj represents the initial pose data; T_end_tool represents the tool matrix from the robot's end effector to the tool coordinate system; (T_end_tool) -1 This represents the inverse of the tool matrix.

[0046] The rotation matrix in the target transformation matrix is ​​converted into Euler angles, and the Euler angles are combined with the translation amount in the target transformation matrix to obtain the target pose data of the robot end effector in the base coordinate system. Example: The 3×3 rotation matrix in the calculated target transformation matrix T_base_end_target is converted into Euler angles (e.g., rx, ry, rz using ZYX conformation). During this process, singular pose branch processing logic is executed (e.g., gimbal lock detection and automatic selection of the optimal rotation path) to prevent failure of the robot's inverse kinematics solution. The converted Euler angles are combined with the translation vectors (x, y, z) in the target transformation matrix to obtain the target pose data (x, y, z, rx, ry, rz) of the robot's end effector in the base coordinate system.

[0047] If the task is to place an object to be manipulated (or a relative alignment task), the relative transformation of the object from its current position to the target mounting position needs to be calculated and superimposed on the robot's current end-effector pose. This mode is often used in precision assembly scenarios requiring "hand-eye coordination," and the specific steps are as follows: Based on the transformation matrix, the initial pose data is adjusted to obtain the absolute pose data of the object in the base coordinate system; based on the transformation matrix and the position data of the acquisition device in the robot, the absolute pose data of the target placement position of the object to be operated in the base coordinate system is obtained. Example: Call the visual perception service in step S101 to obtain the initial pose data T_cam_obj and the position data T_cam_tar of the acquisition device in the robot (such as the pose of the target installation position in the camera coordinate system (obtained by shooting the target workstation)). It is necessary to ensure that the visual data and the current state of the robotic arm are consistent in time.

[0048] Based on the transformation matrix T_base_cam, the two visual data streams mentioned above are transformed into the robot's base coordinate system: Calculate the absolute pose data T_base_obj of the object in the base coordinate system: T_base_obj = T_base_cam × T_cam_obj; Calculate the absolute pose data T_base_tar of the target mounting position in the base coordinate system: T_base_tar = T_base_cam × T_cam_tar; Based on the absolute pose data of the object in the base coordinate system and the absolute pose data of the target placement position corresponding to the object to be operated in the base coordinate system, the relative transformation matrix from the current position of the object to be operated to the target placement position is obtained: T_obj_to_tar = T_base_tar × (T_base_obj) -1 ; Based on the relative transformation matrix from the current position of the object to be manipulated to the target placement position and the end-effector pose data, the target pose data of the robot end-effector in the base coordinate system is obtained. Example: Obtain the robot end effector's pose data T_base_end_current in the base coordinate system (e.g., the pose of the robot arm's end effector in the robot's base coordinate system at the current moment), and superimpose the aforementioned relative transformation matrix onto the current end effector pose to obtain the robot end effector's new target pose data T_base_end_new in the base coordinate system: T_base_end_new = T_obj_to_tar × T_base_end_current.

[0049] The aforementioned relative pose mapping algorithm effectively solves the problem of accumulated absolute coordinate positioning errors and improves the operational accuracy in complex assembly scenarios.

[0050] S104: Generate control commands based on the target pose data to control the motion of the robot and complete the task.

[0051] This step aims to convert the target pose data calculated in step S103 into joint control signals executable by the robot's underlying actuator, and drive the robot's end effector to accurately complete physical tasks such as grasping or placing. The specific implementation process includes the following steps: The target pose data (including position coordinates x, y, z and attitude Euler angles rx, ry, rz) is used as the target state of the end effector and input into the robot's motion control kernel. A motion planning library (such as MoveIt! based on the ROS architecture or the native planner of various robot brands) is invoked to generate a collision-free trajectory in joint space that smoothly transitions from the current joint state to the target state. During this process, inverse kinematics (IK) is performed on the target pose, mapping the target pose in Cartesian space to a sequence of target joint angles for each joint of the robot. If multiple inverse kinematics solutions exist for the target pose, the system can automatically select the optimal joint solution based on preset optimization strategies (such as the shortest path principle, minimum energy consumption principle, or singularity avoidance principle).

[0052] Based on the planned joint trajectories, low-level control commands containing position, velocity, and acceleration information are generated. Specifically, these control commands are sent to the servo drivers of each joint in a high-frequency (e.g., 100Hz to 1kHz) interpolated form. To improve motion smoothness and response speed, a Time-Optimal Path Planning (TOPP) algorithm or a fifth-order polynomial interpolation algorithm is introduced during the trajectory generation stage to ensure continuous jerk (acceleration) during acceleration, constant speed, and deceleration phases, thereby reducing mechanical vibration and protecting precision transmission components.

[0053] The robot drives the end effector to move according to the received control commands. During execution, encoder feedback data of each joint is collected in real time to form a closed-loop control of position and velocity to eliminate tracking errors caused by external disturbances. When the robot end effector reaches the preset target pose tolerance range, the motion is determined to be complete, and the corresponding end effector action is triggered (such as controlling the pneumatic gripper to close for grasping, or opening the gripper to release the object).

[0054] After the physical action is completed, the robot's internal state machine is updated, marking the current task (such as grasping or placing) as "completed," and the final execution result (including log information such as actual positioning accuracy and execution time) is fed back to the upper-level business scheduling system. If a collision alarm or abnormality is detected during execution, the system will immediately trigger an emergency stop protection mechanism and output an error code for subsequent troubleshooting.

[0055] To build a high-concurrency, high-reliability multi-robot collaborative system and ensure that S101 calls have strict isolation and parallel processing capabilities in scenarios where multiple robots are working simultaneously, this implementation method aims to solve common data crosstalk and resource contention problems in multi-robot deployments. Specific implementation details are as follows: Instead of the traditional global absolute path invocation method, it uses the relative service name `infer_pose` for client creation. When the robot node starts, it automatically reads the namespace to which the current node belongs (e.g., ` / RobotA` or ` / RobotB`) and dynamically resolves the relative service name to a unique absolute service path (e.g., ` / RobotA / infer_pose`).

[0056] This mechanism ensures that each robot instance sends requests only to the vision service nodes belonging to its own namespace. Even if there are multiple vision service instances with the same name in the underlying network (such as multiple GPU servers running a service named infer_pose), the namespace prefix isolation ensures that requests are accurately routed to the corresponding service port, eliminating routing errors at the network communication level.

[0057] At the software architecture level, this application instantiates an independent service client object within the process space of each robot node. Each client object internally maintains a private request queue and callback group.

[0058] Client instances of different robot nodes do not share memory state and do not interfere with each other. Even if a robot's vision request is blocked due to network fluctuations or long image processing times, it will only consume the thread resources of that instance and will not block the sending of requests and receiving of responses from other robot nodes. When multiple robots trigger step S101 simultaneously, their respective clients encapsulate the request and push it into an independent queue, then release control and continue executing subsequent logic. Data is processed through callback functions after the vision service returns the result. This design greatly improves the system's concurrent throughput.

[0059] This application performs standardized preprocessing on the target pose data output in step S103 and uses an asynchronous communication mechanism to issue control commands. The specific implementation process includes the following two aspects: The displacement in the target pose data is converted to meters, the attitude angle is kept in radians, and an asynchronous request plus timeout waiting mechanism is used for service calls.

[0060] For example, the x, y, and z values ​​in the target pose data are converted to m = mm / 1000, while the attitude angles are kept in radians; asynchronous calls are configured with timeouts and retry mechanisms, and if a timeout occurs, an error code is returned and upper-level replanning is triggered.

[0061] Regarding the updating of the tool matrix, this application also provides an optional embodiment: Control the robot to move to a preset calibration position, and obtain the target transformation matrix of the robot's end effector in the base coordinate system and the initial pose data in the camera coordinate system; for example: The operator uses a teach pendant or interactive interface to precisely control the robot's end effector (such as a nib or current tool feature point) to align with a fixed feature point (or a specific physical target) on a vision calibration board. At the moment of alignment, the operator reads and records the target change matrix of the robot's end flange in the base coordinate system from the robot controller, denoted as T_base_end.

[0062] The vision system is synchronously triggered to capture images of the current calibration board or feature target through the camera. After image processing and pose calculation, the initial pose data of the target object in the camera coordinate system is obtained, denoted as T_cam_obj.

[0063] In practice, the robot can also be controlled to collect multiple sets of T_base_end and T_cam_obj.

[0064] Based on the transformation matrix, and combined with the end pose data and the target pose number, the latest tool matrix is ​​obtained.

[0065] If only one set of T_base_end and T_cam_obj was collected, it can be directly based on T_end_tool = (T_base_end) -1 The latest tool matrix is ​​obtained by calculating × T_base_cam × T_cam_obj.

[0066] If multiple sets of T_base_end and T_cam_obj are collected, then T_end_tool = (T_base_end) can be used. -1 The transformation matrix for each group is calculated using × T_base_cam × T_cam_obj, and outliers with significant bias are removed using statistical methods (such as the 3σ principle). The remaining valid samples are then averaged (arithmetic mean for translation vectors, and quaternion spherical mean or Lie algebraic mean for rotation matrices) to obtain a robust state-of-the-art tool matrix.

[0067] Update the latest tool matrix to the pre-stored calibration configuration file; for example: The results are written to the calibration configuration file tcp_calibration.json, with the specific storage path mapped to tools.<tool_name> The .results file contains not only the complete 4×4 transformation matrix data, but also the intuitive tcp_pose field, which is updated synchronously.

[0068] In practical implementation, the aforementioned S101-S104 are deeply embedded into the task orchestration and precision screw-driving process of the dual-arm robot. Through deep integration with motion sequence scheduling, hardware status monitoring, adaptive speed adjustment, and asynchronous service calls, a highly robust and complete assembly execution chain is constructed. The specific implementation process is as follows: After the task is started, the initialization phase begins. The dual-arm robot performs a comprehensive self-check procedure, including but not limited to the status of the servo drives, the I / O signal feedback of the end effectors (such as electric screwdrivers and suction cups), and the connectivity of peripherals such as safety light curtains. After the self-check passes, the dual arms automatically perform a homing operation to establish a unified absolute coordinate system reference, laying the foundation for subsequent high-precision collaborative operations.

[0069] Using a state machine model, the task is advanced strictly in the order of "perception request → pose fusion → motion execution → positioning verification → assembly action → result recording": Trigger S101 to accurately distribute the recognition task to the corresponding left / right arm vision node through the service routing mechanism, and obtain the initial pose data of the target workpiece or assembly hole.

[0070] S103 performs pose fusion calculation on the initial data, and combines unit standardization conversion (such as millimeter to meter, radian unification) and asynchronous timeout control to generate target pose data that can be executed by the dual arms.

[0071] Control commands are generated based on the target pose data, controlling the two arms to move collaboratively to the target position according to the planned trajectory. Upon arrival, assembly is not immediately performed; instead, a "position verification" mechanism is triggered. Step S101 collects the current actual pose, and the target error is recalculated according to the algorithm in Step S103. If the error is within the allowable threshold, the process proceeds to the next step; otherwise, positioning is considered a failure.

[0072] After the review is approved, the specific assembly process is executed. For example, the right arm performs screw fastening, and the left arm performs workpiece support or placement alignment.

[0073] After the assembly process is completed, record the final execution results (such as the torque curve of screw tightening, final fit, etc.) and update the task status to complete.

[0074] To ensure production safety and system stability, any failure in any step (such as visual recognition timeout, movement deviation, or stripped threads) will immediately trigger the preset exception handling logic: Based on the error type, attempt to automatically recover, such as triggering "retake" (reacquire images), "recalculate" (replan the trajectory), or "slow down" (reduce the movement speed and try again).

[0075] If automatic recovery fails multiple times, the current task will be suspended and manual intervention will be requested. Simultaneously, the entire process's operation logs, error codes, and on-site data will be written to the task record database in real time, facilitating subsequent problem tracing and process optimization.

[0076] This process enables the stable implementation of everything from the underlying pose algorithm to the upper-level dual-arm assembly operations, significantly improving the intelligence level and success rate of automated assembly lines.

[0077] This application fundamentally differs from the coarse-grasping mode of traditional automated production lines that relies on fixed offsets, achieving flexible and precise operations in complex scenarios. A unified mathematical model simultaneously supports both visual grasping and visual placement, significantly improving the algorithm's generalization ability and reusability. To adapt to the diverse operational needs of industrial sites, an engineering-calibrated asset mechanism is further proposed. This mechanism provides structured and persistent storage of the extrinsic parameters (T_base_cam) of the camera and robot base, as well as the tool extrinsic parameters (T_end_tool) of various end effectors, effectively supporting rapid and seamless switching between different parts and tools. Addressing the pain point of high integration difficulty in industrial settings, a steady-state mechanism is designed, including namespace inheritance, asynchronous timeout control, unified physical units, and three-level intrinsic parameter backoff, significantly reducing the failure rate during system integration. Furthermore, an innovative object coordinate system translation offset mechanism is introduced. By multiplying the translation offset by the right in the transformation chain, precise compensation for micro-assembly targets such as feature points and hole positions is achieved, meeting the process requirements of high-precision assembly.

[0078] Figure 2 A structural diagram of a robot control device provided in an embodiment of this application is shown below. Figure 2 As shown, based on the robot control method provided in the preceding embodiments, this application also provides a robot control device including: The initial pose determination module is used to determine the initial pose data of the object to be operated by the robot end effector in the camera coordinate system. The acquisition module is used to acquire the transformation matrix from the robot's base coordinate system to the camera coordinate system, and the tool matrix from the robot's end effector to the tool coordinate system; The target pose determination module is used to adjust the initial pose data according to the transformation matrix and the tool matrix, and according to the execution task corresponding to the object to be operated by the robot end, to obtain the target pose data of the robot end in the base coordinate system. The execution module is used to generate control commands based on the target pose data, perform motion control on the robot, and complete the execution task.

[0079] As an optional embodiment, the initial pose determination module specifically includes: The acquisition unit is used to acquire part data corresponding to the requested field; the part data includes at least part symmetry axis constraint data and part three-dimensional reference points; The detection unit is used to perform target detection on the image data collected by the robot, and obtain multiple candidate boxes of target parts and the confidence level of each candidate box of target parts. A filtering unit is used to filter the multiple candidate boxes of target parts based on confidence level to determine the selected box of the target part; The calculation unit is used to obtain the initial pose data of the object to be operated by the robot end in the camera coordinate system based on the target part selection box, the image data and the part data.

[0080] As an optional embodiment, the computing unit is specifically used for: Feature points are obtained by extracting features from the selected bounding box of the target part; Based on the feature points, the three-dimensional reference points of the part, and the intrinsic parameter data of the image data, the pose data to be corrected is obtained. The rotational branches of the pose data to be corrected are corrected using the symmetry axis constraint data of the part, so as to obtain the initial pose data of the object to be operated by the robot end in the camera coordinate system.

[0081] As an optional embodiment, the target pose determination module is specifically used for: If the task to be performed is to grasp an object to be manipulated, then the initial pose data is adjusted based on the transformation matrix and the tool matrix using the homogeneous transformation formula to obtain the target transformation matrix of the robot end effector in the base coordinate system. The rotation matrix in the target transformation matrix is ​​converted into Euler angles, and the Euler angles are combined with the translation amount in the target transformation matrix to obtain the target pose data of the robot end effector in the base coordinate system; If the task to be performed is to place an object to be operated, then based on the transformation matrix, the initial pose data is adjusted to obtain the absolute pose data of the object in the base coordinate system; Based on the transformation matrix and the position data of the acquisition device in the robot, the absolute pose data of the target placement position of the object to be operated in the base coordinate system is obtained. Based on the absolute pose data of the object in the base coordinate system and the absolute pose data of the target placement position corresponding to the object to be operated in the base coordinate system, the relative transformation matrix from the current position of the object to be operated to the target placement position is obtained. Based on the relative transformation matrix from the current position of the object to be operated to the target placement position and the end-effector pose data, the target pose data of the robot end-effector in the base coordinate system is obtained; the end-effector pose data includes the end-effector pose data of the robot end-effector in the base coordinate system.

[0082] As an optional embodiment, the homogeneous transformation formula is as follows: T_base_end_target = T_base_cam × T_cam_obj × (T_end_tool) -1 ; Where T_base_end_target represents the target transformation matrix of the robot end effector in the base coordinate system; T_base_cam represents the transformation matrix from the robot base coordinate system to the camera coordinate system; T_cam_obj represents the initial pose data; and T_end_tool represents the tool matrix from the robot end effector to the tool coordinate system.

[0083] As an optional embodiment, the device further includes: The conversion module is used to convert the displacement in the target pose data to meters, while keeping the attitude angle in radians, and to make service calls using an asynchronous request and timeout waiting mechanism.

[0084] As an optional embodiment, the device further includes: The update module is used to control the robot to move to a preset calibration position and obtain the target transformation matrix of the robot end effector in the base coordinate system and the initial pose data in the camera coordinate system; based on the transformation matrix, combined with the end effector pose data and the target pose data, the latest tool matrix is ​​obtained; the latest tool matrix is ​​updated to the pre-stored calibration configuration file.

[0085] This application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a robot control method.

[0086] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a robot control method.

[0087] This application provides a computer program product, including a computer program that, when executed by a processor, implements a robot control method.

[0088] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device and equipment embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device and equipment embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components indicated as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0089] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A robot control method, characterized in that, The method includes: Determine the initial pose data of the object to be manipulated by the robot end effector in the camera coordinate system; Obtain the transformation matrix from the robot's base coordinate system to the camera coordinate system, and the tool matrix from the robot's end effector to the tool coordinate system; Based on the transformation matrix and the tool matrix, the initial pose data is adjusted according to the task to be performed on the object to be operated by the robot end effector to obtain the target pose data of the robot end effector in the base coordinate system. Based on the target pose data, control commands are generated to control the robot's motion and complete the task.

2. The robot control method according to claim 1, characterized in that, The determination of the initial pose data of the object to be manipulated by the robot end effector in the camera coordinate system specifically includes: Obtain the part data corresponding to the requested field; the part data includes at least the part's symmetry axis constraint data and the part's three-dimensional reference points; Target detection is performed on the image data collected by the robot to obtain multiple candidate boxes for target parts and the confidence score corresponding to each candidate box; The multiple candidate boxes for target parts are filtered based on confidence level to determine the selected box for the target part; Based on the target part selection box, the image data, and the part data, the initial pose data of the object to be operated by the robot end in the camera coordinate system is obtained.

3. The robot control method according to claim 2, characterized in that, The process of obtaining the initial pose data of the object to be manipulated by the robot end effector in the camera coordinate system based on the target part selection box, the image data, and the part data specifically includes: Feature points are obtained by extracting features from the selected bounding box of the target part; Based on the feature points, the three-dimensional reference points of the part, and the intrinsic parameter data of the image data, the pose data to be corrected is obtained. The rotational branches of the pose data to be corrected are corrected using the symmetry axis constraint data of the part, so as to obtain the initial pose data of the object to be operated by the robot end in the camera coordinate system.

4. The robot control method according to claim 1, characterized in that, The step of adjusting the initial pose data based on the transformation matrix and the tool matrix, according to the task to be performed on the object to be operated by the robot end effector, to obtain the target pose data of the robot end effector in the base coordinate system, specifically includes: If the task to be performed is to grasp an object to be manipulated, then the initial pose data is adjusted based on the transformation matrix and the tool matrix using the homogeneous transformation formula to obtain the target transformation matrix of the robot end effector in the base coordinate system. The rotation matrix in the target transformation matrix is ​​converted into Euler angles, and the Euler angles are combined with the translation amount in the target transformation matrix to obtain the target pose data of the robot end effector in the base coordinate system; If the task to be performed is to place an object to be operated, then based on the transformation matrix, the initial pose data is adjusted to obtain the absolute pose data of the object in the base coordinate system; Based on the transformation matrix and the position data of the acquisition device in the robot, the absolute pose data of the target placement position of the object to be operated in the base coordinate system is obtained. Based on the absolute pose data of the object in the base coordinate system and the absolute pose data of the target placement position corresponding to the object to be operated in the base coordinate system, the relative transformation matrix from the current position of the object to be operated to the target placement position is obtained. Based on the relative transformation matrix from the current position of the object to be operated to the target placement position and the end-effector pose data, the target pose data of the robot end-effector in the base coordinate system is obtained; the end-effector pose data includes the end-effector pose data of the robot end-effector in the base coordinate system.

5. The robot control method according to claim 4, characterized in that, The homogeneous transformation formula is as follows: T_base_end_target = T_base_cam × T_cam_obj × (T_end_tool) -1 ; Where T_base_end_target represents the target transformation matrix of the robot end effector in the base coordinate system; T_base_cam represents the transformation matrix from the robot base coordinate system to the camera coordinate system; T_cam_obj represents the initial pose data; and T_end_tool represents the tool matrix from the robot end effector to the tool coordinate system.

6. The robot control method according to claim 1, characterized in that, The method further includes: The displacement in the target pose data is converted to meters, the attitude angle is kept in radians, and an asynchronous request plus timeout waiting mechanism is used for service calls.

7. The robot control method according to claim 1, characterized in that, The method further includes: Control the robot to move to the preset calibration position, and obtain the target change matrix of the robot end in the base coordinate system and the initial pose data in the camera coordinate system; Based on the transformation matrix, combined with the end pose data and the target pose number, the latest tool matrix is ​​obtained; Update the latest tool matrix to the pre-stored calibration configuration file.

8. A robot control device, characterized in that, The robot control device includes: The initial pose determination module is used to determine the initial pose data of the object to be operated by the robot end effector in the camera coordinate system. The acquisition module is used to acquire the transformation matrix from the robot's base coordinate system to the camera coordinate system, and the tool matrix from the robot's end effector to the tool coordinate system; The target pose determination module is used to adjust the initial pose data according to the transformation matrix and the tool matrix, and according to the execution task corresponding to the object to be operated by the robot end, to obtain the target pose data of the robot end in the base coordinate system. The execution module is used to generate control commands based on the target pose data, perform motion control on the robot, and complete the execution task.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the robot control method according to any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the robot control method according to any one of claims 1-7.

Images