Reinforcement learning control method and device for mechanical arm based on optical positioning system guidance

Abstract

The application belongs to the field of mechanical arm control, and discloses a mechanical arm reinforcement learning control method and device based on optical positioning system guidance, which can realize instant response and continuous adaptation of the mechanical arm to a new scene. The method comprises the following steps: determining an actual pose and a target pose of a calibrated mechanical arm according to an image stream collected by a calibrated binocular camera; calculating a position error and an attitude error according to the actual pose and the target pose; splicing the position error, the attitude error, the actual pose, the target pose, a pose change amount and a joint state of the mechanical arm to obtain a current state vector; inputting the current state vector into a pre-trained SAC policy network model to obtain an action instruction; determining a next state vector and a multi-target reward according to the action instruction; and iteratively training the SAC policy network model according to the current state vector, the action instruction, the next state vector and the multi-target reward to obtain a mechanical arm reinforcement learning control model.

Description

Technical Field

[0001] This application relates to the field of robotic arm control, and in particular to a reinforcement learning control method and device for robotic arms guided by an optical positioning system. Background Technology

[0002] In cutting-edge fields such as surgical robots and precision assembly, the requirements for the operational precision of robotic arms have reached the sub-millimeter level. Traditional control methods (such as teach-and-playback and offline programming) are stable in structured environments, but they lack adaptability when faced with random changes in target pose or environmental disturbances, resulting in a significant decrease in execution accuracy and task success rate.

[0003] Reinforcement learning offers a new path for intelligent control of robotic arms, enabling agents to autonomously explore optimal strategies through interaction with the environment without relying on precise kinematic models. However, existing reinforcement learning methods mostly use RGB cameras as sensor input, and their accuracy is significantly affected by lighting and texture, reaching only millimeter to centimeter levels, which is insufficient for high-precision scenarios. Optical positioning instruments track reflective marker balls using binocular infrared stereo vision, enabling real-time calculation of the target's six-DOF pose. They offer advantages such as high accuracy and immunity to visible light interference and have been successfully applied in fields such as surgical navigation. However, current robotic arm control based on optical positioning instruments still adheres to traditional control paradigms, relying on precise kinematic models and error compensation strategies, and lacks autonomous learning and adaptive capabilities. Summary of the Invention

[0004] This application provides a reinforcement learning control method and device for robotic arms based on an optical positioning system, which can enable robotic arms to respond instantly and adapt continuously to new scenarios.

[0005] The first aspect of this application proposes a reinforcement learning control method for a robotic arm guided by an optical positioning system, comprising: The actual pose of the calibrated robotic arm and the target pose are determined based on the image stream acquired by the calibrated binocular camera. Calculate the position error and attitude error based on the actual pose and the target pose; The position error, the posture error, the actual pose, the target pose, the pose change, and the joint state of the robotic arm are concatenated to obtain the current state vector. The current state vector is input into a pre-trained SAC policy network model to obtain action instructions; The next state vector and multi-objective reward are determined based on the action instructions; The SAC policy network model is iteratively trained based on the current state vector, the action command, the next state vector, and the multi-objective reward to obtain a reinforcement learning control model for the robotic arm, thereby achieving control of the robotic arm.

[0006] A second aspect of this application provides a reinforcement learning control device for a robotic arm guided by an optical positioning system, comprising: The first determining module is used to determine the actual pose of the calibrated robotic arm and the target pose based on the image stream acquired by the calibrated binocular camera. The calculation module is used to calculate the position error and attitude error based on the actual pose and the target pose; The stitching module is used to stitch together the position error, the posture error, the actual pose, the target pose, the pose change, and the joint state of the robotic arm to obtain the current state vector. The second determining module is used to input the current state vector into a pre-trained SAC policy network model to obtain action instructions; The third determining module is used to determine the next state vector and multi-target reward based on the action instruction; The training module is used to iteratively train the SAC policy network model based on the current state vector, the action command, the next state vector, and the multi-objective reward to obtain a reinforcement learning control model for the robotic arm, so as to achieve control of the robotic arm.

[0007] In one possible design, the first determining module is specifically used for: The image stream is identified to obtain the current point set; The current point set is filtered to obtain effective point set combinations; Determine whether there exists a target set of valid points in the combination of valid points whose number of valid points is greater than or equal to a preset threshold. If so, the actual pose and the target pose are determined based on the current point set and the local three-dimensional coordinate set; If not, the fusion predicted pose of the robotic arm is determined based on the Long Short-Term Memory network, and the fusion predicted pose is determined as the actual pose. The target pose is then determined based on the current point set and the local three-dimensional coordinate set.

[0008] In one possible design, the first determining module determines the fused predicted pose of the robotic arm based on a long short-term memory network, including: The fused predicted pose is determined using the following formula:

[0009] in, For the fused predicted pose, As the confidence level weight, This represents the theoretical pose of the robotic arm's end effector. The predicted pose of the robotic arm is predicted by the Long Short-Term Memory network based on the historical pose queue.

[0010]

[0011] Wherein, LSTM refers to the Long Short-Term Memory network. Here, k represents the historical pose queue, and k is the length of the historical window. The pose of the robotic arm at time t-1, determined by the infrared optical positioning system. The actual angles of the encoders at each joint of the robotic arm. Let be the homogeneous transformation matrix of the first joint in the robotic arm relative to the base of the robotic arm.

[0012] In one possible design, the computing module is specifically used for: The pose error is determined by the following formula:

[0013] in, The pose error is... These are the position coordinates in the actual pose. The coordinates are the position coordinates in the target pose; The attitude error is determined by the following formula:

[0014] in, The attitude error is... This refers to the actual pose. Let the target pose be... This is a function that converts a rotation matrix into a three-dimensional rotation vector.

[0015] In one possible design, the third determining module is specifically used for: The target position error and target attitude error are determined based on the action command; Based on the target position error and the target attitude error, the multi-target reward is determined using the following formula:

[0016] in, For the aforementioned multi-objective reward, The target position error is... The target attitude error is... The angular velocities of each joint of the robotic arm when executing the aforementioned action command. This is the position error penalty coefficient. This is the attitude error penalty coefficient. The motion smoothing penalty coefficient, The incentive coefficient for task success. The process reward coefficient, The process reward value. This is a task success indicator. , The pose error measured before executing the action command. The pose error is measured after the action command is executed.

[0017] In one possible design, the third determining module determines the target position error and the target attitude error based on the action command, including: The motion command is sent to the robotic arm so that the robotic arm executes the motion command and obtains an updated end effector posture; Based on the updated end effector posture, determine the new actual pose and the new target pose of the robotic arm; The target position error and the target attitude error are calculated based on the new actual pose and the new target pose.

[0018] In one possible design, the training module is further used for: A virtual environment is constructed, which includes a robotic arm model, a virtual optical positioning sensor, and the target task object and environment. The virtual optical positioning sensor includes measurement noise, data latency, and marker ball occlusion. The parameters in the virtual environment are adjusted by domain randomization, and the SAC policy network model is pre-trained to obtain the pre-trained SAC policy network model.

[0019] A third aspect of this application proposes a reinforcement learning control device for a robotic arm, comprising: processor; Memory, used to store computer programs; When the processor executes the computer program, it implements the reinforcement learning control method for a robotic arm guided by an optical positioning system as described in any of the above embodiments.

[0020] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the reinforcement learning control method for a robotic arm guided by an optical positioning system as described in any of the above embodiments.

[0021] The fifth aspect of this application proposes a computer program product, comprising a computer program that, when executed by a processor, implements the reinforcement learning control method for a robotic arm guided by an optical positioning system as described in any of the above embodiments. In the embodiments provided in this application, the position error and attitude error calculated from the actual pose measured by the optical positioning device and the target pose, together with the joint physical state, are concatenated into a state vector. This vector is then input into the SAC policy network to directly map action commands, making the generation of the control strategy entirely data-driven rather than model-driven. Simultaneously, the network is iteratively trained using multi-objective rewards calculated from the measured feedback after the action is executed. This allows the SAC policy network to continuously update its parameters based on the reward and penalty signals of real-time errors, autonomously exploring and mastering the optimal correction strategy in different tasks and environments. This transforms error compensation from a fixed empirical formula into a dynamic adaptive learning process. Finally, when faced with unmodeled disturbances or task changes, the trained SAC policy network no longer relies on a preset error compensation table but autonomously decides the optimal action based on the current measured state. This evolves the generation of the control strategy from a fixed empirical formula into a dynamic self-optimization process, enabling the robotic arm to respond instantly and continuously adapt to new scenarios. Attached Figure Description

[0022] Figure 1 A schematic diagram of a system architecture for a reinforcement learning control system for a robotic arm guided by an optical positioning system, provided in one embodiment of this application. Figure 2 A flowchart illustrating a reinforcement learning control method for a robotic arm guided by an optical positioning system, provided in one embodiment of this application; Figure 3 A schematic diagram of a reinforcement learning control device for a robotic arm guided by an optical positioning system, provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application; The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0024] Those skilled in the art will understand that, unless explicitly stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in the specification of this application means the presence of features, integers, steps, operations, elements, modules, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, modules, components, and / or groups thereof. It should be understood that when an element is “connected” or “coupled” to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein may include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any modules and all combinations of one or more associated listed items.

[0025] Those skilled in the art will understand that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0026] First, the terms used in this application will be explained as follows: 1. Optical Tracker / Optical Localizer: A high-precision three-dimensional spatial pose measurement device based on the principle of binocular infrared stereo vision is proposed. It uses two infrared cameras to simultaneously photograph a reflective marker ball fixed on a target object and uses the principle of triangulation to calculate the six-degree-of-freedom pose (position + attitude) of the target object in real time. It has the advantages of high precision, fast sampling frequency, strong anti-electromagnetic interference capability and no influence from visible light conditions.

[0027] 2. Near-infrared optical positioning system: An optical navigation device based on binocular stereo vision for tracking and positioning, built on an FPGA technology platform, achieves a tracking accuracy of 0.08mm (RMS), a sampling frequency of 96Hz, supports both active and passive marker types, can track up to 200 markers simultaneously, and can directly output data such as the 3D coordinates of the markers and the 6D pose of the tool.

[0028] 3. Reflective Marker Sphere: A spherical marker coated with a high-reflectivity retroreflective material (such as 3M Scotchlite) is fixed to the end of a robotic arm or the target object. It can reflect incident infrared light back to the light source along the incident direction, making it easy for the infrared camera of the optical positioning instrument to quickly, clearly and stably extract its center projection position in the image.

[0029] 4. Rigid Body Tool Array (ToolArray / RigidBody): A rigid body, consisting of at least three (preferably four or more) non-collinear reflective marker spheres arranged in a fixed geometry, is fixed to the end effector of a robotic arm or to a target object. An optical locator uniquely identifies the rigid body tool array by recognizing the fixed topological relationships (distances and angles between the spheres) between each marker sphere and calculates its 6-DoF pose in real time.

[0030] 5. Six degrees of freedom pose (6-DoFPose): The parameter combination describing the complete position and orientation of a rigid body in three-dimensional space includes: position (3 degrees of freedom): translation along the X, Y, and Z axes; orientation (3 degrees of freedom): rotation angles about the X, Y, and Z axes. There are a total of 6 independent numerical parameters.

[0031] 6. Axis-Angle Representation: This method for representing attitude differences in three-dimensional space transforms a 3×3 rotation matrix into a three-dimensional vector. The direction of this vector represents the direction of the rotation axis, and the length (i.e., the magnitude) of the vector represents the angle of rotation (in radians) around that axis. Compared to traditional Euler angles, this rotation vector representation method has unique and continuous values ​​under any attitude, avoiding the gimbal lock-up problem.

[0032] 7. Reinforcement Learning (RL): An intelligent agent learns the optimal decision-making strategy through continuous interaction with the environment via trial and error, aiming to maximize cumulative rewards. In this invention, a robotic arm acts as the intelligent agent, optical positioning data constitutes the environmental state, control commands constitute the actions, and pose error determines the reward.

[0033] 8. SoftActor-Critic (SAC): A deep reinforcement learning algorithm maximizes policy entropy (i.e., maintains a certain degree of exploration randomness) while optimizing cumulative reward. It has advantages such as good training stability, high sample utilization, and fast convergence speed, and is particularly suitable for continuous control tasks of robotic arms.

[0034] 9. Long Short-Term Memory (LSTM) Network: A recurrent neural network structure with memory capability can effectively remember input information from several past moments and use it for output inference at the current moment. This invention utilizes LSTM to predict the current pose based on historical pose change patterns when optical positioning data is briefly lost, thus maintaining the continuity of control commands.

[0035] 10. Domain Randomization: A training technique to improve the ability of simulation training strategies to transfer to real environments involves systematically and randomly varying various parameters (such as measurement noise level, target pose range, load weight, etc.) during simulation training, so that the policy network has seen a sufficiently diverse range of scenarios during training and can adaptively handle differences when transferred to the real environment.

[0036] 11. Kabsch Algorithm: A standard algorithm for finding the optimal rigid body transformation (rotation matrix + translation vector) between two sets of 3D point sets. By minimizing the mean square error between the two sets of corresponding points, the optimal rotation matrix and translation vector are solved using SVD decomposition, which is the core algorithm for pose calculation in optical positioning systems.

[0037] 12. Gimbal Lock When attitude is described using Euler angles (three independent angles rotating about the X, Y, and Z axes respectively), the yaw axis coincides with the roll axis when the pitch angle approaches ±90°, leading to the loss of one rotational degree of freedom and control failure. This invention uses a rotation vector to represent attitude error, fundamentally avoiding this problem.

[0038] 13. Reprojection Error The pixel distance between the known 3D spatial points, obtained through camera calibration, and the actual detected image points is the core indicator used to quantify the accuracy of camera calibration.

[0039] Please see Figure 1 , Figure 1 The system architecture diagram of the reinforcement learning control system for a robotic arm guided by an optical positioning system provided in this application includes: 1. Robotic arm body: a six- or seven-degree-of-freedom articulated robotic arm, with each joint equipped with servo drives and high-precision encoders for performing the positional motion of the end effector.

[0040] 2. Optical Positioning System (Core Sensing Layer): A near-infrared optical positioning system is adopted. This system is an optical navigation device that tracks and positions objects based on binocular stereo vision and uses an FPGA technology platform to achieve synchronous image acquisition.

[0041] 3. Computing and Control Platform (Decision Layer): Equipped with a high-performance CPU and GPU, running an optical positioning and pose calculation engine, a reinforcement learning inference network, and a real-time motion control library.

[0042] 4. Communication Link: The near-infrared optical positioning system transmits the 3D coordinates of the marker points and the 6D pose data of the tool to the industrial control computer via Gigabit Ethernet (1000Mbps) or USB 3.0 (5.0Gbps). The industrial control computer is connected to the robotic arm controller via EtherCAT bus, forming a closed-loop control link of "optical positioning perception → pose calculation → state encoding → strategy reasoning → motion execution".

[0043] The following description focuses on the reinforcement learning control method for a robotic arm guided by an optical positioning system, as provided in the embodiments of this application. This reinforcement learning control device can be a server or a service unit within a server, without any specific limitation. For the sake of simplicity, the following description uses a server as an example to illustrate the reinforcement learning control device for a robotic arm guided by an optical positioning system.

[0044] Reference Figure 2 , Figure 2 This is a flowchart illustrating a reinforcement learning control method for a robotic arm guided by an optical positioning system, provided in an embodiment of this application. The method includes: 201. Determine the actual pose and target pose of the calibrated robotic arm based on the image stream acquired by the calibrated binocular camera.

[0045] In this embodiment, the server can first calibrate the binocular camera, the target object, and the robotic arm to obtain a calibration configuration file. This configuration file includes a fixed transformation matrix between the optical positioning device coordinates and the robotic arm base coordinate system. The intrinsic and extrinsic parameters of the binocular camera, the first set of local three-dimensional coordinates of the first group of reflective marker spheres in the first rigid body tool coordinate system, and the second set of local three-dimensional coordinates of the second group of reflective marker spheres in the second rigid body tool coordinate system are described below. The first group of reflective marker spheres and the first rigid body tool coordinate system are associated with the robotic arm, and the second group of reflective marker spheres and the second rigid body tool coordinate system are associated with the target object. I. Calibration of the stereo camera: 1. Single-target positioning: The server takes high-precision chessboard or dot calibration board images (collecting multiple sets of images in different poses) from the optical positioning system in front of the binoculars in different postures, and calculates the focal length, principal point coordinates (the projection position of the optical center on the image), radial distortion coefficient, and tangential distortion coefficient (used for subsequent correction of sub-pixel extraction deviations) of the binocular camera. 2. Dual-target positioning: Using the corresponding projection points of the same calibration plate in the stereo camera, calculate the rigid transformation matrix between the stereo cameras to obtain the rotation matrix and translation matrix (spatial pose of the right camera relative to the left camera) and baseline distance; II. Generation of Rigid Body Tool Array Definition File: A rigid tool array is fixed to both the end effector of the robotic arm and the target object. This rigid tool array includes at least four non-collinear reflective marker spheres. Using the measurement mode of a coordinate measuring machine or optical positioning system, the three-dimensional coordinates of the center of each reflective marker sphere relative to the origin of the rigid body's local coordinate system are determined. Where n is greater than or equal to 4 (selecting more than 4 non-collinear reflective marker spheres ensures that at least 3 spheres are available for pose calculation even when some spheres are occluded). Calculate the spatial Euclidean distance between any two reflective marker spheres to form a template set, and then use the three-dimensional coordinates... The template collection is saved as a rigid body definition file in a specific format.

[0046] III. Hand-eye alignment: By using hand-eye calibration, a fixed transformation matrix is ​​determined between the coordinate system of the optical positioning instrument and the coordinate system of the robotic arm base, thus unifying the pose data measured by the optical instrument to the kinematic reference of the robotic arm.

[0047] It should be noted that if the optical positioning device is fixed externally and does not move with the robotic arm, then two unknown transformations need to be solved simultaneously: the transformation from the optical positioning device coordinate system to the robotic arm base coordinate system, and the transformation from the end effector of the robotic arm to the rigid array of reflective marker spheres. These will be explained in detail below: 1. Establish various coordinate systems: The system comprises a robot arm base coordinate system {B}, a robot arm end effector coordinate system {E}, an optical positioner coordinate system {C}, and a tool coordinate system {T}. The robot arm base coordinate system serves as the fixed reference system for the robot arm, upon which all joint kinematics are based. The robot arm end effector coordinate system is located at the center of the end flange and moves with the robot arm. The optical positioner coordinate system serves as the measurement reference system for the binocular camera and is fixed to an external support. The tool coordinate system is fixed to the local coordinate system of the reflective marker spherical rigid body array at the end of the robot arm. 2. Complete Robot-World / Hand-Eye Calibration (AX=YB Model): At any given moment, starting from the robot arm base coordinate system {B}, it is possible to reach the tool coordinate system {T} along either of two paths: Path 1: Through robotic arm kinematics:

[0048] Path 2: Measurement via optical positioning:

[0049] in, This is the transformation matrix from the robot arm base to the end effector (calculated in real time by the robot arm's forward kinematics based on the joint angles). The transformation matrix from the optical positioning instrument to the target is calculated by the optical positioning instrument in real time based on the reflective marker ball array. This is a fixed transformation matrix from the robotic arm base to the optical positioning device. For the end effector of the robotic arm to the rigid array of reflective marker spheres, there is a fixed transformation matrix. Path 1 and Path 2 are closed at {T}, forming a closed-loop constraint:

[0050] in, Let be the fixed transformation matrix from the end effector of the robotic arm to the rigid array of reflective marker spheres. This is a fixed transformation matrix from the robotic arm base to the optical positioning device; For the robotic arm to move to two different poses i and j in space:

[0051]

[0052] After conversion, we get the following formula:

[0053] Solving the above formula yields X, and substituting X back into the closed-loop equation gives Y:

[0054] X and Y are thus calculated.

[0055] IV. Initialization and Zero-position Calibration of Robotic Arm Kinematic Parameters According to the robotic arm technical manual, the Denavit-Hartenberg (DH) parameters of each joint in the robotic arm are entered. These DH parameters include link length (along the X-axis), link torsion angle (around the X-axis), joint offset (along the Z-axis), and joint angle (around the Z-axis, variable). Move each joint of the robotic arm to the factory-calibrated physical zero position and record the encoder pulse readings of each joint as the zero position offset.

[0056] Finally, all the calibration results are summarized and packaged to generate a calibration completion configuration file, which includes: Focal length, principal point, and distortion parameters of the left and right cameras; Baseline distance, stereo correction rotation matrix; Fixed transformation matrix ; Names of rigid bodies and their three-dimensional coordinates and the topological distance matrix; The DH parameters for each joint include both joint limit and velocity amplitude safety values.

[0057] Afterwards, the server identifies the image stream synchronously acquired by the calibrated binocular cameras to obtain the current point set.

[0058] The current point set is a real-time image frame in the image stream acquired by the binocular camera. The optical positioning system uses the binocular triangulation formula to convert the two-dimensional pixel coordinates of all detected reflective points (whether they are the end effector of the robotic arm, the target object, or environmental interference) in the current frame into three-dimensional spatial coordinates, thus obtaining the current point set. The coordinates of this current point set are the measured three-dimensional coordinates relative to the optical positioning system. The details are explained below: 1. Sub-pixel extraction of the two-dimensional center of the marker sphere: The optical positioning system has two built-in infrared cameras, one on the left and one on the right. After calibration, the server synchronously acquires images through the two infrared cameras. Based on the FPGA platform, highly synchronous image acquisition is achieved, and the sub-pixel-level image coordinates of the center of each marked sphere are extracted using the gray-scale centroid method or ellipse fitting method. .

[0059] 2. Solving 3D coordinates using binocular triangulation: Based on the precisely calibrated intrinsic and extrinsic parameters of the binocular camera, the corresponding left and right marker points are matched using epipolar constraints, and the three-dimensional coordinates of each marker ball in the optical positioning instrument coordinate system are calculated. The core formula for solving three-dimensional coordinates is:

[0060] in, The focal length of the left camera. Baseline distance and These are the subpixel coordinates of the same marked ball in the left and right images.

[0061] It should be noted that the optical positioning system can directly output the three-dimensional coordinate data of each marker ball, without requiring the user to implement the binocular matching algorithm themselves.

[0062] It should also be noted that for a given pair of matching points and The three-dimensional coordinates of the corresponding marker point in the coordinate system of the optical positioning instrument The following formula can be used for calculation:

[0063]

[0064]

[0065] in, and This refers to the equivalent focal length of the left and right cameras in a binocular camera. Let B be the coordinates of the principal point and B be the corrected limit length. If lens distortion or non-ideal imaging occurs, the calibrated projection matrices P1 and P2 are used to directly solve the homogeneous trigonometric problem for the matching points.

[0066] Finally, based on the current point set, the actual pose and target pose of the robotic arm's end effector are determined, as follows: The actual pose of the end effector of the robotic arm is the current real position and orientation of the end effector in three-dimensional space. The target pose is the current position and orientation of the target work object (such as the entry point of a surgical drill, the workpiece to be gripped, the hole to be aligned, etc.) that the robotic arm needs to reach or operate on in three-dimensional space. It is another rigid body tool array (also containing at least 3 to 4 non-collinear reflective marker spheres) fixed on the target object, or a reference pose point that has been pre-calibrated in space.

[0067] After determining the current point set, the server can filter the current point set to obtain a valid point set. The filtering is based on the topological distance matrix during calibration, which means that all three-dimensional coordinates in the current point set (e.g., including 4 points at the end of the robotic arm + 4 points of the target object + 1 environmental interference point = 9 points) are used to find a set of points in all three-dimensional coordinates through the calibration configuration file (the distance between any two of the 4 marker balls at the end, such as 50mm for ball AB and 70mm for ball BC). If the pairwise distances between the points in this set perfectly match the rigid body topological distances in the configuration file, then this set of points is a valid point set. Thus, a combination of valid point sets can be obtained. Subsequently, the server matches the first set of reflective marker spheres in the first rigid body tool coordinate system and the second set of reflective marker spheres in the second rigid body tool coordinate system with each valid point set in the valid point set combination to determine whether there is a target valid point set in the valid point set combination where the number of valid points is greater than or equal to a preset threshold (e.g., 3). If a target valid point set exists within the valid point set combination, then the actual pose and the target pose are determined based on the current point set and the local 3D coordinate set, as follows: Known indivual( Mark the local coordinates of the sphere in the rigid body tool coordinate system The global coordinates output by the optical positioning system The optimal rigid body transformation is solved by SVD decomposition, i.e., the rotation matrix is ​​found. With translation vector Minimize the following objective function:

[0068] The Kabsch algorithm (i.e., the least squares registration algorithm based on SVD) is used to solve the problem and obtain the actual pose of the robotic arm's end effector. or the target pose of the target object .

[0069] If the target valid point set is not found in the valid point set combination, the fused predicted pose of the robotic arm is determined according to the Long Short-Term Memory network, and the fused predicted pose is determined as the actual pose. The target pose is then determined based on the current point set and the local 3D coordinate set. The details are as follows: The methods for determining the target pose have been explained in detail above, and will not be repeated here. This section only details how to determine the actual pose when there is no valid target point set in the valid point set combination: The fusion prediction pose is determined using the following formula:

[0070] in, For the fused predicted pose, As the confidence level weight, This represents the theoretical pose of the robotic arm's end effector. The predicted pose of the robotic arm is predicted by the Long Short-Term Memory Network based on historical window data;

[0071]

[0072] LSTM stands for Long Short-Term Memory network. Here, k represents the historical pose queue, and k is the length of the historical window. The pose of the robotic arm at time t-1, determined by the infrared optical positioning system. These are the actual angles of the encoders at each joint of the robotic arm. Let be the homogeneous transformation matrix of the first joint in the robotic arm relative to the base of the robotic arm.

[0073] It should be noted that the LSTM network simultaneously outputs an estimate of the uncertainty in the predicted pose. (e.g., the variance of the predicted distribution), and set a safety threshold. : when When the predicted fused pose is reliable, the server uses the fused predicted pose. It replaces the actual observation (i.e., the actual pose) to maintain the continuous and smooth output of control commands; when When the predicted fusion pose is unreliable, the server automatically downgrades to a safe stop mode, meaning the robotic arm decelerates and stops while maintaining its current pose, waiting for the optical positioning system to recover effective observations before continuing the task.

[0074] It should be noted that it can also determine whether the number of consecutive frame drops exceeds N frames. When the number of consecutive frame drops does not exceed N frames, the server also uses fusion to predict the pose. Instead of the actual observation (i.e. the actual pose), when the number of consecutive lost frames exceeds N frames, the robotic arm decelerates and stops while maintaining the current pose. Assuming the sampling frequency of the optical positioning instrument is 96Hz, the time corresponding to each frame is 1 / 96Hz = 10.4ms. That is, if N is 10, if the reflective marker ball is blocked within 104ms, it means that the predicted pose is unreliable, so the arm stops and maintains the current pose.

[0075] The above fault tolerance mechanism ensures that the server can maintain stable output in the case of brief occlusion (usually <100ms), avoiding robot arm shaking or loss of control due to pose data jumps; in the case of long-term occlusion or system abnormality, the safety stop mechanism ensures operational safety.

[0076] 202. Calculate the position error and attitude error based on the actual pose and the target pose.

[0077] In this embodiment, position error is the position deviation of the end effector of the robotic arm relative to the target object in three-dimensional space, and attitude error is the attitude deviation of the end effector of the robotic arm relative to the target object in three-dimensional space. The server determines the pose error using the following formula:

[0078] in, For pose error, These are the position coordinates in the actual pose. The position coordinates in the target pose; The attitude error is determined using the following formula:

[0079] in, For attitude error, This is the actual pose. For the target pose, This is a function that converts a rotation matrix into a three-dimensional rotation vector.

[0080] It should be noted that the position error is directly taken as the three-dimensional Euclidean distance difference, while the attitude error is represented by a rotation vector. The direction of this vector represents the direction of the rotation axis, and the length of the vector (i.e., the magnitude) represents the angle of rotation around that axis (in radians). Compared with traditional Euler angles (three independent rotations around the X / Y / Z axes respectively), the rotation vector representation has a unique and continuous value under any attitude, avoiding the gimbal lock problem and ensuring the stability of the reinforcement learning policy network training.

[0081] 203. The position error, attitude error, actual pose, target pose, pose change, and joint state of the robotic arm are concatenated to obtain the current state vector.

[0082] In this embodiment, the server can directly concatenate the position error, posture error, actual pose, target pose, pose change, and joint states of the robotic arm into a one-dimensional state tensor in a fixed order to obtain the current state vector. The specific formula is as follows:

[0083] in, Let this be the current state vector. The current angles of each joint of the robotic arm. This represents the angular velocity of each joint of the robotic arm. The angle of each joint reflects the current position of each joint, such as joint 1 rotating 30 degrees. The angular velocity of the joint reflects the current speed of rotation of each joint, such as the current speed of joint 2 being 0.5 rad / s. Here, n is the number of degrees of freedom of the robotic arm, which is directly read from the high-precision encoder built into the servo driver of each joint of the robotic arm body.

[0084] It should be noted that this position change refers to the change in position of the robotic arm's end effector between adjacent time points. Specifically, the server can calculate the position change using the following formula:

[0085] in, This represents the change in pose. Let be the actual pose of the robotic arm's end effector at time t. The actual pose of the end effector of the robotic arm at time t-1.

[0086] 204. Input the current state vector into the pre-trained SAC policy network model to obtain action instructions.

[0087] In this embodiment, the server can construct a virtual environment including a robotic arm model, virtual optical positioning sensors, and the target object and environment. The virtual optical positioning sensors include measurement noise, data latency, and marker ball occlusion. Parameters in the virtual environment are adjusted through domain randomization, and a pre-trained SAC policy network model is obtained. In other words, the server can construct a high-fidelity robotic arm model in the simulator and embed a virtual sensor model containing noisy and delayed optical sensors to simulate the situation where the reflective marker ball is randomly occluded. The target pose, load weight, and measurement noise figure are systematically and randomly varied in the simulation environment. Extensive trial-and-error training is performed using the SAC algorithm combined with HER (hindsight experience replay) to obtain the pre-trained SAC policy network model. Using the massive continuous trajectory data generated by the simulation, an LSTM network is trained offline, enabling the model to predict the current pose of the robotic arm based on its historical poses. The pre-trained policy network weights and LSTM weights are loaded, connected to real hardware, and online policy fine-tuning is performed with a very small number of real samples.

[0088] After obtaining the pre-trained SAC policy network model through simulation, the server can input the current state vector into the pre-trained SAC policy network model to obtain action instructions. .

[0089] 205. Determine the next state vector and multi-target reward based on the action instructions.

[0090] In this embodiment, the server can first determine the target position error and target pose error based on the action command, and then determine the next state vector and multi-target reward based on the target position error and target pose error. The details are as follows: After receiving the motion command, the server sends the motion command to the servo driver of the robotic arm via the EtherCAT bus to drive the joint motors to move the end effector of the robotic arm to approach the target, thereby obtaining the updated end effector posture of the robotic arm. Based on the updated end effector posture, the server determines the new actual pose and the new target pose of the robotic arm's end effector. The server then calculates the target position error and the target pose error based on the new actual pose and the new target pose.

[0091] It should be noted that the calculation of the actual pose, target pose, position error, and attitude error of the end effector has been clearly explained above. The calculation of the new actual pose, new target pose, target position error, and target attitude error here is similar to that described above, and will not be repeated here. The next state vector is determined in a similar way to the current state vector in step 204 above. After obtaining the corresponding features, they are concatenated into a one-dimensional vector in a fixed order. The server determines multi-objective rewards using the following formula:

[0092] in, For multi-objective rewards, For the target position error, For the target attitude error, This refers to the joint angular velocities of each joint of the robotic arm when executing motion commands. This is the position error penalty coefficient. This is the attitude error penalty coefficient. The motion smoothing penalty coefficient, The incentive coefficient for task success. The process reward coefficient, The process reward value. This is a task success indicator. , The pose error measured before executing the action command. The pose error measured after executing the action command is explained below, along with the meaning of each item in the multi-target reward: 1. Location accuracy bonus item : It is a three-dimensional coordinate difference vector. The three-dimensional straight-line distance between the end effector of the robotic arm and the target object is calculated directly from the position data output by the optical definition system. This incentive strategy aims to reduce positional deviation.

[0093] 2. Attitude accuracy bonus item : The magnitude of the attitude rotation vector is the total rotation angle (in radians) between the end-effector attitude and the target attitude, calculated from the attitude data output by the optical positioning system. This encourages strategies to reduce attitude deviation.

[0094] 3. Motion smoothness penalty item : It is the sum of the squares of the angular velocities of each joint, used to penalize violent movements and shaking, ensuring smooth movement of the robotic arm, and reducing energy consumption and mechanical wear.

[0095] 4. Incentives for Task Success : When the position error is less than a preset threshold And the attitude error is less than the preset threshold. At that time, a positive reward signal is given to guide the strategy to converge toward the target pose.

[0096] 5. Process Rewards :

[0097] in This is the combined error from the previous moment. The combined error at the current moment (a weighted sum of position and pose errors) is used as a process reward to encourage the agent to continuously reduce pose error and alleviate the sparse reward problem.

[0098] 206. The SAC policy network model is iteratively trained based on the current state vector, the action command, the next state vector, and the multi-objective reward to obtain the reinforcement learning control model of the robotic arm, so as to realize the control of the robotic arm.

[0099] In this embodiment, the server can obtain a real state transition tuple after the current iteration. The real state transition tuple is timestamped and then stored in the experience replay pool, which stores a large amount of simulation data and a small amount of real data. The server randomly draws a small batch of data (containing both simulated and real data) from the experience replay pool and calculates the current target loss based on the small batch of data. The target loss includes the mean square error between the true and predicted values ​​of the critic network and the loss of the policy network. Finally, the server performs backpropagation and parameter updates based on the target loss, and iteratively trains the updated SAC policy network model to obtain the reinforcement learning control model for the robotic arm, thereby achieving control of the robotic arm.

[0100] In summary, it can be seen that in the embodiments provided in this application, the position error and attitude error calculated from the actual pose measured by the optical positioning device and the target pose, together with the joint physical state, are concatenated into a state vector, which is then input into the SAC policy network to directly map action commands. This makes the generation of the control strategy entirely data-driven rather than model-driven. At the same time, the network is iteratively trained using the multi-objective reward calculated from the measured feedback after the action is executed. This allows the SAC policy network to continuously update its own parameters based on the reward and punishment signals of real-time error, autonomously explore and master the optimal correction strategy in different tasks and environments, thereby transforming error compensation from a fixed empirical formula into a dynamic adaptive learning process. Finally, when faced with unmodeled disturbances or task changes, the trained policy network no longer relies on a preset error compensation table, but autonomously decides the optimal action based on the current measured state, improving the model's autonomous learning and adaptive capabilities.

[0101] The embodiments of this application have been described above from the perspective of a reinforcement learning control method for a robotic arm guided by an optical positioning system. The embodiments of this application will now be described below from the perspective of a reinforcement learning control device for a robotic arm guided by an optical positioning system.

[0102] Please see Figure 3 , Figure 3 This application provides a virtual structural diagram of a robotic arm reinforcement learning control device 300 guided by an optical positioning system. The device includes: The first determining module 301 is used to determine the actual pose of the calibrated robotic arm and the target pose based on the image stream acquired by the calibrated binocular camera. Calculation module 302 is used to calculate position error and attitude error based on the actual pose and the target pose; The stitching module 303 is used to stitch together the position error, the posture error, the actual pose, the target pose, the pose change, and the joint state of the robotic arm to obtain the current state vector. The second determining module 304 is used to input the current state vector into a pre-trained SAC policy network model to obtain action instructions; The third determining module 305 is used to determine the next state vector and multi-target reward based on the action instruction; The training module 306 is used to iteratively train the SAC policy network model based on the current state vector, the action command, the next state vector, and the multi-objective reward to obtain a reinforcement learning control model for the robotic arm, so as to realize the control of the robotic arm.

[0103] In one possible design, the first determining module 301 is specifically used for: The image stream is identified to obtain the current point set; The current point set is filtered to obtain effective point set combinations; Determine whether there exists a target set of valid points in the combination of valid points whose number of valid points is greater than or equal to a preset threshold. If so, the actual pose and the target pose are determined based on the current point set and the local three-dimensional coordinate set; If not, the fusion predicted pose of the robotic arm is determined based on the Long Short-Term Memory network, and the fusion predicted pose is determined as the actual pose. The target pose is then determined based on the current point set and the local three-dimensional coordinate set.

[0104] In one possible design, the first determining module 301 determines the fusion predicted pose of the robotic arm based on a long short-term memory network, including: The fused predicted pose is determined using the following formula:

[0105] in, For the fused predicted pose, As the confidence level weight, This represents the theoretical pose of the robotic arm's end effector. The predicted pose of the robotic arm is predicted by the Long Short-Term Memory network based on the historical pose queue.

[0106]

[0107] Wherein, LSTM refers to the Long Short-Term Memory network. Here, k represents the historical pose queue, and k is the length of the historical window. The pose of the robotic arm at time t-1, determined by the infrared optical positioning system. The actual angles of the encoders at each joint of the robotic arm. Let be the homogeneous transformation matrix of the first joint in the robotic arm relative to the base of the robotic arm.

[0108] In one possible design, the computing module 302 is specifically used for: The pose error is determined by the following formula:

[0109] in, The pose error is... These are the position coordinates in the actual pose. The coordinates are the position coordinates in the target pose; The attitude error is determined by the following formula:

[0110] in, The attitude error is... This refers to the actual pose. Let the target pose be... This is a function that converts a rotation matrix into a three-dimensional rotation vector.

[0111] In one possible design, the third determining module 305 is specifically used for: The target position error and target attitude error are determined based on the action command; Based on the target position error and the target attitude error, the multi-target reward is determined using the following formula:

[0112] in, For the aforementioned multi-objective reward, The target position error is... The target attitude error is... The angular velocities of each joint of the robotic arm when executing the aforementioned action command. This is the position error penalty coefficient. This is the attitude error penalty coefficient. The motion smoothing penalty coefficient, The incentive coefficient for task success. The process reward coefficient, The process reward value. This is a task success indicator. , The pose error measured before executing the action command. The pose error is measured after the action command is executed.

[0113] In one possible design, the third determining module 305 determines the target position error and the target attitude error based on the action command, including: The motion command is sent to the robotic arm so that the robotic arm executes the motion command and obtains an updated end effector posture; Based on the updated end effector posture, determine the new actual pose and the new target pose of the robotic arm; The target position error and the target attitude error are calculated based on the new actual pose and the new target pose.

[0114] In one possible design, the training module 306 is further used for: A virtual environment is constructed, which includes a robotic arm model, a virtual optical positioning sensor, and the target task object and environment. The virtual optical positioning sensor includes measurement noise, data latency, and marker ball occlusion. The parameters in the virtual environment are adjusted by domain randomization, and the SAC policy network model is pre-trained to obtain the pre-trained SAC policy network model.

[0115] Reference Figure 4 This application also provides a computer device, which may be a server, and its internal structure may be as follows: Figure 4 As shown. The computer device includes a processor, memory, network interface, and database connected via a bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and the database. The internal memory provides an environment for the operation of the non-volatile storage medium and the execution of the computer programs. The database stores data such as a reinforcement learning control method for a robotic arm guided by an optical positioning system. The network interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps of a reinforcement learning control method for a robotic arm guided by an optical positioning system, which includes: The actual pose of the calibrated robotic arm and the target pose are determined based on the image stream acquired by the calibrated binocular camera. Calculate the position error and attitude error based on the actual pose and the target pose; The position error, the posture error, the actual pose, the target pose, the pose change, and the joint state of the robotic arm are concatenated to obtain the current state vector. The current state vector is input into a pre-trained SAC policy network model to obtain action instructions; The next state vector and multi-objective reward are determined based on the action instructions; The SAC policy network model is iteratively trained based on the current state vector, the action command, the next state vector, and the multi-objective reward to obtain a reinforcement learning control model for the robotic arm, thereby achieving control of the robotic arm.

[0116] One embodiment of this application also provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements a reinforcement learning control method for a robotic arm guided by an optical positioning system. The reinforcement learning control method for a robotic arm guided by an optical positioning system includes: The actual pose of the calibrated robotic arm and the target pose are determined based on the image stream acquired by the calibrated binocular camera. Calculate the position error and attitude error based on the actual pose and the target pose; The position error, the posture error, the actual pose, the target pose, the pose change, and the joint state of the robotic arm are concatenated to obtain the current state vector. The current state vector is input into a pre-trained SAC policy network model to obtain action instructions; The next state vector and multi-objective reward are determined based on the action instructions; The SAC policy network model is iteratively trained based on the current state vector, the action command, the next state vector, and the multi-objective reward to obtain a reinforcement learning control model for the robotic arm, thereby achieving control of the robotic arm.

[0117] In some possible implementations, various aspects of the methods provided in this application can also be implemented as a program product, which can be implemented using any combination of one or more readable media. The program product includes program code that, when run on a computer device, is configured to cause the computer device to perform the steps of the methods described above according to various exemplary embodiments of this application. The computer device can execute the reinforcement learning control method for a robotic arm guided by an optical positioning system as described in the embodiments of this application, which includes: The actual pose of the calibrated robotic arm and the target pose are determined based on the image stream acquired by the calibrated binocular camera. Calculate the position error and attitude error based on the actual pose and the target pose; The position error, the posture error, the actual pose, the target pose, the pose change, and the joint state of the robotic arm are concatenated to obtain the current state vector. The current state vector is input into a pre-trained SAC policy network model to obtain action instructions; The next state vector and multi-objective reward are determined based on the action instructions; The SAC policy network model is iteratively trained based on the current state vector, the action command, the next state vector, and the multi-objective reward to obtain a reinforcement learning control model for the robotic arm, thereby achieving control of the robotic arm.

[0118] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A reinforcement learning control method for a robotic arm guided by an optical positioning system, characterized in that, include: The actual pose of the calibrated robotic arm and the target pose are determined based on the image stream acquired by the calibrated binocular camera. Calculate the position error and attitude error based on the actual pose and the target pose; The position error, the posture error, the actual pose, the target pose, the pose change, and the joint state of the robotic arm are concatenated to obtain the current state vector. The current state vector is input into a pre-trained SAC policy network model to obtain action instructions; The next state vector and multi-objective reward are determined based on the action instructions; The SAC policy network model is iteratively trained based on the current state vector, the action command, the next state vector, and the multi-objective reward to obtain a reinforcement learning control model for the robotic arm, thereby achieving control of the robotic arm.

2. The method according to claim 1, characterized in that, The process of determining the actual pose and target pose of the calibrated robotic arm based on the image stream acquired by the calibrated binocular camera includes: The image stream is identified to obtain the current point set; The current point set is filtered to obtain effective point set combinations; Determine whether there exists a target set of valid points in the combination of valid points whose number of valid points is greater than or equal to a preset threshold. If so, the actual pose and the target pose are determined based on the current point set and the local three-dimensional coordinate set; If not, the fusion predicted pose of the robotic arm is determined based on the Long Short-Term Memory network, and the fusion predicted pose is determined as the actual pose. The target pose is then determined based on the current point set and the local three-dimensional coordinate set.

3. The method according to claim 2, characterized in that, The process of determining the fusion prediction pose of the robotic arm based on a long short-term memory network includes: The fused predicted pose is determined using the following formula: in, For the fused predicted pose, As the confidence level weight, This represents the theoretical pose of the robotic arm's end effector. The predicted pose of the robotic arm is predicted by the Long Short-Term Memory network based on the historical pose queue. Wherein, LSTM refers to the Long Short-Term Memory network. Here, k represents the historical pose queue, and k is the length of the historical window. The pose of the robotic arm at time t-1, determined by the infrared optical positioning system. The actual angles of the encoders at each joint of the robotic arm. Let be the homogeneous transformation matrix of the first joint in the robotic arm relative to the base of the robotic arm.

4. The method according to any one of claims 1 to 3, characterized in that, The calculation of position error and attitude error based on the actual pose and the target pose includes: The pose error is determined by the following formula: in, The pose error is... These are the position coordinates in the actual pose. The coordinates are the position coordinates in the target pose; The attitude error is determined by the following formula: in, The attitude error is... This refers to the actual pose. Let the target pose be... This is a function that converts a rotation matrix into a three-dimensional rotation vector.

5. The method according to any one of claims 1 to 3, characterized in that, The process of determining multi-target rewards based on the action instructions includes: The target position error and target attitude error are determined based on the action command; Based on the target position error and the target attitude error, the multi-target reward is determined using the following formula: in, For the aforementioned multi-objective reward, The target position error is... The target attitude error is... The angular velocities of each joint of the robotic arm when executing the aforementioned action command. This is the position error penalty coefficient. This is the attitude error penalty coefficient. The motion smoothing penalty coefficient, The incentive coefficient for task success. The process reward coefficient, The process reward value. This is a task success indicator. , The pose error measured before executing the action command. The pose error is measured after the action command is executed.

6. The method according to claim 5, characterized in that, The step of determining the target position error and the target attitude error according to the action command includes: The motion command is sent to the robotic arm so that the robotic arm executes the motion command and obtains an updated end effector posture; Based on the updated end effector pose, determine the new actual pose and new target pose of the robotic arm; The target position error and the target attitude error are calculated based on the new actual pose and the new target pose.

7. The method according to any one of claims 1 to 3, characterized in that, The method further includes: A virtual environment is constructed, which includes a robotic arm model, a virtual optical positioning sensor, and the target task object and environment. The virtual optical positioning sensor includes measurement noise, data latency, and marker ball occlusion. The parameters in the virtual environment are adjusted by domain randomization, and the SAC policy network model is pre-trained to obtain the pre-trained SAC policy network model.

8. A reinforcement learning control device for a robotic arm guided by an optical positioning system, characterized in that, include: The first determining module is used to determine the actual pose of the calibrated robotic arm and the target pose based on the image stream acquired by the calibrated binocular camera. The calculation module is used to calculate the position error and attitude error based on the actual pose and the target pose; The stitching module is used to stitch together the position error, the posture error, the actual pose, the target pose, the pose change, and the joint state of the robotic arm to obtain the current state vector. The second determining module is used to input the current state vector into a pre-trained SAC policy network model to obtain action instructions; The third determining module is used to determine the next state vector and multi-target reward based on the action instruction; The training module is used to iteratively train the SAC policy network model based on the current state vector, the action command, the next state vector, and the multi-objective reward to obtain a reinforcement learning control model for the robotic arm, so as to achieve control of the robotic arm.

9. A reinforcement learning control device for a robotic arm, characterized in that, include: processor; Memory, used to store computer programs; Wherein, when the processor executes the computer program, it implements the reinforcement learning control method for a robotic arm based on an optical positioning system as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the reinforcement learning control method for a robotic arm based on an optical positioning system as described in any one of claims 1 to 7.

Images