An adaptive teleoperation control method based on reinforcement learning

Abstract

The application discloses a kind of adaptive teleoperation control methods based on reinforcement learning.The motion state parameters and environmental contact force parameters of the end of manipulator are collected to calculate position error and contact force error, and state vector is constructed;According to the state vector, a teleoperation reinforcement learning model is constructed, the teleoperation reinforcement learning model is interactively trained with the teleoperation environment, and a trained teleoperation reinforcement learning model is obtained;The current state vector is constructed and input into the trained teleoperation reinforcement learning model, and the adjustment amount of impedance control parameter is output, and then the control parameters of the impedance controller are updated, and the joint control torque is calculated by combining the preset expected trajectory, motion state parameter and environmental contact force parameter, to drive the manipulator to perform work.The application enables the robot to automatically adjust its dynamic characteristics when facing dynamic uncertain environments such as wind disturbance and wire swing, to realize stable, accurate and compliant operation, and significantly improves the autonomy and safety of the live-line working robot.

Description

Technical Field

[0001] This invention relates to the field of robot control, and in particular to an adaptive teleoperation control method based on reinforcement learning. Background Technology

[0002] In complex dynamic environments (such as wind disturbances and conductor sway), traditional teleoperated robotic arms using fixed impedance parameters are prone to positioning errors and oscillations, making it difficult to guarantee mission accuracy and safety. Impedance control introduces virtual mass, damping, and stiffness parameters (M... d B d K d While methods can adjust the compliance characteristics of robotic arms using parameters, fixed parameters cannot adapt to environmental changes. In recent years, deep reinforcement learning (Deep RL) has been used in robot control to enhance system robustness in disturbed environments by optimizing strategies online through trial and error. However, research combining reinforcement learning with impedance control for remote operating systems to dynamically adjust impedance parameters to adapt to disturbances such as wind is limited. Existing technologies lack a control method capable of sensing environmental changes in real time and adaptively adjusting impedance parameters, making it difficult to maintain stability and efficiency in scenarios such as live-line work.

[0003] The limitations of traditional control methods can be clearly described through their mathematical models. In traditional impedance control models, the diagonal parameter matrices representing virtual mass, virtual damping, and virtual stiffness are fixed. This fixed-parameter model cannot adaptively adjust its dynamic characteristics under dynamic environments (such as wind disturbances and target swaying), leading to increased tracking errors or contact force instability. Summary of the Invention

[0004] To address the problems in the background technology, this invention proposes an adaptive teleoperation control method based on reinforcement learning, which reduces positioning errors and oscillations in dynamic environments and improves control accuracy and safety.

[0005] The technical solution adopted in this invention is: The method of the present invention includes the following steps: S1. Collect the motion state parameters and environmental contact force parameters of the robotic arm end effector, and calculate the position error and contact force error based on the motion state parameters and environmental contact force parameters to construct the state vector of the reinforcement learning model; S2. Construct a teleoperation reinforcement learning model based on the state vector, and train the teleoperation reinforcement learning model interactively with the teleoperation environment to obtain a trained teleoperation reinforcement learning model. S3. Collect the motion state parameters and environmental contact force parameters of the current robotic arm end effector, then construct the current state vector, and input the current state vector into the trained teleoperation reinforcement learning model to output the adjustment amount of the impedance control parameters. S4. Update the control parameters of the impedance controller according to the adjustment amount of the impedance control parameters, and calculate the joint control torque according to the updated impedance controller control parameters, the preset expected trajectory, motion state parameters and environmental contact force parameters, and then drive the robotic arm to perform remote operation based on the joint control torque.

[0006] The motion state parameters include the end position and end velocity of the robotic arm, and the environmental contact force parameters include the contact force between the end of the robotic arm and the environment.

[0007] The state vector includes position error and contact force error. The position error is specifically the difference between the actual position and the desired position of the end, and the contact force error is specifically the difference between the actual contact force and the desired contact force.

[0008] The teleoperation reinforcement learning model specifically employs a proximal policy optimization algorithm.

[0009] The reward function of the teleoperation reinforcement learning model is specifically set according to the following formula: in, This represents the total reward function value of the teleoperation reinforcement learning model. , , , These represent task tracking rewards, contact force control rewards, motion smoothness rewards, and time penalty rewards, respectively. , , , These represent the weight coefficients for task tracking reward, contact force control reward, motion smoothness reward, and time penalty reward, respectively. Indicates positional error. Indicates attitude error. The L2 norm of a vector. The weighting parameters represent the attitude error. This represents the contact force between the end effector of the robotic arm and the environment. Indicates the desired contact force. Indicates the joint control torque. Indicates the task completion time.

[0010] The impedance control parameters include virtual mass, virtual damping, and virtual stiffness.

[0011] The joint control torque is obtained according to the following formula: in, This represents the virtual mass in the impedance control parameters. This represents the virtual damping in the impedance control parameters. This represents the virtual stiffness in the impedance control parameters. , , These represent the actual acceleration, actual velocity, and actual position of the robotic arm's end effector, respectively. , , These represent the desired acceleration, desired velocity, and desired position at the end effector of the robotic arm, respectively. This represents the contact force between the end effector of the robotic arm and the environment. , , Let these represent the acceleration vector, velocity vector, and position vector of the robotic arm joints, respectively. The inertia matrix of the robotic arm is represented. Represents the matrix of Coriolis force and centrifugal force. Represents the gravity vector. The matrix representing the transpose of the Jacobian matrix at the end effector of the robotic arm. This indicates the joint control torque.

[0012] The beneficial effects of this invention are: Traditional impedance control parameters (virtual mass M) d Damping B d Stiffness K d Impedance control is typically fixed and difficult to adapt to dynamic environments (such as wind disturbance and conductor sway). Innovatively, reinforcement learning agents are introduced to dynamically adjust impedance parameters based on real-time perceived environmental conditions (position error, contact force error), achieving an integrated closed loop of "perception-decision-control". This is a fundamental improvement over traditional impedance control.

[0013] This invention utilizes a reinforcement learning agent to dynamically adjust the parameters (virtual mass, damping, and stiffness) of an impedance controller based on the real-time interaction between the robotic arm's end effector and the environment (such as positional error and contact force). A carefully designed reward function guides the agent's learning, and a combination of offline pre-training and online fine-tuning enhances the policy's robustness. This invention enables the robot to automatically adjust its dynamic characteristics when facing dynamic and uncertain environments such as wind disturbances and swaying wires, achieving stable, precise, and compliant operation, significantly improving the autonomy and safety of live-line working robots. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the system framework in this embodiment.

[0015] Figure 2 This is a flowchart of the adaptive parameter adjustment process in this embodiment.

[0016] Figure 3 This is a system flowchart of the control method in this embodiment.

[0017] Figure 4 This embodiment presents a flowchart of the reinforcement learning training process.

[0018] Figure 5 This embodiment shows the working scenario of the robotic arm in a dynamic environment. Detailed Implementation

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0021] like Figure 4 As shown, this embodiment uses a live-line working robot as an application scenario to illustrate the implementation process of this method, which specifically includes the following steps: S1. Collect the motion state parameters and environmental contact force parameters of the robotic arm end effector, and calculate the position error and contact force error based on the motion state parameters and environmental contact force parameters to construct the state vector of the reinforcement learning model; S2. Construct a teleoperation reinforcement learning model based on the state vector, and train the teleoperation reinforcement learning model interactively with the teleoperation environment to obtain a trained teleoperation reinforcement learning model. The state space is input into a pre-trained reinforcement learning agent, which then outputs adjustments to the impedance control parameters.

[0022] S3. Collect the motion state parameters and environmental contact force parameters of the current robotic arm end effector, then construct the current state vector, and input the current state vector into the trained teleoperation reinforcement learning model to output the adjustment amount of the impedance control parameters. S4. Update the control parameters of the impedance controller according to the adjustment amount of the impedance control parameters, and calculate the joint control torque according to the updated impedance controller control parameters, the preset expected trajectory, motion state parameters and environmental contact force parameters, and then drive the robotic arm to perform remote operation based on the joint control torque.

[0023] like Figure 2 and 3 As shown, in this embodiment, the teleoperation reinforcement learning model adopts an offline-online hybrid training mode, including pre-training in a digital twin simulation environment and online fine-tuning using real-time data in a real operating environment.

[0024] 1) Construct a dynamic model of the robotic arm and an environmental model in a digital twin simulation environment to simulate dynamic scenarios such as wind disturbance and wire swaying.

[0025] 2) Initialize the reinforcement learning agent (PPO algorithm) and reward function parameters.

[0026] 3) Perform offline pre-training and optimize the policy network through a large number of simulation episodes until the reward function converges.

[0027] 4) Deploy the pre-trained strategy to the real robot system and fine-tune it online using real-time sensor data to further optimize the strategy to adapt to the real environment.

[0028] 5) During the online fine-tuning process, the strategy performance should be evaluated regularly to ensure stable control effects.

[0029] In this embodiment, offline training uses the NVIDIA Isaac Gym simulation platform, and online fine-tuning uses real-time data streaming. Satisfactory performance can be achieved in about 24 hours of training.

[0030] The motion state parameters include the end position and end velocity of the robotic arm, and the environmental contact force parameters include the contact force between the end of the robotic arm and the environment.

[0031] The state vector includes at least position error and contact force error. Specifically, the position error is the difference between the actual position and the desired position of the end effector. The contact force error is specifically the difference between the actual contact force and the expected contact force, i.e. .

[0032] The teleoperation reinforcement learning model specifically employs the proximal policy optimization (PPO) algorithm.

[0033] The reward function of the teleoperation reinforcement learning model is specifically set according to the following formula: in, This represents the total reward function value of the teleoperation reinforcement learning model. , , , These represent task tracking rewards, contact force control rewards, motion smoothness rewards, and time penalty rewards, respectively. , , , These represent the weighting coefficients for task tracking rewards, contact force control rewards, motion smoothness rewards, and time penalty rewards, respectively. These are typically determined through experience or optimization. For example, they could be... , , , . Indicates positional error. Indicates attitude error. The L2 norm of a vector. The weighting parameters represent the attitude error. This represents the contact force between the end effector of the robotic arm and the environment. Indicates the desired contact force. Indicates the joint control torque. Indicates the task completion time.

[0034] The impedance control parameters include virtual mass. Virtual damping and virtual stiffness .

[0035] The traditional impedance control model is as follows: in, It is a fixed diagonal parameter matrix, representing virtual mass, virtual damping, and virtual stiffness, respectively. These are the actual position, velocity, and acceleration vector of the robotic arm's end effector. These are the desired position, velocity, and acceleration vectors of the robotic arm's end effector, respectively. It is the contact force vector between the robotic arm's end effector and the environment. Fixed-parameter models cannot adaptively adjust their dynamic characteristics in dynamic environments (such as wind disturbances or target swaying), leading to increased tracking errors or unstable contact forces.

[0036] In this specific embodiment, the impedance controller calculates the joint control torque based on the updated parameters, the desired trajectory, and the state and contact force information, using an impedance model and a robotic arm dynamics model. ; The joint control torque is obtained according to the following formula: Impedance model: Robotic arm dynamics model: in, This represents the virtual mass in the impedance control parameters. This represents the virtual damping in the impedance control parameters. This represents the virtual stiffness in the impedance control parameters. , , These represent the actual acceleration, actual velocity, and actual position of the robotic arm's end effector, respectively. , , These represent the desired acceleration, desired velocity, and desired position at the end effector of the robotic arm, respectively. This represents the contact force between the end effector of the robotic arm and the environment. , , Let these represent the acceleration vector, velocity vector, and position vector of the robotic arm joints, respectively. The inertia matrix of the robotic arm is represented. Represents the matrix of Coriolis force and centrifugal force. Represents the gravity vector. The matrix representing the transpose of the Jacobian matrix at the end effector of the robotic arm. This indicates the joint control torque.

[0037] In this embodiment, the motion state parameters and environmental contact force parameters of the robotic arm end are acquired in real time through a sensor system. The sensor system includes a position sensor for measuring the end position and velocity, and a force / torque sensor for measuring the end contact force. The sensor data is transmitted to the main controller through a real-time communication bus.

[0038] Specifically, the state perception unit collects motion state information and environmental contact force information of the robotic arm's end effector in real time through a sensor system. The data sources include: the position of the robotic arm's end effector measured by a position sensor (such as an encoder). and speed Force / torque sensors (such as six-dimensional force sensors) measure the contact force between the end effector and the environment. The sensor data is transmitted to the main controller via a real-time communication bus (such as EtherCAT or CAN bus), where the main controller performs data preprocessing and calculates the position error. and contact force error This leads to the construction of state vectors. .

[0039] like Figure 1 and Figure 5 As shown, the adaptive parameter adjustment process of the present invention includes: state perception, state calculation, intelligent decision-making, controller update, action execution, and reward feedback. Specifically, the motion state and contact force information of the robotic arm's end effector are collected in real time through a sensor system to construct a state vector. The pre-trained reinforcement learning agent is input, and the output impedance parameter adjustment action is performed. After updating the impedance controller, the joint control torque is calculated to drive the robotic arm to perform the operation.

[0040] System initialization: When starting a teleoperation task, the system initializes the robotic arm's position and impedance parameters. .

[0041] In this embodiment, system initialization: When the teleoperation task begins, the system initializes the robotic arm position and impedance parameters. =10kg, =100 N·s / m, =500N / m.

[0042] State perception and data transmission: The state perception unit collects data in real time through a sensor system installed at the end effector of the robotic arm. Data sources include: position sensors (such as high-precision encoders) used to measure the angles of each joint of the robotic arm and the position of the end effector in Cartesian space calculated through forward kinematics. and speed Force / torque sensors (such as six-dimensional force sensors) are mounted on the wrist of a robotic arm to measure the contact force between the end effector and the environment. .

[0043] The sensor data is transmitted to the main controller via a real-time communication bus (such as EtherCAT or CAN bus). The main controller performs data preprocessing and calculates the position error. and contact force error and construct the state vector .

[0044] State computation: Constructing the state space for reinforcement learning based on sensor data The state vector Defined as: in , This state space comprehensively reflects the position tracking deviation and its changing trend caused by disturbances such as wind and conductor swaying, as well as abnormal fluctuations in contact force.

[0045] Intelligent decision-making: Reinforcement learning agents (using algorithms such as PPO) make decisions based on the current state. Output Action That is, for impedance parameters Adjustment amount Policy Network As a complex mathematical function, learning how to apply the combined effects of perturbations (state) Output the optimal parameter adjustment instructions.

[0046] Controller update and torque calculation: using actions Update impedance controller parameters: The impedance controller is then combined with the updated Expected trajectory Calculate the desired end effector force based on the current state. Then, combine this with the robotic arm's dynamics model: The joint control torque can be obtained by inverse dynamics or by calculating the torque. .

[0047] Action execution: The calculated joint control torque The data is sent to the robotic arm joint driver, which then drives the robotic arm to perform the task.

[0048] Reward Feedback and Strategy Updates: The system calculates real-time rewards based on performance. Design the reward function. Taking into account factors such as task success rate, contact force stability, and smoothness. The data is stored in the experience replay pool and used to update the RL policy network parameters online, enabling the control policy to continuously adapt to environmental changes.

[0049] The above process forms a closed-loop feedback, enabling the system to continuously respond to environmental disturbances and achieve online adaptive adjustment of impedance parameters.

[0050] In this embodiment, through the above process, the robot can still maintain a position error of less than 2mm and a contact force fluctuation of less than 5N even in windy environments, which significantly improves the stability and accuracy of operation.

[0051] In summary, the method of this invention can effectively cope with dynamic and uncertain environments and realize adaptive remote operation control of robotic arms, which is of great significance for improving the autonomy and safety of live-line working robots.

[0052] The above detailed embodiments illustrate the technical solution and beneficial effects of the present invention. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An adaptive teleoperation control method based on reinforcement learning, characterized in that, The method includes the following steps: S1. Collect the motion state parameters and environmental contact force parameters of the robotic arm end effector, and calculate the position error and contact force error based on the motion state parameters and environmental contact force parameters to construct the state vector of the reinforcement learning model; S2. Construct a teleoperation reinforcement learning model based on the state vector, and train the teleoperation reinforcement learning model interactively with the teleoperation environment to obtain a trained teleoperation reinforcement learning model. S3. Collect the motion state parameters and environmental contact force parameters of the current robotic arm end effector, then construct the current state vector, and input the current state vector into the trained teleoperation reinforcement learning model to output the adjustment amount of the impedance control parameters. S4. Update the control parameters of the impedance controller according to the adjustment amount of the impedance control parameters, and calculate the joint control torque according to the updated impedance controller control parameters, the preset expected trajectory, motion state parameters and environmental contact force parameters, and then drive the robotic arm to perform remote operation based on the joint control torque.

2. The adaptive teleoperation control method based on reinforcement learning according to claim 1, characterized in that: The motion state parameters include the end position and end velocity of the robotic arm, and the environmental contact force parameters include the contact force between the end of the robotic arm and the environment.

3. The adaptive teleoperation control method based on reinforcement learning according to claim 1, characterized in that: The state vector includes position error and contact force error. The position error is specifically the difference between the actual position and the desired position of the end, and the contact force error is specifically the difference between the actual contact force and the desired contact force.

4. The adaptive teleoperation control method based on reinforcement learning according to claim 1, characterized in that: The teleoperation reinforcement learning model specifically employs a proximal policy optimization algorithm.

5. The adaptive teleoperation control method based on reinforcement learning according to claim 1, characterized in that: The reward function of the teleoperation reinforcement learning model is specifically set according to the following formula: in, This represents the total reward function value of the teleoperation reinforcement learning model. , , , These represent task tracking rewards, contact force control rewards, motion smoothness rewards, and time penalty rewards, respectively. , , , These represent the weight coefficients for task tracking reward, contact force control reward, motion smoothness reward, and time penalty reward, respectively. Indicates positional error. Indicates attitude error. The L2 norm of a vector. The weighting parameters represent the attitude error. This represents the contact force between the end effector of the robotic arm and the environment. Indicates the desired contact force. Indicates the joint control torque. Indicates the task completion time.

6. The adaptive teleoperation control method based on reinforcement learning according to claim 1, characterized in that: The impedance control parameters include virtual mass, virtual damping, and virtual stiffness.

7. The adaptive teleoperation control method based on reinforcement learning according to claim 1, characterized in that: The joint control torque is obtained according to the following formula: in, This represents the virtual mass in the impedance control parameters. This represents the virtual damping in the impedance control parameters. This represents the virtual stiffness in the impedance control parameters. , , These represent the actual acceleration, actual velocity, and actual position of the robotic arm's end effector, respectively. , , These represent the desired acceleration, desired velocity, and desired position at the end effector of the robotic arm, respectively. This represents the contact force between the end effector of the robotic arm and the environment. , , Let these represent the acceleration vector, velocity vector, and position vector of the robotic arm joints, respectively. The inertia matrix of the robotic arm is represented. Represents the matrix of Coriolis force and centrifugal force. Represents the gravity vector. The matrix representing the transpose of the Jacobian matrix at the end effector of the robotic arm. This indicates the joint control torque.

Images