A time-delay variable-gain robot virtual-real interaction remote control system and method

Abstract

The application relates to a time-delay variable-gain robot virtual-real interaction remote control system and method, which comprises the following steps: collecting wrist, finger and foot pedal action data; mapping the collected action data to obtain movement instructions of a slave robot; selecting and recording movement instructions in a certain time period according to different pressing states of the foot pedal action to obtain corresponding movement instruction sequences; receiving the movement instruction sequences, mapping and proportionally adjusting the movement instructions according to the distance between a virtual simulation model of the slave robot and a virtual simulation environment space barrier and the time delay generated by processing and feedback of the movement instructions sent from a master controller to a slave controller to obtain operation instructions; and driving the slave robot to perform corresponding actions according to the operation instructions and feeding back the action state. The application solves the problems of unstable time-delay jitter, unstable operation caused by limited perception and easy collision in the current robot remote operation technology.

Description

Technical Field

[0001] This invention belongs to the field of robot teleoperation technology, and relates to a time-delay variable gain robot virtual-real interaction teleoperation system and method. Background Technology

[0002] Tele-operation refers to the process by which an operator remotely controls a robot to complete a task using a handheld device or other human-machine interface. During task execution, the operator sends commands from the master robot to the slave robot and simultaneously receives feedback from both the robot and external sources, thus completing the tele-operation task. However, during tele-operation, issues such as the robot's high dimensionality and unstable communication latency can lead to low operational efficiency and poor safety.

[0003] Among existing telecontrol algorithms, the patent "Force-Guided Teleoperation System and Control Method Based on Dual-Arm Collaborative Potential Field" (CN202410208048.5) generates dynamic virtual constraint forces based on the dual-arm collaborative potential field. Two force feedback hand controllers assist the operator in manipulating the robotic arms to complete dual-arm collaborative tasks. This method mainly utilizes environmental information to generate virtual contact forces, which is not flexible enough in handling dynamic environments, and the virtual forces may negatively impact operational accuracy. The patent "A Data-Driven and Mechanism-Based Hybrid Modeling and Control Method for Virtual-Real Fusion Control" (CN202211663067.4) discloses a virtual-real fusion telecontrol method that generates predictive state vectors through a predictive model to form stable predictive control commands. However, this method has limited effectiveness in handling unstable situations. Summary of the Invention

[0004] The technical problem solved by this invention is to overcome the shortcomings of the prior art and propose a time-delay variable gain robot virtual-real interaction telecontrol system and method to solve the problems of unstable time delay jitter, operation instability caused by limited perception, and easy collision in the current robot teleoperation technology.

[0005] The solution of this invention is: a time-delay variable gain robot virtual-real interaction remote control system, the master end including a master end controller, a human motion acquisition module, a motion virtual-real mapping module and an operation safety assessment module; the slave end including a slave end controller and a slave end robot;

[0006] The master controller is a data interaction and processing platform for the human motion acquisition module, the motion virtual-real mapping module, and the operation safety assessment module, and displays the virtual simulation model and virtual simulation environment of the slave robot.

[0007] The human motion acquisition module includes a wrist position acquisition module, a finger position acquisition module, and a foot position acquisition module. All three position acquisition modules are connected to and communicate with the main controller via a data bus.

[0008] The wrist position acquisition module acquires the real-time wrist pose in Cartesian space.

[0009] The finger position acquisition module uses a remote control stick to acquire the position of the human finger. Pushing the remote control stick obtains the control increment of the remote control stick as the first raw data and sends it to the motion virtual-real mapping module.

[0010] The foot position acquisition module uses two pedals to capture the pressing action of the left and right feet. When either pedal is pressed, the module acquires the wrist pose from the moment of pressing to the current moment, and sends the wrist pose increment as the second raw data to the motion virtual-real mapping module. When both pedals are released, the motion virtual-real mapping module sets the wrist pose increment to 0. The data of the left and right feet pressing the pedals is sent as the third raw data to the motion virtual-real mapping module.

[0011] The motion virtual-real mapping module receives the first raw data, the second raw data, and the third raw data, maps them to obtain the motion commands of the slave robot, and selects to record the motion commands for a certain time period based on the pressing state of the two pedals of the foot position acquisition module, and sends the corresponding motion command sequence to the operation safety assessment module.

[0012] The operation safety assessment module receives the motion command sequence sent by the motion virtual-real mapping module. Based on the distance between the virtual simulation model of the slave robot and the obstacles in the virtual simulation environment, as well as the time delay caused by the motion command being sent from the master controller to the slave controller for processing and feedback, the module adjusts the mapping ratio of the motion command and obtains the operation command, which is then sent to the slave controller.

[0013] The slave controller receives operation instructions sent by the operation safety assessment module and drives the slave robot to perform corresponding actions according to the operation instructions.

[0014] Furthermore, the acquisition of real-time wrist pose in Cartesian space specifically involves:

[0015] The wrist position acquisition module uses two master robotic arms to acquire the pose of the left and right wrists of the human body. When in use, the human wrist is fixed to the end of the master robotic arm, and the end pose of the master robotic arm is used as the human wrist pose to obtain the real-time wrist pose in Cartesian space.

[0016] Furthermore, the mapping yields motion commands for the slave robot, as follows:

[0017] (1) Establish the master arm base coordinate system of the master robot and the slave arm base coordinate system of the slave robot, and align the two base coordinate systems: Let the master arm base coordinate system be... Let the coordinate system of the arm base be The alignment process yields the transformation matrix. By transforming the matrix Establish operational intuition between the master arm and the slave arm; for example, when the master arm is pushed forward, the slave arm moves forward synchronously.

[0018] (2) Establish the coordinate system of the master robotic arm's remote control joystick and the coordinate system of the slave robot's mobile platform, and align the remote control joystick coordinate system with the mobile platform coordinate system: Let the forward movement of the remote control joystick coordinate system be the positive x-direction, the leftward movement be the positive y-direction, the forward movement of the mobile platform coordinate system be the positive x-direction, the left side of the robot body be the positive y-direction, and the vertical upward movement of the robot body be the positive z-direction, so that v x =k x *Δx,ω z =k y *Δy; where v x For the speed command in the x-direction of the mobile platform, k x ω is the velocity mapping coefficient of the remote control stick in the x-direction, and Δx is the amount of push of the remote control stick in the x-direction; z For the angular velocity command in the z-direction of the mobile platform, k y Let Δy be the velocity mapping coefficient of the remote control stick in the y-direction, and Δy be the amount of push of the remote control stick in the y-direction.

[0019] (3) Map the collected first raw data, second raw data, and third raw data into motion commands for the slave robot: The motion command for the slave robot arm is the pose transformation relationship from the end of the arm to the base of the arm at time t, represented by quaternions and position vectors; wherein the pose transformation relationship from the end of the arm to the base of the arm at time t is: Let represent the pose transformation relationship from the end effector of the main boom to the main boom base at time t. The pose transformation relationship from the end of the main boom to the main boom base at the moment the pedal is pressed (T0). The pose transition from the end of the arm to the base of the arm is defined at the moment the pedal is depressed (T0); the motion command for the slave moving platform is the velocity command v in the x-direction. x and the angular velocity command ω in the z direction z The motion commands of the slave robot are a set of motion commands of the slave robotic arm and motion commands of the slave mobile platform.

[0020] Furthermore, the step of selecting and recording a motion command for a certain time period based on the pressing state of the two pedals of the foot position acquisition module to obtain the corresponding motion command sequence includes:

[0021] One of the two pedals is defined as the simulation pedal, and the other is defined as the control pedal. The use of the first and second raw data is switched according to the third raw data pedal information. When the third raw data comes from the simulation pedal, the corresponding first and second raw data are used to drive the virtual simulation model of the slave robot. When the third raw data comes from the control pedal, the corresponding first and second raw data are used to drive the real slave robot. All simulations start from the current state of the slave robot.

[0022] 1) When neither the simulation pedal nor the control pedal is pressed, the virtual simulation model of the slave robot displays the current status of the slave robot in real time in the virtual simulation environment;

[0023] 2) When the simulation pedal is pressed down and the control pedal is released, motion commands are recorded and the virtual simulation model of the slave robot is driven to move in the virtual simulation environment;

[0024] 3) When the simulation pedal is pressed and the control pedal is pressed at the same time, the motion command recording ends. The motion commands recorded during the period from "pressing down the simulation pedal and lifting the control pedal" to "pressing down the simulation pedal and pressing down the control pedal at the same time" are used as the first motion command sequence and sent as a whole to the operation safety assessment module. When both the simulation pedal and the control pedal are lifted, the first motion command sequence stops being sent.

[0025] 4) When the simulation pedal is raised and the control pedal is pressed, the motion virtual-real mapping module sends the motion command at each moment to the operation safety assessment module in real time until both the simulation pedal and the control pedal are raised. The motion command recorded during the period from "raising the simulation pedal and pressing the control pedal" to "both the simulation pedal and the control pedal are raised" is used as the second motion command sequence.

[0026] Furthermore, based on the distance between the virtual simulation model of the slave robot and the spatial obstacles in the virtual simulation environment, the mapping ratio of the motion commands is adjusted as follows:

[0027]

[0028] d = min(sdf(obj, robot))

[0029] Where sdf is the distance sign function, used to calculate the distance between models in the virtual scene, obj is the environmental obstacle point vector with dimension 3×n, robot is the robot model, and after obtaining the distances between n obstacles and the robot, the minimum value is found to obtain d, K. min For the minimum gain, d min K1 is the safe distance; K1 is the distance safety gain, when K1 < 0.3K min When d ≤ d, it indicates a high-risk collision state. minWhen the collision risk is indicated, it indicates a medium-risk collision state; otherwise, it indicates a low-risk collision state.

[0030] Furthermore, based on the time delay generated when motion commands are sent from the master controller to the slave controller for processing and feedback, the mapping ratio of the motion commands is adjusted as follows:

[0031] Each motion command to be sent is marked with a local timestamp T1 and a frame number d1 by the master controller and sent to the slave controller at a fixed sending period. After receiving the motion command of this frame, the slave controller performs slave robot control, reads the slave robot status, and forms the latest status feedback frame of the slave robot to the master controller. The latest status feedback frame includes timestamp T1, frame number d1, and timestamp T2 after T1 is superimposed with the time Δt spent on running and processing.

[0032] When the master controller receives the latest status feedback frame, it records the arrival time of the feedback frame as timestamp T3. It then calculates the timestamp for a complete motion command cycle, using T3-T1 to obtain the two network transmission delays, and using T2-T1 to obtain the slave processing delay. Finally, it calculates the delay security gain K2 using the following formula:

[0033]

[0034] Among them, K M For maximum gain, K m For minimum gain, T max To predict the maximum delay, if T max If no return frame is received within the specified time, T will be automatically used. max As a time estimate;

[0035] When K2 is large, the slave-side following ratio increases, and the overall operating speed increases. When K2 is small, the slave-side following ratio decreases, and the overall operating speed decreases.

[0036] Furthermore, the operation safety assessment module uses gain K = K1K2 to adjust the mapping ratio of the motion command, and sends the operation command to the slave controller.

[0037] Gain K is used to calculate the pose increment of the master end effector, thereby changing the motion command of the slave end effector.

[0038]

[0039] in, Let be the original pose transformation relationship from the end effector of the main boom to the main boom base at time t. This represents the original pose transformation relationship from the end of the main arm to the main arm base at time t-Δt in the previous frame. The calculation pose transformation relationship from the end of the main arm to the main arm base is calculated at time t-Δt in the previous frame.

[0040] And the commands used to calculate the joystick control increments to change the motion commands of the slave mobile platform:

[0041] v x =Kk x Δx,ω z =Kk y Δy.

[0042] Furthermore, environmental spatial structure information collected by the environmental sensor of the slave robot is added to the virtual simulation environment and represented in the form of point cloud. In order to increase the control accuracy of the slave robot in the obstacle environment, the point cloud data generated by the environmental sensor scanning the real slave robot is removed in the virtual simulation environment, and only the environmental point cloud is retained.

[0043] Furthermore, the virtual simulation model of the slave robot uses a robot joint linkage model driven by the joint position data of the slave robot.

[0044] A time-delay variable gain method for remote control of a robot through virtual-real interaction includes the following steps:

[0045] Collect data on wrist, finger, and foot movements;

[0046] The collected motion data is mapped to obtain motion commands for the slave robot. Based on the different pressing states of the foot pedal action, the motion commands for a certain time period are selected and recorded to obtain the corresponding motion command sequence.

[0047] The system receives a sequence of motion commands and adjusts the mapping ratio of the motion commands based on the distance between the virtual simulation model of the slave robot and the obstacles in the virtual simulation environment, as well as the time delay caused by the motion commands being sent from the master controller to the slave controller for processing and feedback, to obtain the operation commands.

[0048] Drive the slave robot to perform corresponding actions according to the operation instructions and provide feedback on the action status.

[0049] The advantages of this invention compared to the prior art are:

[0050] (1) This invention proposes a time-delay variable gain robot virtual-real interaction telecontrol method, which introduces a real-time safety assessment method in the virtual-real interaction telecontrol, and flexibly switches between virtual asynchronous control and real-time synchronous control mode, thereby reducing the impact of time delay uncertainty and collision risk during the telecontrol process.

[0051] (2) The operation safety assessment module of the present invention assesses collision safety through the distance sign function, assesses real-time latency through the frame mark field, and calculates the safety evaluation value of the operation command in real time for incremental command adjustment, thereby improving the safety of remote control.

[0052] (3) In the remote control system proposed in this invention, the human motion acquisition module collects the movements of the human wrist, fingers, and feet as data input, thereby capturing the complete human motion, which can be used to generate different types of control commands. The acquisition module's acquisition method is adapted to human usage habits, which can reduce the learning cost.

[0053] (4) The virtual-real interaction method of the present invention can quickly switch through the input device to realize the integration of virtual-real synchronous and asynchronous control modes, which can improve the operator's environmental perception and thus quickly and flexibly handle a variety of remote operation tasks.

[0054] (5) The motion virtual-real mapping module of the present invention realizes command mapping through coordinate system alignment, which can intuitively reflect human body movements. The operation safety assessment module adopts an incremental mode to expand the operator's operating range and meet the needs of large-scale operation and fine operation. This mode is universally applicable to various types of robots. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the human motion acquisition module of the remote control system according to an embodiment of the present invention;

[0056] Figure 2 This is a schematic diagram of the data flow between the master and slave ends of the remote control system according to an embodiment of the present invention;

[0057] Figure 3 This is a flowchart illustrating the gain calculation process of the remote control system operation safety assessment module according to an embodiment of the present invention.

[0058] Figure 4 This is a flowchart of a time-delay variable gain robot virtual-real interaction telecontrol method according to an embodiment of the present invention. Detailed Implementation

[0059] This invention proposes a time-delay-variable-gain robot virtual-real interactive telecontrol system and method. By estimating the transmit / receive loopback delay and calculating the distance to safe obstacles, the control gain is dynamically adjusted to reduce the impact of unstable delays and collision risks during operation, thereby improving the stability and safety of control. The flexible switching between asynchronous control in the virtual environment and real-time synchronous operation enhances the comprehensiveness of perception in virtual-real combined telecontrol.

[0060] The present invention will be further described below with reference to the embodiments.

[0061] Example 1

[0062] This invention proposes a time-delay variable gain robot virtual-real interaction remote control system. The master end realizes human motion acquisition and converts it into operation commands, which are sent to the slave end controller. The slave end controller drives the slave robot to perform corresponding actions according to the received operation commands.

[0063] Specifically, the master end includes a master end controller, a human motion acquisition module, a motion virtual-real mapping module, and an operation safety assessment module; the slave end includes a slave end controller and a slave end robot.

[0064] The master controller is a data interaction and processing platform for the human motion acquisition module, the motion virtual-real mapping module, and the operation safety assessment module, and displays the virtual simulation model and virtual simulation environment of the slave robot.

[0065] The human motion acquisition module includes a wrist position acquisition module, a finger position acquisition module, and a foot position acquisition module. All three position acquisition modules are connected and communicate with the main controller via a data bus.

[0066] like Figure 1 As shown, the wrist position acquisition module uses two master robotic arms 203 to acquire the poses of the left and right wrists of the human body 204. In use, the human wrist is fixedly connected to the end of the master robotic arm 203, so the end pose of the master robotic arm 203 is used as the human wrist pose, thus obtaining the real-time wrist pose in Cartesian space.

[0067] The finger position acquisition module uses a remote control joystick 202 to acquire the position of the fingers 204 on the human body. Pushing the remote control joystick 202 obtains the control increment of the joystick, and the control increment of the joystick is sent as the first raw data to the motion virtual-real mapping module.

[0068] The foot position acquisition module uses two pedals 205 to capture the pressing actions of the left and right feet of the human body 204. When either pedal 205 is pressed down, the wrist pose from the moment of pressing down to the current moment is acquired, and the wrist pose increment is sent as the second raw data to the motion virtual-real mapping module; when both pedals 205 are released, the motion virtual-real mapping module sets the wrist pose increment to 0.

[0069] Data on the left and right feet pressing the pedals 205 are sent as the third raw data to the motion virtual-real mapping module.

[0070] The motion virtual-real mapping module receives the first raw data, the second raw data, and the third raw data, maps them to obtain the motion commands of the slave robot, and selects to record the motion commands for a certain time period based on the pressing state of the two pedals 205 of the foot position acquisition module, and sends the corresponding motion command sequence to the operation safety assessment module.

[0071] In this embodiment, the specific implementation method of the motion virtual-real mapping module to obtain the motion commands of the slave robot based on the first original data, the second original data, and the third original data is as follows:

[0072] (1) Establish the master arm base coordinate system of the master robot and the slave arm base coordinate system of the slave robot, and align the two base coordinate systems: Let the master arm base coordinate system be... Let the coordinate system of the arm base be The alignment process yields the transformation matrix. By transforming the matrix Establish operational intuition between the master arm and the slave arm; for example, when the master arm is pushed forward, the slave arm moves forward synchronously.

[0073] (2) Establish the coordinate system of the master robotic arm's remote control joystick and the coordinate system of the slave robot's mobile platform, and align the remote control joystick coordinate system with the mobile platform coordinate system: Let the forward movement of the remote control joystick coordinate system be the positive x-direction, the leftward movement be the positive y-direction, the forward movement of the mobile platform coordinate system be the positive x-direction, the left side of the robot body be the positive y-direction, and the vertical upward movement of the robot body be the positive z-direction, so that v x =k x *Δx,ω z =k y *Δy; where v x For the speed command in the x-direction of the mobile platform, k x ω is the velocity mapping coefficient of the remote control stick in the x-direction, and Δx is the amount of push of the remote control stick in the x-direction; z For the angular velocity command in the z-direction of the mobile platform, k y Let Δy be the velocity mapping coefficient of the remote control stick in the y-direction, and Δy be the amount of push of the remote control stick in the y-direction.

[0074] (3) Map the collected first raw data, second raw data, and third raw data into motion commands for the slave robot: The motion command for the slave robot arm is the pose transformation relationship from the end of the arm to the base of the arm at time t, represented by quaternions and position vectors; wherein the pose transformation relationship from the end of the arm to the base of the arm at time t is: Let represent the pose transformation relationship from the end effector of the main boom to the main boom base at time t. The pose transformation relationship from the end of the main boom to the main boom base at the moment the pedal is pressed (T0). The pose transition from the end of the arm to the base of the arm is defined at the moment the pedal is depressed (T0); the motion command for the slave moving platform is the velocity command v in the x-direction. x and the angular velocity command ω in the z direction z The motion commands of the slave robot are a set of motion commands of the slave robotic arm and motion commands of the slave mobile platform.

[0075] In this embodiment, the motion virtual-real mapping module selects and records motion commands for a certain time period based on the pressing state of the two pedals 205 of the foot position acquisition module to obtain the corresponding motion command sequence. The preferred method is as follows:

[0076] like Figure 4 As shown, one of the two pedals 205 is defined as the simulation pedal and the other is defined as the control pedal. The use of the first and second raw data is switched according to the third raw data pedal information. When the third raw data comes from the simulation pedal, the corresponding first and second raw data are used to drive the virtual simulation model of the slave robot. When the third raw data comes from the control pedal, the corresponding first and second raw data are used to drive the real slave robot. All simulations start from the current state of the slave robot.

[0077] 1) When neither the simulation pedal nor the control pedal is pressed, the virtual simulation model of the slave robot displays the current status of the slave robot in real time in the virtual simulation environment;

[0078] 2) When the simulation pedal is pressed down and the control pedal is released, motion commands are recorded and the virtual simulation model of the slave robot is driven to move in the virtual simulation environment;

[0079] 3) When the simulation pedal is pressed and the control pedal is pressed at the same time, the motion command recording ends. The motion commands recorded during the period from "pressing down the simulation pedal and lifting the control pedal" to "pressing down the simulation pedal and pressing down the control pedal at the same time" are used as the first motion command sequence and sent as a whole to the operation safety assessment module. When both the simulation pedal and the control pedal are lifted, the first motion command sequence stops being sent.

[0080] 4) When the simulation pedal is raised and the control pedal is pressed, the motion virtual-real mapping module sends the motion command at each moment to the operation safety assessment module in real time until both the simulation pedal and the control pedal are raised. The motion command recorded during the period from "raising the simulation pedal and pressing the control pedal" to "both the simulation pedal and the control pedal are raised" is used as the second motion command sequence.

[0081] In this embodiment, the sensory display of the slave robot is shown around the user interface of the master controller, while the virtual simulation environment is located at the center of the master controller's user interface. Environmental spatial structure information collected by the slave robot's environmental sensors (such as LiDAR or RGBD cameras) is added to the virtual simulation environment and represented as a point cloud. To increase the control accuracy of the slave robot in obstacle environments, the point cloud data generated by the environmental sensors scanning the actual slave robot is removed from the virtual simulation environment, retaining only the environmental point cloud. The removal process uses a capsule-shaped bounding box to select the point cloud of the main body portion. The virtual simulation model of the slave robot uses a robot joint linkage model driven by the slave robot's joint position data.

[0082] The operation safety assessment module receives the motion command sequence sent by the motion virtual-real mapping module. Based on the distance between the virtual simulation model of the slave robot and obstacles in the virtual simulation environment, and the time delay caused by the motion commands being sent from the master controller to the slave controller for processing and feedback, the module adjusts the mapping ratio of the motion commands to obtain the operation commands, which are then sent to the slave controller. For example... Figure 4 As shown in S103.

[0083] Specifically, the operational safety assessment module includes a distance safety assessment module and a time delay safety assessment module. The process for calculating the assessment gain in the operational safety assessment module is as follows: Figure 3 As shown.

[0084] The distance safety assessment module is used to calculate the distance between the virtual simulation model of the slave robot and spatial obstacles in the virtual simulation environment after the motion command sequence is generated, and to obtain the distance safety gain K1. The calculation method is as follows:

[0085]

[0086] d = min(sdf(obj, robot))

[0087] Where sdf is the distance sign function, used to calculate the distance between models in the virtual scene, obj is the environmental obstacle point vector with dimension 3×n, robot is the robot model, and after obtaining the distances between n obstacles and the robot, the minimum value is found to obtain d, K. min For the minimum gain, d min For a safe distance.

[0088] After the distance safety assessment module obtains the distance safety gain sequence calculated from environmental collision and its own model collision, it is synchronously sent to the operation safety assessment module. The distance safety gain is also used for user interface display; when K1 < 0.3K... min When d ≤ d, it indicates a high-risk collision state, and the interface environment color is red. min When the collision risk is indicated by yellow, the interface environment color is yellow; otherwise, when the collision risk is indicated by green, the interface environment color is green.

[0089] The latency safety assessment module is used to calculate the latency safety gain K2 based on the latency generated when motion commands are sent from the master controller to the slave controller for processing and feedback. The calculation method is as follows:

[0090] Each motion command to be sent is tagged with a local timestamp T1 and frame number d1 by the master controller and sent to the slave controller at a fixed transmission cycle. Upon receiving the motion command frame, the slave controller controls the slave robot, reads its status, and generates a latest status feedback frame for the slave robot, which is then sent to the master controller. This latest status feedback frame includes timestamp T1, frame number d1, and a timestamp T2 after adding the processing time Δt to T1. When the master controller receives this latest status feedback frame, it records the arrival time as timestamp T3. The latency safety assessment module calculates the timestamps for a complete motion command cycle, using T3-T1 to obtain the two network transmission delays, and T2-T1 to obtain the slave processing delay. The latency safety gain K2 is then calculated using the following formula:

[0091]

[0092] Where K M For maximum gain, K m For minimum gain, T max To predict the maximum delay, if T max If no return frame is received within the specified time, T will be automatically used. max As a time estimate.

[0093] The delay safety gain K2 is used to adjust the ratio of the master's incremental speed to the slave's incremental command. When the gain is large, the slave can reproduce the master's actions more quickly. When the gain is small, the slave's following ratio decreases, and the overall running speed decreases.

[0094] The operation safety assessment module comprehensively considers obstacle avoidance safety and time delay stability. It uses a gain K = K1K2 to adjust the mapping ratio of motion commands, resulting in operation commands that are then sent to the slave controller. The gain K is used to calculate the pose increment of the master robotic arm's end effector, thereby altering the slave robotic arm's motion commands.

[0095]

[0096] in, Let be the original pose transformation relationship from the end effector of the main boom to the main boom base at time t. This represents the original pose transformation relationship from the end of the main arm to the main arm base at time t-Δt in the previous frame. The calculation pose transformation relationship from the end of the main arm to the main arm base is calculated at time t-Δt in the previous frame.

[0097] And the commands used to calculate the joystick control increments to change the motion commands of the slave mobile platform:

[0098] v x =Kk x Δx,ω z =Kky Δy.

[0099] The slave controller receives operation instructions sent by the operation safety assessment module and drives the slave robot to perform corresponding actions according to the operation instructions.

[0100] The data flow of the entire remote control system from master to slave is as follows: Figure 2 As shown.

[0101] Based on the remote control system of this invention, a method for remotely controlling a robot with time-delay variable gain and virtual-real interaction is provided, such as... Figure 4 As shown, it includes the following steps:

[0102] S1. Collect data on wrist, finger, and foot movements;

[0103] S2. Map the collected motion data to obtain motion commands for the slave robot. Based on different pedaling states, select and record motion commands for a specific time period to obtain the corresponding motion command sequence. Different pedaling states include the following four:

[0104] 1) When neither the simulation pedal nor the control pedal is pressed, the virtual simulation model of the slave robot displays the current status of the slave robot in real time in the virtual simulation environment;

[0105] 2) When the simulation pedal is pressed down and the control pedal is released, motion commands are recorded and the virtual simulation model of the slave robot is driven to move in the virtual simulation environment;

[0106] 3) When the simulation pedal is pressed and the control pedal is pressed at the same time, the motion command recording ends. The motion commands recorded during the period from "pressing down the simulation pedal and lifting the control pedal" to "pressing down the simulation pedal and pressing down the control pedal at the same time" are used as the first motion command sequence and sent as a whole to the operation safety assessment module. When both the simulation pedal and the control pedal are lifted, the first motion command sequence stops being sent.

[0107] 4) When the simulation pedal is raised and the control pedal is pressed, the motion virtual-real mapping module sends the motion command at each moment to the operation safety assessment module in real time until both the simulation pedal and the control pedal are raised. The motion command recorded during the period from "raising the simulation pedal and pressing the control pedal" to "both the simulation pedal and the control pedal are raised" is used as the second motion command sequence.

[0108] S3. Receive the motion command sequence, and adjust the motion command mapping ratio according to the distance between the virtual simulation model of the slave robot and the spatial obstacles in the virtual simulation environment, as well as the time delay caused by the motion command being sent from the master controller to the slave controller for processing and feedback, to obtain the operation command.

[0109] S4. Drive the slave robot to perform the corresponding actions according to the operation instructions, and provide feedback on the action status.

[0110] In summary, this invention proposes a teleoperation system and method that combines virtual simulation and real-time safety assessment. It introduces a real-time safety assessment mechanism into the teleoperation of virtual and real interaction, and flexibly switches between virtual asynchronous control and real-time synchronous control modes, thereby reducing the impact of time delay uncertainty and collision risk during teleoperation.

[0111] This invention innovatively proposes a quantitative evaluation method for operational safety, which introduces a distance sign function to assess collision safety, evaluates real-time latency through a frame tag field, and calculates the safety evaluation value of the operation command in real time for incremental mapping adjustment, thereby improving the safety of remote control.

[0112] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications to the technical solutions of the present invention by utilizing the methods and techniques disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall fall within the protection scope of the technical solutions of the present invention.

[0113] The contents not described in detail in this specification are common knowledge to those skilled in the art.

Claims

1. A time-delay variable gain robot virtual-real interaction telecontrol system, characterized in that, The master end includes a master controller, a human motion acquisition module, a motion virtual-real mapping module, and an operation safety assessment module; the slave end includes a slave controller and a slave robot. The master controller is a data interaction and processing platform for the human motion acquisition module, the motion virtual-real mapping module, and the operation safety assessment module, and displays the virtual simulation model and virtual simulation environment of the slave robot. The human motion acquisition module includes a wrist position acquisition module, a finger position acquisition module, and a foot position acquisition module. All three position acquisition modules are connected to and communicate with the main controller via a data bus. The wrist position acquisition module acquires the real-time wrist pose in Cartesian space. The finger position acquisition module uses a remote control stick to acquire the position of the human finger. Pushing the remote control stick obtains the control increment of the remote control stick as the first raw data and sends it to the motion virtual-real mapping module. The foot position acquisition module uses two pedals to capture the pressing action of the left and right feet. When either pedal is pressed, the module acquires the wrist pose from the moment of pressing to the current moment, and sends the wrist pose increment as the second raw data to the motion virtual-real mapping module. When both pedals are released, the motion virtual-real mapping module sets the wrist pose increment to 0. The data of the left and right feet pressing the pedals is sent as the third raw data to the motion virtual-real mapping module. The motion virtual-real mapping module receives the first raw data, the second raw data, and the third raw data, maps them to obtain the motion commands of the slave robot, and selects to record the motion commands for a certain time period based on the pressing state of the two pedals of the foot position acquisition module, and sends the corresponding motion command sequence to the operation safety assessment module. The operation safety assessment module receives the motion command sequence sent by the motion virtual-real mapping module. Based on the distance between the virtual simulation model of the slave robot and the obstacles in the virtual simulation environment, as well as the time delay caused by the motion command being sent from the master controller to the slave controller for processing and feedback, the module adjusts the mapping ratio of the motion command and obtains the operation command, which is then sent to the slave controller. The slave controller receives operation instructions sent by the operation safety assessment module and drives the slave robot to perform corresponding actions according to the operation instructions.

2. The time-delay variable gain robot virtual-real interaction telecontrol system according to claim 1, characterized in that, The acquisition of real-time wrist pose in Cartesian space is specifically as follows: The wrist position acquisition module uses two master robotic arms to acquire the pose of the left and right wrists of the human body. When in use, the human wrist is fixed to the end of the master robotic arm, and the end pose of the master robotic arm is used as the human wrist pose to obtain the real-time wrist pose in Cartesian space.

3. The time-delay variable gain robot virtual-real interaction telecontrol system according to claim 2, characterized in that, The mapping process yields motion commands for the slave robot, as follows: (1) Establish the master arm base coordinate system of the master robot and the slave arm base coordinate system of the slave robot, and align the two base coordinate systems: Let the master arm base coordinate system be... Let the coordinate system of the arm base be The alignment process yields the transformation matrix. By transforming the matrix Establish operational intuition between the master arm and slave arm; (2) Establish the coordinate system of the master robotic arm's remote control joystick and the coordinate system of the slave robot's mobile platform, and align the remote control joystick coordinate system with the mobile platform coordinate system: Let the forward movement of the remote control joystick coordinate system be the positive x-direction, the leftward movement be the positive y-direction, the forward movement of the mobile platform coordinate system be the positive x-direction, the left side of the robot body be the positive y-direction, and the vertical upward movement of the robot body be the positive z-direction, so that v x =k x *Δx,ω z =k y *Δy; where v x For the speed command in the x-direction of the mobile platform, k x ω is the velocity mapping coefficient of the remote control stick in the x-direction, and Δx is the amount of push of the remote control stick in the x-direction; z For the angular velocity command in the z-direction of the mobile platform, k y Let Δy be the velocity mapping coefficient of the remote control stick in the y-direction, and Δy be the amount of push of the remote control stick in the y-direction. (3) Map the collected first raw data, second raw data, and third raw data into motion commands for the slave robot: The motion command for the slave robot arm is the pose transformation relationship from the end of the arm to the base of the arm at time t, represented by quaternions and position vectors; wherein the pose transformation relationship from the end of the arm to the base of the arm at time t is: Let represent the pose transformation relationship from the end effector of the main boom to the main boom base at time t. The pose transformation relationship from the end of the main boom to the main boom base at the moment the pedal is pressed (T0). The pose transition from the end of the arm to the base of the arm is defined at the moment the pedal is depressed (T0); the motion command for the slave moving platform is the velocity command v in the x-direction. x and the angular velocity command ω in the z direction z The motion commands of the slave robot are a set of motion commands of the slave robotic arm and motion commands of the slave mobile platform.

4. The time-delay variable gain robot virtual-real interaction telecontrol system according to claim 1, characterized in that, The movement command for a certain time period is selected and recorded based on the pressing state of the two pedals of the foot position acquisition module to obtain the corresponding movement command sequence, including: One of the two pedals is defined as the simulation pedal, and the other is defined as the control pedal. The use of the first and second raw data is switched according to the third raw data pedal information. When the third raw data comes from the simulation pedal, the corresponding first and second raw data are used to drive the virtual simulation model of the slave robot. When the third raw data comes from the control pedal, the corresponding first and second raw data are used to drive the real slave robot. All simulations start from the current state of the slave robot. 1) When neither the simulation pedal nor the control pedal is pressed, the virtual simulation model of the slave robot displays the current status of the slave robot in real time in the virtual simulation environment; 2) When the simulation pedal is pressed down and the control pedal is released, the motion command is recorded and the virtual simulation model of the slave robot is driven to move in the virtual simulation environment; 3) When the simulation pedal is pressed down and the control pedal is pressed down at the same time, the motion command recording ends. The motion commands recorded during the period from "pressing down the simulation pedal and lifting the control pedal" to "pressing down the simulation pedal and pressing down the control pedal at the same time" are used as the first motion command sequence and sent as a whole to the operation safety assessment module. When both the simulation pedal and the control pedal are lifted, the first motion command sequence stops being sent. 4) When the simulation pedal is raised and the control pedal is pressed, the motion virtual-real mapping module sends the motion command at each moment to the operation safety assessment module in real time until both the simulation pedal and the control pedal are raised. The motion command recorded during the period from "raising the simulation pedal and pressing the control pedal" to "both the simulation pedal and the control pedal are raised" is used as the second motion command sequence.

5. A time-delay variable gain robot virtual-real interaction telecontrol system according to claim 3, characterized in that, Based on the distance between the virtual simulation model of the slave robot and the obstacles in the virtual simulation environment, the mapping ratio of motion commands is adjusted as follows: d = min(sdf(obj, robot)) Where sdf is the distance sign function, used to calculate the distance between models in the virtual scene, obj is the environmental obstacle point vector with dimension 3×n, robot is the robot model, and after obtaining the distances between n obstacles and the robot, the minimum value is found to obtain d, K. min For the minimum gain, d min K1 is the safe distance; K1 is the distance safety gain, when K1 < 0.3K min When d ≤ d, it indicates a high-risk collision state. min When the collision risk is indicated, it indicates a medium-risk collision state; otherwise, it indicates a low-risk collision state.

6. A time-delay variable gain robot virtual-real interaction telecontrol system according to claim 5, characterized in that, Based on the time delay generated when motion commands are sent from the master controller to the slave controller for processing and feedback, the mapping ratio of the motion commands is adjusted as follows: Each motion command to be sent is marked with a local timestamp T1 and a frame number d1 by the master controller and sent to the slave controller at a fixed sending period. After receiving the motion command of this frame, the slave controller performs slave robot control, reads the slave robot status, and forms the latest status feedback frame of the slave robot to the master controller. The latest status feedback frame includes timestamp T1, frame number d1, and timestamp T2 after T1 is superimposed with the time Δt spent on running and processing. When the master controller receives the latest status feedback frame, it records the arrival time of the feedback frame as timestamp T3. It then calculates the timestamp for a complete motion command cycle, using T3-T1 to obtain the two network transmission delays, and using T2-T1 to obtain the slave processing delay. Finally, it calculates the delay security gain K2 using the following formula: Among them, K M For maximum gain, K m For minimum gain, T max To predict the maximum delay, if T max If no return frame is received within the specified time, T will be automatically used. max As a time estimate; When K2 is large, the slave-side following ratio increases, and the overall operating speed increases. When K2 is small, the slave-side following ratio decreases, and the overall operating speed decreases.

7. A time-delay variable gain robot virtual-real interaction telecontrol system according to claim 6, characterized in that, The operation safety assessment module uses gain K = K1K2 to adjust the mapping ratio of motion commands, and sends the operation commands to the slave controller. Gain K is used to calculate the pose increment of the master end effector, thereby changing the motion command of the slave end effector. in, Let be the original pose transformation relationship from the end effector of the main boom to the main boom base at time t. This represents the original pose transformation relationship from the end of the main arm to the main arm base at time t-Δt in the previous frame. The calculation pose transformation relationship from the end of the main arm to the main arm base is calculated at time t-Δt in the previous frame. And the commands used to calculate the joystick control increments to change the motion commands of the slave mobile platform: v x =Kk x Δx,ω z =Kk y Δy。 8. A time-delay variable gain robot virtual-real interaction telecontrol system according to claim 1, characterized in that, In the virtual simulation environment, environmental spatial structure information collected by the environmental sensor of the slave robot is added and represented in the form of point cloud. In order to increase the control accuracy of the slave robot in the obstacle environment, the point cloud data generated by the environmental sensor scanning the real slave robot is removed in the virtual simulation environment, and only the environmental point cloud is retained.

9. A time-delay variable gain robot virtual-real interaction telecontrol system according to claim 1, characterized in that, The virtual simulation model of the slave robot uses a robot joint linkage model driven by the joint position data of the slave robot.

10. A time-delay variable gain robot virtual-real interaction telecontrol method based on the system described in any one of claims 1 to 9, characterized in that, Includes the following steps: Collect data on wrist, finger, and foot movements; The collected motion data is mapped to obtain motion commands for the slave robot. Based on the different pressing states of the foot pedal action, the motion commands for a certain time period are selected and recorded to obtain the corresponding motion command sequence. The system receives a sequence of motion commands and adjusts the mapping ratio of the motion commands based on the distance between the virtual simulation model of the slave robot and the obstacles in the virtual simulation environment, as well as the time delay caused by the motion commands being sent from the master controller to the slave controller for processing and feedback, to obtain the operation commands. Drive the slave robot to perform corresponding actions according to the operation instructions and provide feedback on the action status.

Images