Time delay variable gain robot virtual-real interaction remote control system and method

Abstract

The invention relates to a time-delay variable-gain robot virtual-real interaction remote control system and method. The method comprises the following steps: acquiring data of wrist, finger and pedal actions; mapping the collected action data to obtain a motion instruction of the slave end robot, and selecting and recording the motion instruction in a certain time period according to different treading states of the pedal action to obtain a corresponding motion instruction sequence; a motion instruction sequence is received, and according to the distance between a slave-end robot virtual simulation model and a virtual simulation environment space obstacle and the time delay generated when the motion instruction is sent from the master-end controller to the slave-end controller to be processed and fed back, mapping proportion adjustment is conducted on the motion instruction to obtain an operation instruction; and according to the operation instruction, the slave end robot is driven to execute a corresponding action, and action state feedback is carried out. The problems that in the current robot teleoperation technology, operation is unstable and operation is prone to collision due to unstable time delay jitter and sensing limitation are solved.

Description

Technical Field

[0001] The invention belongs to the technical field of robot teleoperation, and relates to a time-delay variable-gain robot virtual-reality interactive telecontrol system and method. Background Art

[0002] Tele-operation refers to the process in which an operator remotely controls a robot to complete a task through a human-machine interaction device such as a joystick. When performing a task, the operator sends instructions to the slave robot through the master robot and receives feedback information from the robot and the outside world at the same time, thereby completing the tele-operation task. However, during the tele-operation process, problems such as high robot dimensions and unstable communication delays may lead to low operating efficiency and poor safety.

[0003] Among the existing remote control algorithms, the patent "Force-guided remote control system and control method based on dual-arm collaborative potential field" (CN202410208048.5) generates dynamic virtual constraint force based on the dual-arm collaborative potential field, and assists the operator in manipulating the robotic arm to complete the dual-arm collaborative task through two force feedback hand controllers. This method mainly uses environmental information to generate virtual contact force, which is not flexible enough when dealing with dynamic environments, and the virtual force may have a negative impact on the operation accuracy. The patent "A data-driven and mechanism-mixed modeling and control method for virtual-real fusion control" (CN202211663067.4) discloses a virtual-real fusion remote control method, which generates a predicted state vector through a prediction model to form a stable predicted control instruction, but the method has limited effect in dealing with unstable time delays. Summary of the invention

[0004] The technical problem solved by the present invention is: to overcome the shortcomings of the prior art, to propose a robot virtual-reality interactive remote control system and method with time delay and variable gain, to solve the problems of unstable time delay jitter, unstable operation caused by limited perception, and easy collision in operation existing in the current robot remote control technology.

[0005] The technical solution of the present invention is: a time-delay variable gain robot virtual-reality interactive remote control system, the master end includes a master end controller, a human motion acquisition module, a motion virtual-reality mapping module and an operation safety assessment module; the slave end includes a slave end controller and a slave end robot;

[0006] The master controller is the data interaction and processing platform of the human motion acquisition module, motion virtual-reality mapping module, and operation safety assessment module, and displays the slave robot virtual simulation model and virtual simulation environment;

[0007] The human motion acquisition module includes a wrist position acquisition module, a finger position acquisition module and a foot position acquisition module. The three position acquisition modules are connected and communicated with the main controller through a data bus.

[0008] Wrist position acquisition module, which collects real-time wrist position in Cartesian space;

[0009] The finger position acquisition module uses a remote control rod to acquire the position of human fingers, pushes the remote control rod, obtains the remote control rod control increment as the first raw data and sends it to the motion virtual-real mapping module;

[0010] The foot position acquisition module uses two pedals to capture the pressing action of the left and right feet of the human body. When any one of the pedals is stepped on, the wrist posture from the moment of stepping on to the current moment is obtained, and the wrist posture increment is sent to the motion virtual-real mapping module as the second original data; when both pedals are released, the motion virtual-real mapping module sets the wrist posture increment to 0; the data of the left and right feet stepping on the pedals is sent to the motion virtual-real mapping module as the third original data;

[0011] The motion virtual-real mapping module receives the first raw data, the second raw data, and the third raw data, maps and obtains the motion instructions of the slave robot, and selects and records the motion instructions of a certain time period according to the pressing state of the two pedals of the foot position acquisition module, obtains the corresponding motion instruction sequence and sends it to the operation safety assessment module;

[0012] The operation safety assessment module receives the motion instruction sequence sent by the motion virtual-reality mapping module, adjusts the mapping ratio of the motion instruction according to the distance between the virtual simulation model of the slave robot and the space obstacle of the virtual simulation environment, and the time delay caused by the motion instruction being sent from the master controller to the slave controller for processing and feedback, and obtains the operation instruction to send to the slave controller;

[0013] The slave controller receives the operation instructions sent by the operation safety assessment module and drives the slave robot to perform corresponding actions according to the operation instructions.

[0014] Furthermore, the real-time wrist posture of the Cartesian space is collected as follows:

[0015] The wrist position acquisition module uses two main-end robotic arms to respectively collect the postures of the left and right wrists of the human body. When in use, the human wrist is fixedly connected to the end of the main-end robotic arm, and the end posture of the main-end robotic arm is used as the human wrist posture to obtain the real-time wrist posture in Cartesian space.

[0016] Furthermore, the mapping obtains the motion instructions of the slave robot in the following manner:

[0017] (1) Establish the main arm base coordinate system of the master end robot and the slave arm base coordinate system of the slave end robot, and align the two base coordinate systems: Let the main arm base coordinate system be Let the coordinate system of the slave arm base be The alignment process obtains the transformation matrix By transforming the matrix Establish the operation intuition between the master arm and the slave arm. For example, if you push the master arm forward, the slave arm will also move forward synchronously.

[0018] (2) Establish the remote control rod coordinate system of the master robot and the mobile platform coordinate system of the slave robot, and align the remote control rod coordinate system with the mobile platform coordinate system: let the remote control rod coordinate system push forward as the positive x direction, push left as the positive y direction, the mobile platform coordinate system forward as the positive x direction, the left side of the fuselage as the positive y direction, and the vertical upward direction of the fuselage as the positive z direction, so that v x =k x *Δx,ω z =k y *Δy; where v x is the speed command of the mobile platform in the x direction, k x is the velocity mapping coefficient of the remote control rod in the x direction, Δx is the push amount of the remote control rod in the x direction; ω z is the angular velocity command of the mobile platform in the z direction, k y is the velocity mapping coefficient of the remote control stick in the y direction, Δy is the pushing amount of the remote control stick in the y direction;

[0019] (3) Mapping the collected first raw data, second raw data, and third raw data into motion instructions of the slave robot: the motion instruction of the slave robot arm is the posture transformation relationship from the end of the arm to the base of the slave arm at time t, expressed in the form of quaternion and position vector; wherein the posture transformation relationship from the end of the arm to the base of the slave arm at time t is: is the posture transformation relationship from the end of the main arm to the base of the main arm at time t, is the posture transformation relationship from the end of the main arm to the base of the main arm at the moment T0 when the pedal is pressed, is the position conversion relationship from the end of the arm to the base of the slave arm when the pedal is pressed T0; the motion command of the slave end mobile platform is the speed command v in the x direction x and the angular velocity command ω in the z direction z The motion instructions of the slave robot are a collection of the motion instructions of the slave robot arm and the motion instructions of the slave mobile platform.

[0020] Furthermore, the motion instructions for a certain period of time are selected and recorded according to the pressing states of the two pedals of the sole position acquisition module to obtain a corresponding motion instruction sequence, including:

[0021] One of the two pedals is defined as a simulation pedal, and the other is defined as a control pedal. The usage of the first and second raw data is switched according to the third raw data pedal information. When the third raw data comes from the simulation pedal, the corresponding first and second raw data are used to drive the virtual simulation model of the slave robot. When the third raw data comes from the control pedal, the corresponding first and second raw data are used to drive the real slave robot. All simulations start from the current slave robot state.

[0022] 1) When neither the simulation pedal nor the control pedal is pressed, the slave robot virtual simulation model displays the current slave robot status in real time in the virtual simulation environment;

[0023] 2) When the simulation pedal is pressed down and the control pedal is lifted up, the motion instructions are recorded and the slave robot virtual simulation model is driven to move in the virtual simulation environment;

[0024] 3) When the simulation pedal is stepped on and the control pedal is stepped on at the same time, the motion instruction recording ends, and the motion instructions recorded during the period from "stepping on the simulation pedal and lifting the control pedal" to "stepping on the simulation pedal and stepping on the control pedal at the same time" are taken as the first motion instruction sequence, which is sent as a whole to the operation safety assessment module. When both the simulation pedal and the control pedal are lifted, the first motion instruction sequence stops being sent;

[0025] 4) When the simulation pedal is lifted and the control pedal is pressed, the motion virtual-reality mapping module sends the motion instructions at each moment to the operation safety assessment module in real time until both the simulation pedal and the control pedal are lifted. The motion instructions recorded during the period from "lifting the simulation pedal and pressing the control pedal" to "both the simulation pedal and the control pedal are lifted" are used as the second motion instruction sequence.

[0026] Furthermore, according to the distance between the slave robot virtual simulation model and the space obstacle of the virtual simulation environment, the motion command is mapped proportionally and adjusted as follows:

[0027]

[0028] d=min(sdf(obj,robot))

[0029] Among them, sdf is the distance sign function, which is used to calculate the distance between each model in the virtual scene, obj is the environmental obstacle point vector, the dimension is 3×n, robot is the robot model, after obtaining the distance between n obstacles and the robot, the minimum value is obtained to obtain d, K min is the minimum gain, d min is the safety distance; K1 is the distance safety gain, when K1<0.3K min When d≤d minWhen , it indicates a medium risk state of collision, and in other cases, it indicates a low risk state of collision.

[0030] Furthermore, according to the time delay caused by the motion command being sent from the master controller to the slave controller for processing and feedback, the mapping ratio of the motion command is adjusted as follows:

[0031] Each motion instruction to be sent will be marked with the local timestamp T1 and frame number d1 of the master controller, and sent to the slave controller at a fixed sending cycle. After receiving the motion instruction of this frame, the slave controller controls the slave robot, reads the status of the slave robot, and forms the latest status feedback frame of the slave robot to the master controller. The latest status feedback frame includes the timestamp T1, the frame number d1, and the timestamp T2 after T1 is superimposed on the time Δt spent on the operation processing.

[0032] When the master controller receives the latest state feedback frame, it records the arrival time of the feedback frame as timestamp T3, performs timestamp calculation for a complete motion instruction cycle, uses T3-T1 to obtain two network transmission delays, uses T2-T1 to obtain the slave processing delay, and uses the following formula to calculate the delay safety gain K2:

[0033]

[0034] Among them, K M is the maximum gain, K m is the minimum gain, T max is the expected maximum delay, if T max If no return frame is received within the time, T max As a time estimate;

[0035] When K2 is larger, the slave-end following ratio increases and the overall operating speed increases. When K2 is smaller, the slave-end following ratio decreases and the overall operating speed decreases.

[0036] Furthermore, the operation safety assessment module uses the gain K=K1K2 to adjust the mapping ratio of the motion instruction, obtains the operation instruction and sends it to the slave controller;

[0037] The gain K is used to calculate the end position increment of the master robot arm to change the motion command of the slave robot arm:

[0038]

[0039] in, is the original posture transformation relationship from the end of the main arm to the base of the main arm at time t, Calculate the original posture transformation relationship from the end of the main arm to the base of the main arm at time t-Δt in the previous frame. The calculated posture transformation relationship from the end of the main arm to the base of the main arm at the calculation time t-Δt in the previous frame;

[0040] And used to calculate the control increment of the remote control stick to change the movement instruction of the slave mobile platform:

[0041] v x =Kk x Δx,ω z =Kk y Δy.

[0042] Furthermore, the environmental spatial structure information collected by the environmental sensor of the slave robot is added to the virtual simulation environment and expressed in the form of point cloud. In order to increase the control accuracy of the slave robot in an obstacle environment, the point cloud data generated by the environmental sensor detection and scanning of the real slave robot is removed in the virtual simulation environment, and only the environmental point cloud is retained.

[0043] Furthermore, the slave robot virtual simulation model uses a robot joint link model driven by the slave robot joint position data.

[0044] A time-delay variable gain robot virtual-reality interactive remote control method, comprising the following steps:

[0045] Collect wrist, finger, and foot movement data;

[0046] The collected motion data is mapped to obtain the motion instructions of the slave robot, and the motion instructions of a certain period of time are selected and recorded according to the different stepping states of the pedaling action to obtain the corresponding motion instruction sequence;

[0047] Receive a motion command sequence, and adjust the mapping ratio of the motion command to obtain an operation command according to the distance between the slave robot virtual simulation model and the space obstacle of the virtual simulation environment, and the time delay caused by the motion command being sent from the master controller to the slave controller for processing and feedback;

[0048] Drive the slave robot to perform corresponding actions according to the operation instructions and provide action status feedback.

[0049] The beneficial effects of the present invention compared with the prior art are:

[0050] (1) The present invention proposes a time-delay variable gain robot virtual-reality interactive remote control method, introduces a real-time safety assessment method in the virtual-reality interactive remote operation, and flexibly switches between virtual asynchronous control and real-time synchronous control, thereby reducing the impact of time delay uncertainty and collision risk during remote control.

[0051] (2) The operation safety assessment module of the present invention assesses collision safety through a distance sign function, assesses real-time delay through a frame tag field, and calculates the safety evaluation value of the operation instruction in real time for incremental instruction adjustment, thereby improving remote control safety.

[0052] (3) In the remote control system proposed by the present invention, the human body motion acquisition module acquires the human body's wrist, finger, and foot movements as data input, thereby capturing the complete human body movements, which can be used to generate different types of control instructions. The acquisition module's acquisition method adapts to people's usage habits and can reduce learning costs.

[0053] (4) The virtual-reality interaction method of the present invention realizes the integration of virtual-reality synchronous and asynchronous control modes through rapid switching of input devices, which can enhance the comprehensiveness of the operator's environmental perception, thereby quickly and flexibly handling a variety of remote operation tasks.

[0054] (5) The motion virtual-real mapping module of the present invention realizes command mapping through coordinate system alignment, which can intuitively reflect human body movements. The operation safety assessment module adopts an incremental mode to expand the operator's operating range and meet the needs of large-scale operation and fine operation. This mode is universal on various types of robots. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a schematic diagram of a human motion acquisition module of a remote control system according to an embodiment of the present invention;

[0056] Figure 2 This is a schematic diagram of data flow between the master and slave ends of the remote control system according to an embodiment of the present invention;

[0057] Figure 3 This is a flow chart of gain calculation of the operation safety assessment module of the remote control system according to an embodiment of the present invention;

[0058] Figure 4 This is a flow chart of a method for virtual-reality interactive remote control of a robot with time-delay variable gain according to an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The present invention proposes a time-delay variable gain robot virtual-reality interactive remote control system and method. By estimating the transmission and reception loop delay and calculating the safe obstacle distance, the control gain is dynamically adjusted to reduce the impact of unstable delay and collision risk during operation, thereby improving the stability and safety of control. Through the flexible switching of asynchronous control of the virtual environment and real-time synchronous operation, the comprehensiveness of perception in virtual-reality combined remote operation is improved.

[0060] The present invention will be further described below in conjunction with the embodiments.

[0061] Example 1

[0062] The present invention proposes a time-delay variable gain robot virtual-reality interactive remote control system, in which the master end collects human motions and converts them into operation instructions and sends them to the slave end controller, and the slave end controller drives the slave end robot to perform corresponding actions according to the received operation instructions.

[0063] Specifically, the master end includes a master end controller, a human motion acquisition module, a motion virtual-reality mapping module and an operation safety assessment module; the slave end includes a slave end controller and a slave end robot.

[0064] The master controller is the data interaction and processing platform of the human motion acquisition module, motion virtual-reality mapping module, and operation safety assessment module, and displays the slave robot virtual simulation model and virtual simulation environment.

[0065] The human motion acquisition module includes a wrist position acquisition module, a finger position acquisition module and a foot position acquisition module. The three position acquisition modules are connected and communicated with the main controller through a data bus.

[0066] like Figure 1 As shown, the wrist position acquisition module uses two master-end robotic arms 203 to respectively acquire the postures of the left and right wrists of a human body 204. When in use, the human wrist is fixedly connected to the end of the master-end robotic arm 203, so that the end posture of the master-end robotic arm 203 is used as the human wrist posture, and the real-time wrist posture in Cartesian space is obtained.

[0067] The finger position acquisition module uses the remote control rod 202 to acquire the finger positions of the human body 204. The remote control rod 202 is pushed to obtain the remote control rod control increment, and the remote control rod control increment is sent to the motion virtual-real mapping module as the first original data.

[0068] The foot position acquisition module uses two pedals 205 to capture the pressing action of the left and right feet of the human body 204. When any pedal 205 is stepped on, the wrist posture from the stepping moment to the current moment is obtained, and the wrist posture increment is obtained as the second raw data and sent to the motion virtual-real mapping module; when both pedals 205 are released, the motion virtual-real mapping module sets the wrist posture increment to 0.

[0069] The data of the left and right feet stepping on the pedal 205 is sent to the motion virtual-real mapping module as the third original data.

[0070] The motion virtual-real mapping module receives the first raw data, the second raw data, and the third raw data, maps the motion instructions of the slave robot, and selects to record the motion instructions for a certain time period according to the pressing status of the two pedals 205 of the foot position acquisition module, and obtains the corresponding motion instruction sequence and sends it to the operation safety assessment module.

[0071] In this embodiment, the specific implementation method of the motion virtual-real mapping module to obtain the motion instruction of the slave robot according to the first original data, the second original data, and the third original data is as follows:

[0072] (1) Establish the main arm base coordinate system of the master end robot and the slave arm base coordinate system of the slave end robot, and align the two base coordinate systems: Let the main arm base coordinate system be Let the coordinate system of the slave arm base be The alignment process obtains the transformation matrix By transforming the matrix Establish the operation intuition between the master arm and the slave arm. For example, if you push the master arm forward, the slave arm will also move forward synchronously.

[0073] (2) Establish the remote control rod coordinate system of the master robot and the mobile platform coordinate system of the slave robot, and align the remote control rod coordinate system with the mobile platform coordinate system: let the remote control rod coordinate system push forward as the positive x direction, push left as the positive y direction, the mobile platform coordinate system forward as the positive x direction, the left side of the fuselage as the positive y direction, and the vertical upward direction of the fuselage as the positive z direction, so that v x =k x *Δx,ω z =k y *Δy; where v x is the speed command of the mobile platform in the x direction, k x is the velocity mapping coefficient of the remote control rod in the x direction, Δx is the push amount of the remote control rod in the x direction; ω z is the angular velocity command of the mobile platform in the z direction, k y is the velocity mapping coefficient of the remote control stick in the y direction, Δy is the pushing amount of the remote control stick in the y direction;

[0074] (3) Mapping the collected first raw data, second raw data, and third raw data into motion instructions of the slave robot: the motion instruction of the slave robot arm is the posture transformation relationship from the end of the arm to the base of the slave arm at time t, expressed in the form of quaternion and position vector; wherein the posture transformation relationship from the end of the arm to the base of the slave arm at time t is: is the posture transformation relationship from the end of the main arm to the base of the main arm at time t, is the posture transformation relationship from the end of the main arm to the base of the main arm at the moment T0 when the pedal is pressed, is the position conversion relationship from the end of the arm to the base of the slave arm when the pedal is pressed T0; the motion command of the slave end mobile platform is the speed command v in the x direction x and the angular velocity command ω in the z direction z The motion instructions of the slave robot are a collection of the motion instructions of the slave robot arm and the motion instructions of the slave mobile platform.

[0075] In this embodiment, the motion virtual-real mapping module selects and records the motion instructions for a certain period of time according to the pressing states of the two pedals 205 of the sole position acquisition module to obtain the corresponding motion instruction sequence, and the following method is preferred:

[0076] like Figure 4 As shown, one of the two pedals 205 is defined as a simulation pedal, and the other is defined as a control pedal. The usage of the first and second raw data is switched according to the third raw data pedal information. When the third raw data comes from the simulation pedal, the corresponding first and second raw data are used to drive the virtual simulation model of the slave robot. When the third raw data comes from the control pedal, the corresponding first and second raw data are used to drive the real slave robot. All simulations start from the current slave robot state.

[0077] 1) When neither the simulation pedal nor the control pedal is pressed, the slave robot virtual simulation model displays the current slave robot status in real time in the virtual simulation environment;

[0078] 2) When the simulation pedal is pressed down and the control pedal is lifted up, the motion instructions are recorded and the slave robot virtual simulation model is driven to move in the virtual simulation environment;

[0079] 3) When the simulation pedal is stepped on and the control pedal is stepped on at the same time, the motion instruction recording ends, and the motion instructions recorded during the period from "stepping on the simulation pedal and lifting the control pedal" to "stepping on the simulation pedal and stepping on the control pedal at the same time" are taken as the first motion instruction sequence, which is sent as a whole to the operation safety assessment module. When both the simulation pedal and the control pedal are lifted, the first motion instruction sequence stops being sent;

[0080] 4) When the simulation pedal is lifted and the control pedal is pressed, the motion virtual-reality mapping module sends the motion instructions at each moment to the operation safety assessment module in real time until both the simulation pedal and the control pedal are lifted. The motion instructions recorded during the period from "lifting the simulation pedal and pressing the control pedal" to "both the simulation pedal and the control pedal are lifted" are used as the second motion instruction sequence.

[0081] In this embodiment, the perception of the slave robot is displayed around the user interface of the master controller, and the virtual simulation environment is located in the center of the user interface of the master controller. The environmental spatial structure information collected by the slave robot's environmental sensor (such as lidar, RGBD camera) is added to the virtual simulation environment and expressed in the form of point cloud. In order to increase the control accuracy of the slave robot in an obstacle environment, the point cloud data generated by the environmental sensor detection and scanning on the real slave robot is removed in the virtual simulation environment, and only the environmental point cloud is retained. The removal process uses a capsule-shaped bounding box to select the point cloud of the main body part, and the slave robot virtual simulation model uses a robot joint linkage model driven by the slave robot joint position data.

[0082] The operation safety assessment module receives the motion command sequence sent by the motion virtual-reality mapping module, adjusts the mapping ratio of the motion command according to the distance between the slave robot virtual simulation model and the obstacles in the virtual simulation environment, and the delay caused by the motion command being sent from the master controller to the slave controller for processing and feedback, and obtains the operation command to send to the slave controller. Figure 4 As shown in S103.

[0083] Specifically, the operation safety assessment module includes a distance safety assessment module and a delay safety assessment module. The process of calculating the assessment gain by the operation safety assessment module is as follows: Figure 3 shown.

[0084] The distance safety assessment module is used to calculate the distance between the slave robot virtual simulation model and the space obstacle of the virtual simulation environment after the motion instruction sequence is generated, and obtain the distance safety gain K1. The calculation method is:

[0085]

[0086] d=min(sdf(obj,robot))

[0087] Among them, sdf is the distance sign function, which is used to calculate the distance between each model in the virtual scene, obj is the environmental obstacle point vector, the dimension is 3×n, robot is the robot model, after obtaining the distance between n obstacles and the robot, the minimum value is obtained to obtain d, K min is the minimum gain, d min For a safe distance.

[0088] After the distance safety assessment module obtains the distance safety gain sequence calculated by the environmental collision and the self-model collision, it is synchronously sent to the operation safety assessment module. The distance safety gain is also used for user interface display. When K1<0.3K min When d≤d min , it indicates a medium-risk collision state, and the interface environment color is yellow. In other cases, it indicates a low-risk collision state, and the interface environment color is green.

[0089] The delay safety evaluation module is used to calculate the delay safety gain K2 according to the delay caused by the motion command sent from the master controller to the slave controller for processing and feedback. The calculation method is:

[0090] Each motion instruction to be sent will be marked with the local timestamp T1 and frame number d1 of the master controller, and sent to the slave controller with a fixed sending cycle. After receiving the motion instruction of this frame, the slave controller controls the slave robot, reads the status of the slave robot, and forms the latest status feedback frame of the slave robot to the master controller. The latest status feedback frame includes the timestamp T1, the frame number d1, and the timestamp T2 after T1 is superimposed on the running processing time Δt. When the master controller receives the latest status feedback frame, the arrival time of the feedback frame is recorded as the timestamp T3. The delay safety assessment module calculates the timestamp of a complete motion instruction cycle, uses T3-T1 to obtain the two network transmission delays, uses T2-T1 to obtain the slave processing delay, and uses the following formula to calculate the delay safety gain K2:

[0091]

[0092] Where K M is the maximum gain, K m is the minimum gain, T max is the expected maximum delay, if T max If no return frame is received within the time, T max As a time estimate.

[0093] The time-delay safety gain K2 is used to adjust the ratio of the master-end incremental speed to the slave-end incremental instruction. When the gain is large, the slave-end can more closely replicate the master-end action. When the gain is small, the slave-end following ratio decreases and the overall operating speed decreases.

[0094] The operation safety assessment module comprehensively considers obstacle avoidance safety and delay stability, uses gain K=K1K2 to adjust the mapping ratio of the motion command, and obtains the operation command to send to the slave controller. Gain K is used to calculate the end position increment of the master robot arm to change the slave robot arm motion command:

[0095]

[0096] in, is the original posture transformation relationship from the end of the main arm to the base of the main arm at time t, Calculate the original posture transformation relationship from the end of the main arm to the base of the main arm at time t-Δt in the previous frame. The calculated posture transformation relationship from the end of the main arm to the base of the main arm at the calculation time t-Δt in the previous frame;

[0097] And used to calculate the control increment of the remote control stick to change the movement instruction of the slave mobile platform:

[0098] v x =Kk x Δx,ω z =Kky Δy.

[0099] The slave controller receives the operation instructions sent by the operation safety assessment module and drives the slave robot to perform corresponding actions according to the operation instructions.

[0100] The data flow of the entire remote control system from master to slave is as follows Figure 2 shown.

[0101] Based on the remote control system of the present invention, a time-delay variable gain robot virtual-real interactive remote control method is provided, such as Figure 4 As shown, the following steps are included:

[0102] S1, collect wrist, finger and foot movement data;

[0103] S2. Map the collected motion data to obtain the motion instructions of the slave robot, select and record the motion instructions of a certain period of time according to the different pedaling states, and obtain the corresponding motion instruction sequence. The different pedaling states include the following four:

[0104] 1) When neither the simulation pedal nor the control pedal is pressed, the slave robot virtual simulation model displays the current slave robot status in real time in the virtual simulation environment;

[0105] 2) When the simulation pedal is pressed down and the control pedal is lifted up, the motion instructions are recorded and the slave robot virtual simulation model is driven to move in the virtual simulation environment;

[0106] 3) When the simulation pedal is stepped on and the control pedal is stepped on at the same time, the motion instruction recording ends, and the motion instructions recorded during the period from "stepping on the simulation pedal and lifting the control pedal" to "stepping on the simulation pedal and stepping on the control pedal at the same time" are taken as the first motion instruction sequence, which is sent as a whole to the operation safety assessment module. When both the simulation pedal and the control pedal are lifted, the first motion instruction sequence stops being sent;

[0107] 4) When the simulation pedal is lifted and the control pedal is pressed, the motion virtual-reality mapping module sends the motion instructions at each moment to the operation safety assessment module in real time until both the simulation pedal and the control pedal are lifted. The motion instructions recorded during the period from "lifting the simulation pedal and pressing the control pedal" to "both the simulation pedal and the control pedal are lifted" are used as the second motion instruction sequence.

[0108] S3, receiving a motion instruction sequence, and adjusting the mapping ratio of the motion instructions to obtain operation instructions according to the distance between the slave robot virtual simulation model and the virtual simulation environment space obstacle, and the time delay caused by the motion instructions being sent from the master controller to the slave controller for processing and feedback.

[0109] S4. Drive the slave robot to perform corresponding actions according to the operation instructions and provide action status feedback.

[0110] In summary, the present invention proposes a remote operation system and method that combines virtual simulation and real-time safety assessment, introduces a real-time safety assessment mechanism in the remote operation of virtual-reality interaction, and flexibly switches between virtual asynchronous control and real-time synchronous control modes, thereby reducing the impact of delay uncertainty and collision risk during the remote control process.

[0111] The present invention innovatively proposes a quantitative evaluation method for operation safety, introduces a distance sign function to evaluate collision safety, evaluates real-time delay through frame marking fields, and calculates the safety evaluation value of operation instructions in real time for incremental mapping adjustment, thereby improving remote control safety.

[0112] Although the present invention has been disclosed as above in the form of a preferred embodiment, it is not intended to limit the present invention. Any person skilled in the art may make possible changes and modifications to the technical solution of the present invention by using the methods and technical contents disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope of the technical solution of the present invention.

[0113] The contents not described in detail in the specification of the present invention belong to the common knowledge of those skilled in the art.

Claims

1. A time-delay variable gain robot virtual-reality interactive remote control system, characterized in that: The master end includes a master end controller, a human motion acquisition module, a motion virtual-reality mapping module, and an operation safety assessment module; the slave end includes a slave end controller and a slave end robot; The master controller is the data interaction and processing platform of the human motion acquisition module, motion virtual-reality mapping module, and operation safety assessment module, and displays the slave robot virtual simulation model and virtual simulation environment; The human motion acquisition module includes a wrist position acquisition module, a finger position acquisition module and a foot position acquisition module. The three position acquisition modules are connected and communicated with the main end controller through a data bus; Wrist position acquisition module, which collects real-time wrist position in Cartesian space; The finger position acquisition module uses a remote control rod to acquire the position of human fingers, pushes the remote control rod, obtains the remote control rod control increment as the first raw data and sends it to the motion virtual-real mapping module; The foot position acquisition module uses two pedals to capture the pressing action of the left and right feet of the human body. When any one of the pedals is stepped on, the wrist posture from the moment of stepping on to the current moment is obtained, and the wrist posture increment is sent to the motion virtual-real mapping module as the second original data; when both pedals are released, the motion virtual-real mapping module sets the wrist posture increment to 0; the data of the left and right feet stepping on the pedals is sent to the motion virtual-real mapping module as the third original data; The motion virtual-real mapping module receives the first raw data, the second raw data, and the third raw data, maps and obtains the motion instructions of the slave robot, and selects and records the motion instructions of a certain time period according to the pressing state of the two pedals of the foot position acquisition module, obtains the corresponding motion instruction sequence and sends it to the operation safety assessment module; The operation safety assessment module receives the motion instruction sequence sent by the motion virtual-reality mapping module, adjusts the mapping ratio of the motion instruction according to the distance between the virtual simulation model of the slave robot and the space obstacle of the virtual simulation environment, and the time delay caused by the motion instruction being sent from the master controller to the slave controller for processing and feedback, and obtains the operation instruction to send to the slave controller; The slave controller receives the operation instructions sent by the operation safety assessment module and drives the slave robot to perform corresponding actions according to the operation instructions.

2. The time-delay variable gain robot virtual-reality interactive remote control system according to claim 1, characterized in that: The real-time wrist posture of the Cartesian space is collected as follows: The wrist position acquisition module uses two main-end robotic arms to respectively collect the postures of the left and right wrists of the human body. When in use, the human wrist is fixedly connected to the end of the main-end robotic arm, and the end posture of the main-end robotic arm is used as the human wrist posture to obtain the real-time wrist posture in Cartesian space.

3. The time-delay variable gain robot virtual-reality interactive remote control system according to claim 2, characterized in that: The mapping obtains the motion instructions of the slave robot in the following way: (1) Establish the main arm base coordinate system of the master end robot and the slave arm base coordinate system of the slave end robot, and align the two base coordinate systems: Let the main arm base coordinate system be Let the coordinate system of the slave arm base be The alignment process obtains the transformation matrix By transforming the matrix Establish the operation intuition between the master arm and the slave arm. For example, if you push the master arm forward, the slave arm will also move forward synchronously. (2) Establish the remote control rod coordinate system of the master robot and the mobile platform coordinate system of the slave robot, and align the remote control rod coordinate system with the mobile platform coordinate system: let the remote control rod coordinate system push forward as the positive x direction, push left as the positive y direction, the mobile platform coordinate system forward as the positive x direction, the left side of the fuselage as the positive y direction, and the vertical upward direction of the fuselage as the positive z direction, so that v x =k x *Δx,ω z =k y *Δy; where v x is the speed command of the mobile platform in the x direction, k x is the velocity mapping coefficient of the remote control rod in the x direction, Δx is the push amount of the remote control rod in the x direction; ω z is the angular velocity command of the mobile platform in the z direction, k y is the velocity mapping coefficient of the remote control stick in the y direction, Δy is the pushing amount of the remote control stick in the y direction; (3) Mapping the collected first raw data, second raw data, and third raw data into motion instructions of the slave robot: the motion instruction of the slave robot arm is the posture transformation relationship from the end of the arm to the base of the slave arm at time t, expressed in the form of quaternion and position vector; wherein the posture transformation relationship from the end of the arm to the base of the slave arm at time t is: is the posture transformation relationship from the end of the main arm to the base of the main arm at time t, is the posture transformation relationship from the end of the main arm to the base of the main arm at the moment T0 when the pedal is pressed, is the position conversion relationship from the end of the arm to the base of the slave arm when the pedal is pressed T0; the motion command of the slave end mobile platform is the speed command v in the x direction x and the angular velocity command ω in the z direction z The motion instructions of the slave robot are a collection of the motion instructions of the slave robot arm and the motion instructions of the slave mobile platform.

4. The time-delay variable gain robot virtual-reality interactive remote control system according to claim 1, characterized in that: The step of selecting and recording the motion instructions for a certain period of time according to the pressing states of the two pedals of the sole position acquisition module to obtain the corresponding motion instruction sequence includes: One of the two pedals is defined as a simulation pedal, and the other is defined as a control pedal. The usage of the first and second raw data is switched according to the third raw data pedal information. When the third raw data comes from the simulation pedal, the corresponding first and second raw data are used to drive the virtual simulation model of the slave robot. When the third raw data comes from the control pedal, the corresponding first and second raw data are used to drive the real slave robot. All simulations start from the current slave robot state. 1) When neither the simulation pedal nor the control pedal is pressed, the slave robot virtual simulation model displays the current slave robot status in real time in the virtual simulation environment; 2) When the simulation pedal is pressed down and the control pedal is lifted up, the motion instructions are recorded and the slave robot virtual simulation model is driven to move in the virtual simulation environment; 3) When the simulation pedal is stepped on and the control pedal is stepped on at the same time, the motion instruction recording ends, and the motion instructions recorded during the period from "stepping on the simulation pedal and lifting the control pedal" to "stepping on the simulation pedal and stepping on the control pedal at the same time" are taken as the first motion instruction sequence, which is sent as a whole to the operation safety assessment module. When both the simulation pedal and the control pedal are lifted, the first motion instruction sequence stops being sent; 4) When the simulation pedal is lifted and the control pedal is pressed, the motion virtual-reality mapping module sends the motion instructions at each moment to the operation safety assessment module in real time until both the simulation pedal and the control pedal are lifted and stops sending. The motion instructions recorded during the period from "lifting the simulation pedal and pressing the control pedal" to "both the simulation pedal and the control pedal are lifted" are taken as the second motion instruction sequence.

5. The time-delay variable gain robot virtual-reality interactive remote control system according to claim 3, characterized in that: According to the distance between the slave robot virtual simulation model and the obstacles in the virtual simulation environment, the motion command mapping ratio is adjusted as follows: d=min(sdf(obj,robot)) Among them, sdf is the distance sign function, which is used to calculate the distance between each model in the virtual scene, obj is the environmental obstacle point vector, the dimension is 3×n, robot is the robot model, after obtaining the distance between n obstacles and the robot, the minimum value is obtained to obtain d, K min is the minimum gain, d min is the safety distance; K1 is the distance safety gain, when K1<0.3K min When d≤d min When , it indicates a medium risk state of collision, and in other cases, it indicates a low risk state of collision.

6. The time-delay variable gain robot virtual-reality interactive remote control system according to claim 5, characterized in that: According to the delay caused by the motion command being sent from the master controller to the slave controller for processing and feedback, the motion command mapping ratio is adjusted as follows: Each motion instruction to be sent will be marked with the local timestamp T1 and frame number d1 of the master controller, and sent to the slave controller at a fixed sending cycle. After receiving the motion instruction of this frame, the slave controller controls the slave robot, reads the status of the slave robot, and forms the latest status feedback frame of the slave robot to the master controller. The latest status feedback frame includes the timestamp T1, the frame number d1, and the timestamp T2 after T1 is superimposed on the time Δt spent on the operation processing. When the master controller receives the latest state feedback frame, it records the arrival time of the feedback frame as timestamp T3, performs timestamp calculation for a complete motion instruction cycle, uses T3-T1 to obtain two network transmission delays, uses T2-T1 to obtain the slave processing delay, and uses the following formula to calculate the delay safety gain K2: Among them, K M is the maximum gain, K m is the minimum gain, T max is the expected maximum delay, if T max If no return frame is received within the time, T max As a time estimate; When K2 is larger, the slave-end following ratio increases and the overall operating speed increases. When K2 is smaller, the slave-end following ratio decreases and the overall operating speed decreases.

7. The time-delay variable gain robot virtual-reality interactive remote control system according to claim 6, characterized in that: The operation safety assessment module uses the gain K=K1K2 to adjust the mapping ratio of the motion instruction, and obtains the operation instruction to send to the slave controller; The gain K is used to calculate the end position increment of the master robot arm to change the motion command of the slave robot arm: in, is the original posture transformation relationship from the end of the main arm to the base of the main arm at time t, Calculate the original posture transformation relationship from the end of the main arm to the base of the main arm at time t-Δt in the previous frame. The calculated posture transformation relationship from the end of the main arm to the base of the main arm at the calculation time t-Δt in the previous frame; And used to calculate the control increment of the remote control stick to change the movement instruction of the slave mobile platform: v x =Kk x Δx,ω z =Kk y Δy。 8. The time-delay variable gain robot virtual-reality interactive remote control system according to claim 1, characterized in that: The environmental spatial structure information collected by the environmental sensor of the slave robot is added to the virtual simulation environment and expressed in the form of point cloud. In order to increase the control accuracy of the slave robot in an obstacle environment, the point cloud data generated by the environmental sensor detection and scanning of the real slave robot is removed in the virtual simulation environment, and only the environmental point cloud is retained.

9. The time-delay variable gain robot virtual-reality interactive remote control system according to claim 1, characterized in that: The slave robot virtual simulation model uses a robot joint link model driven by the slave robot joint position data.

10. A time-delay variable gain robot virtual-reality interactive remote control method based on the system according to any one of claims 1 to 9, characterized in that: The following steps are involved: Collect wrist, finger, and foot movement data; The collected motion data is mapped to obtain the motion instructions of the slave robot, and the motion instructions of a certain period of time are selected and recorded according to the different stepping states of the pedaling action to obtain the corresponding motion instruction sequence; Receive a motion command sequence, and adjust the mapping ratio of the motion command to obtain an operation command according to the distance between the slave robot virtual simulation model and the space obstacle of the virtual simulation environment, and the time delay caused by the motion command being sent from the master controller to the slave controller for processing and feedback; Drive the slave robot to perform corresponding actions according to the operation instructions and provide action status feedback.

Images