A three-arm space robot multi-modal shared teleoperation system and method

Abstract

The application discloses a three-arm space robot multi-modal shared teleoperation system and method, which comprises at least a master teleoperation system, a communication module and a slave robot system, the master teleoperation system comprises at least two force feedback hand controllers, a microphone array and upper computer software, and the slave robot system comprises two end work arms equipped with grippers, an observation arm with a double-eye camera at the end, a vision unit, a force sensor and lower computer software; an operator controls two work arms of the extravehicular robot to perform a task, and controls the observation arm to obtain a better local visual angle; the method fuses the multi-modal teleoperation control method comprising pose control, voice control and force control with the robot autonomous control through a shared control algorithm, can realize the position, posture and contact force of the robot controlled by the operator and the robot autonomously performing other relatively simple tasks according to the demand of the operator, reduces the operation burden of the operator and improves the control efficiency.

Description

Technical Field

[0001] This invention belongs to the technical field of space teleoperation control, and mainly relates to a multimodal shared teleoperation system and method for a three-armed space robot. Background Technology

[0002] In recent years, my country's space program has developed rapidly. With the Chinese space station entering a stable operational phase, astronauts are increasingly performing extravehicular activities (EVAs) to maintain equipment and replace consumables and fuel. However, EVA missions face challenges such as long preparation times, frequent operations, high workload, and wide operational areas, and also involve certain safety risks. Therefore, it is necessary to develop space robots to replace astronauts in performing EVA missions. However, EVA missions require complex and precise operations, such as on-orbit assembly, on-orbit installation, and solar cell replacement. Traditional single-arm robots cannot meet these requirements. Therefore, space robots with dual or even multiple arms are needed to coordinate and perform tasks.

[0003] Furthermore, due to the complex lighting environment and diverse task requirements of extravehicular activities (EVAs), it is difficult to develop a fully autonomous space robot system suitable for the needs of EVA missions based on current levels of automation and sensor technology. The main current solution is to develop teleoperated robot systems directly controlled by operators to perform complex space tasks. This places high demands on the operator's experience, especially when controlling robot systems with two or more robotic arms, which puts a significant burden on the operator's physical and mental well-being. Operators need to multitask, simultaneously controlling the position, attitude, and end-effector contact forces of multiple robotic arms while constantly monitoring various feedback information from the robot. Therefore, human-robot shared teleoperation has become an effective method to solve these problems. Summary of the Invention

[0004] This invention addresses the shortcomings of existing technologies by disclosing a multimodal shared teleoperation system and method for a three-armed space robot. It includes at least a master teleoperation system, a communication module, and a slave robot system. The master teleoperation system comprises at least two force feedback hand controllers located on the left and right arms, a microphone array, and host computer software. The slave robot system includes two working arms equipped with grippers at their ends, an observation arm with a binocular camera at its end, a vision unit, force sensors, and lower-level computer software. The operator inside the spacecraft uses the two force feedback hand controllers to control the two working arms of the robot outside the spacecraft to perform tasks, using voice commands. The command-controlled observation arm obtains a better local perspective. This method integrates multimodal teleoperation control methods, including pose control, voice control, and force control, with robot autonomous control through a shared control algorithm. It can dynamically allocate the dimensions and weights of the output commands of each control method within the robot arm's joint space according to the operator's needs. This enables human-robot joint control of the robot arm's position, posture, and contact force, allowing the operator to focus on controlling the robot arm to perform important and challenging tasks, while the robot autonomously performs other simpler tasks. This reduces the operator's workload and improves control efficiency. The master-end teleoperation system provides multimodal feedback including force, vision, and hearing, which can improve the operator's sense of presence, control accuracy, and comfort in unstructured environments.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is: a multimodal shared teleoperation system for a three-armed space robot, comprising at least a master teleoperation system, a communication module, and a slave robot system;

[0006] The master remote operating system includes at least two force feedback hand controllers located in the left and right hands, a microphone array, and host computer software. The host computer software includes at least virtual simulation human-computer interaction software, a hand controller driver module, and a voice recognition module.

[0007] The slave robot system includes a working arm, an observation arm, a force sensor, a vision unit, and lower-level software. Two working arms are provided, each equipped with a gripper at its end. The observation arm has a binocular camera at its end. The lower-level software includes at least a force control algorithm, a pose control algorithm, a target recognition algorithm, an autonomous control algorithm, and a shared control algorithm. The vision unit provides visual information about the surroundings of the slave robot to the master teleoperating system and is used for target recognition in the autonomous control algorithm.

[0008] The communication module is used to construct a short-to-medium range, low-latency wireless local area network to realize wireless communication between the master remote operating system and the slave robot system.

[0009] As an improvement to the present invention, in the master-end remote operating system:

[0010] The force feedback hand controller is used to collect the pose information input by the operator, receive force sensor data, and provide three-dimensional force feedback to the operator.

[0011] The microphone array is used to collect the operator's audio signals.

[0012] The virtual simulation human-computer interaction software includes a real-time rendered 3D robot model, teleoperation mapping parameter adjustment, collision detection and early warning functions, provides feedback information and an interactive interface for the operator, and outputs interactive commands.

[0013] The hand controller driver module is used to calculate the operator's input pose information and output it as an operation command.

[0014] The voice recognition module is used to parse and output the operator's voice commands.

[0015] As an improvement to the present invention, in the lower-level machine software:

[0016] The force control algorithm is used to control the interaction force between the working arm and the environment. Based on the force signal setpoint in the interactive command output by the master remote operating system and the feedback data from the force sensor, it outputs a force control command q. f ;

[0017] The pose control algorithm is used to control the pose of the working arm and the observation arm. It can read the hand controller operation commands, voice commands, and interactive commands output by the master remote operating system, and output the pose control command q. pr ;

[0018] The target recognition algorithm is used to identify target objects in the environment and obtain their pose and contour.

[0019] The autonomous control module performs autonomous path planning based on the target recognition results and generates robot autonomous commands q using a bidirectional random search tree method. a ;

[0020] The shared control algorithm: selects the bit control instruction q according to the interactive instruction. pr Or force control command q f The operator's teleoperation command q h and link it to the robot's autonomous command q a The fusion command q is obtained through fusion. c Based on a dynamic weight allocator, the dimensions and weights of each method output instruction are dynamically allocated in the joint space of the robotic arm through interactive instructions, so as to realize human-machine joint control of the position, posture and contact force of the robotic arm.

[0021] As another improvement of the present invention, the master teleoperation system further includes an incremental control module, a teleoperation mapping parameter adjustment module, and a robotic arm collision detection and early warning module;

[0022] The incremental control module enables and controls the movement of the robotic arm via a button on the force feedback hand controller handle;

[0023] The teleoperation mapping parameter adjustment module is used to adjust the master-slave position ratio mapping parameters and force mapping parameters, and to adjust the movement step size of the robotic arm and the magnitude of the feedback force provided by the force feedback hand controller;

[0024] The robotic arm collision detection and early warning module is used to detect the risk of collisions between arms or between the arm and the robot platform during the movement of the robotic arm, and to issue an early warning.

[0025] As another improvement of the present invention, when the operator controls the slave robot with two working arms and one observation arm through the master remote control operating system, the slave computer outputs control commands for three methods—pose control, force control, and autonomous control—to the shared control algorithm to realize the control of the robotic arm, wherein:

[0026] When using pose control, the instructions output by the master teleoperation system are read in real time, and the instructions are parsed into the Cartesian target pose x of the robotic arm end effector through master-slave operation space mapping. d Then obtain the joint angle q of the robotic arm at this time. t The Cartesian pose x of the robotic arm's end effector at this moment is obtained by solving forward kinematics. t And obtain the error term e(t) = x d -x t The error term e(t) is input into the PID controller to iterate and obtain the pose u(t). The iterated pose u(t) is then used to obtain the new robot arm joint angle q through inverse kinematics. t+1 Then, in the next cycle, x is obtained again through forward kinematics. t+1 Compare it with the target pose x d Error calculations are performed and input into the PID controller to form a control closed loop. After N iterations, the position control command q is output. pr ;

[0027] When force control is used, the operator sets the target six-dimensional force signal F of the work arm through the master remote control system. d Read the force data signal F from the force sensor at the end of the boom at this time. t , will F d With F t The error term is calculated and input into the PID controller to obtain the iterative force signal F. e The force control command q is obtained through the dynamic model. fIn the next cycle, the force data signal F from the force sensor will be used again. t+1 With F d Perform error calculation and input it into the PID controller, combined with the Jacobian matrix J of the robotic arm at this time. T (q), similar to pose control, forms a control closed loop and iteratively obtains new force control commands q. f+1 ;

[0028] When using autonomous control, the pose of the robotic arm's end effector is used as the root node of an extended random tree. The target's pose is determined based on the target recognition algorithm's results and used as the root node of a second extended random tree. Both extended random trees are then expanded alternately in both directions with the same step size and random sampling, adding child nodes alternately until the two trees meet, at which point the path planning algorithm converges. Once converged, the effective path is found by backtracking along the root node at the meeting point of the two extended random trees. Inverse kinematics is then performed on each child node along the effective path to obtain the values ​​of the robotic arm's joint angles in space, and the output is the autonomous command q. a .

[0029] As a further improvement of the present invention, the algorithm for shared control among the three methods of pose control, force control, and autonomous control includes the following steps:

[0030] S1: Based on the operation mode set by the operator, the bit control instruction q is selected through interactive instructions. pr Or force control command q f The operator's teleoperation command q h ;

[0031] S2: Based on the target recognition result C i Depending on the type of task, the operator sends the main terminal interaction command to the dynamic weight allocator to calculate the values ​​of each diagonal element of the weight matrix S, thereby realizing the dynamic update of the shared control weights during the task process.

[0032] S3: Based on the calculation result S of the dynamic weight allocator, the operator's teleoperation command q is... h With the robot's autonomous command q a The final fusion command q is obtained through fusion. c .

[0033] As a further improvement of the present invention, the weight matrix S in step S2 is determined by a dynamic weight allocator:

[0034] When the target recognition algorithm does not find the target object, or the target recognition result is C i Below the recognition threshold C LAt this time, the dynamic weight allocator sets S to the identity matrix. The operator then uses a hand controller or keyboard to interactively control the pose and force of the working arm and observes the pose of the arm through voice control.

[0035] When the target recognition result C i Greater than or equal to threshold C L At that time, based on the type of task, the operator sets the interactive instructions in the main remote operating system, and the dynamic weight allocator calculates the value of S, so as to realize the human-machine joint control of the position, posture and contact force of the robotic arm;

[0036] When the target recognition result C i Greater than or equal to threshold C L When S is set to a zero matrix, the robot can fully autonomously control the pose and force of the robotic arm.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] (1) Compared with traditional single-arm robots, three-arm robots have the advantages of higher degree of freedom, larger workspace, and more flexible multi-arm collaborative operation, which can meet the needs of extravehicular activity tasks. In addition, the observation arm can provide a better local perspective, which is convenient for operators to perform precise operations.

[0039] (2) By using a hybrid control strategy, the operator is allowed to directly operate the robot to exercise its judgment and decision-making ability, while ensuring that the robot has a certain degree of autonomy and can assist the operator in completing complex extravehicular activity tasks.

[0040] (3) By integrating multimodal teleoperation control methods including pose control, voice control, and force control with robot autonomous control through shared control algorithms, the output commands of each control method can be dynamically allocated to the dimensions and weights of the robot arm joint space according to the operator's needs. This enables human and machine to jointly control the position, posture, and contact force of the robot arm, allowing the operator to focus on controlling the robot arm to perform important and difficult tasks, while the robot autonomously performs other simpler tasks, reducing the operator's workload and improving control efficiency.

[0041] (4) The master remote control system provides multimodal feedback including force, vision and hearing, which can improve the operator's sense of presence, control accuracy and comfort in unstructured environments;

[0042] (5) Adding human-computer interaction functions that meet the actual application requirements, such as incremental control module, remote operation mapping parameter adjustment module, and robotic arm collision detection and early warning module, to the system design can improve the safety and practicality of the system. Attached Figure Description

[0043] Figure 1This is a structural block diagram of a multimodal shared teleoperation system for a three-armed space robot according to the present invention;

[0044] Figure 2 This is a schematic diagram of the mechanical structure of a three-armed space robot according to Embodiment 4 of the present invention;

[0045] Figure 3 This is a schematic diagram of the force feedback hand controller in Embodiment 4 of the present invention;

[0046] Figure 4 This is a schematic diagram of the main-end virtual simulation human-computer interaction software interface in Embodiment 4 of the present invention;

[0047] In the diagram: 1-Left working arm; 2-Structured light camera; 3-Robot platform; 4-Right working arm; 5-Connecting bearing; 6-Observation arm; 7-Binocular camera; 8-Right working arm collision warning; 9-Teleoperation mapping parameter adjustment panel; 10-Left working arm collision warning; 11-Main menu panel; 12-Observation arm collision alarm; 13-Voice command display panel; 14-Important information display interface for the robotic arm; 15-Robot platform collision alarm; 16-Emergency stop button; 17-Force feedback hand controller handle; 18-Force feedback hand controller first button; 19-Force feedback hand controller second button. Detailed Implementation

[0048] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0049] Example 1

[0050] A multimodal shared teleoperation system for a three-armed space robot, such as Figure 1 As shown, it includes at least a master remote operating system, a communication module, and a slave robot system; the communication module consists of a router and master-slave communication software, which construct a short-to-medium range, low-latency wireless local area network to realize wireless communication between the master and slave ends.

[0051] The main remote control system includes two force feedback hand controllers (left and right hands), a microphone array, and host computer software. The force feedback hand controllers are Geomagic Touch controllers, used to collect the operator's pose information in Cartesian space, receive force sensor data, and provide three-dimensional force feedback to the operator. The microphone array is used to collect the operator's audio signal and has noise reduction and echo cancellation functions. The host computer software includes virtual simulation human-computer interaction software, a hand controller driver module, and a speech recognition module.

[0052] The virtual simulation human-computer interaction software is developed using Unity and includes a real-time rendered 3D robot model, teleoperation mapping parameter adjustment, collision detection and early warning functions. It can provide operators with real-time multimodal feedback information including vision, touch, and hearing, as well as a rich graphical interactive interface, and output interactive commands. The hand controller driver module is used to calculate the operator's input pose information and output it as operation commands, and has incremental control functions. The speech recognition module is a trained offline neural network used to parse and output the operator's speech commands.

[0053] The slave robot system includes two end-effectors equipped with tools, an observation arm with a binocular camera at its end, force sensors, a vision unit, and lower-level software. All robotic arms are six-degree-of-freedom (DOF) robotic arms. The two end-effectors are mounted on the robot's shoulders, with grippers at their ends for performing tasks. The observation arm is mounted on the robot's waist, with a binocular camera at its end, providing a local view beyond the operator's vision unit. Force sensors, six-dimensional force sensors, are mounted on the wrists of the two end-effectors to collect and feedback force information generated during contact between the end-effector tools and the environment, as well as force information feedback in the robotic arm's force control function. The vision unit consists of a structured light camera mounted on the robot's head and a binocular camera mounted on the end of the observation arm, providing visual information about the robot's surroundings to the master end and for target recognition in the autonomous control algorithm. The lower-level software includes force control algorithms, pose control algorithms, target recognition algorithms, autonomous control algorithms, and shared control algorithms.

[0054] The force control algorithm is used to control the forces acting between the boom and the environment. Based on the force signal setpoint in the interactive command output by the master teleoperation system and the feedback data from the force sensor, it outputs a force control command q. f ;

[0055] The pose control algorithm is used to control the pose of the working arm and the observation arm. It can read the hand controller operation commands, voice commands, and interactive commands output by the master remote operating system, and output the pose control command q. pr ;

[0056] The target recognition algorithm is used to identify target objects in the environment and obtain their pose and contour. First, based on the point cloud of RGB image texture, the viewpoints of the structured light camera on the robot's head and the binocular cameras on the observation arm are stitched together. Next, the DBSCAN algorithm is used for cluster analysis to retain point cloud data with similar densities. Then, the DIC algorithm is used for image feature point matching to obtain the point cloud contour and pose of the target object. Finally, the recognition results are processed. i Similarity detection is performed between the offline trained template and the test image to determine whether the target object has been detected;

[0057] The autonomous control module performs autonomous path planning based on the target recognition results and generates autonomous robot commands q using a bidirectional random search tree method. a ;

[0058] The shared control algorithm selects the bit control instruction q based on the interactive instructions. pr Or force control command q f The operator's teleoperation command q h and link it to the robot's autonomous command q a The final fusion command q is obtained through fusion. c The dynamic weight allocator can dynamically allocate the dimensions and weights of each method output instruction in the joint space of the robotic arm according to the operator's needs through interactive commands, so as to realize human-machine joint control of the position, posture and contact force of the robotic arm.

[0059] Using this system, operators can use two force feedback hand controllers inside the spacecraft cabin to control the two working arms of the external robot to perform tasks, and use voice commands to control the observation arm to obtain a better local perspective. At the same time, by using a shared control algorithm, the system integrates multimodal teleoperation control methods, including pose control, voice control, and force control, with the robot's autonomous control, reducing the operator's workload and improving control efficiency.

[0060] Example 2

[0061] The difference between this embodiment and embodiment 1 is that the master teleoperation system also includes an incremental control module, a teleoperation mapping parameter adjustment module, and a robotic arm collision detection and early warning module.

[0062] In the incremental control module, the force feedback controller's pose information is only sent to the controller's drive module when the operator presses the button on the handle; it is not sent when the button is released. When the master teleoperation system receives the button release signal, it records the pose information of the robotic arm's end effector's task space. At this time, the operator can move the handle of the force feedback controller to a suitable position, but the robotic arm remains stationary as it does not receive a signal. When the operator presses the button again, the robotic arm continues to move according to the operator's control. Therefore, the incremental control module, through a mouse-like interaction, allows the robotic arm's end effector to maintain its pose when the operator releases the button on the force feedback controller's handle. The operator can then move the handle to a suitable position and press the button again to continue controlling the robotic arm's movement. This addresses the problem of limited control range caused by a large difference in the workspace range between the master force feedback controller and the slave robotic arm. Furthermore, when the operator feels uncomfortable gripping the force feedback controller, they can quickly adjust their grip, improving the operator's control accuracy and comfort.

[0063] In the teleoperation mapping parameter adjustment module, due to the high difficulty of space operations, the robotic arm's posture control typically requires both rapid movement over a large range and precise operation within a small area, while also needing the ability to flexibly adjust the magnitude of force feedback. Therefore, the module supports real-time adjustment of position ratio mapping parameters and force mapping parameters by the astronauts inside the cabin. The position ratio mapping parameter is the ratio of the movement scale of the force feedback hand controller to the movement scale of the robotic arm. A larger parameter indicates a larger range of movement for the robotic arm; when the parameter is 0, the robotic arm will not move. The force mapping parameter adjusts the magnitude of the feedback force provided by the hand controller when the robotic arm's end effector contacts the environment. A larger parameter indicates a stronger feedback force provided by the hand controller; when the parameter is 0, the force feedback magnitude will be 0.

[0064] The principle and interaction method of the robotic arm collision detection and early warning module are as follows:

[0065] The robotic arm collision detection and early warning module is developed based on the Unity collision mechanism. It can detect the risk of collisions between arms and between the arm and the robot platform in real time during the robotic arm's movement. It can issue early warnings through multimodal feedback methods such as auditory, visual, and repulsive force provided by the hand controller. The collision warning is divided into two stages: early warning before the collision and alarm during the collision.

[0066] The pre-collision warning is as follows: In the virtual simulation human-computer interaction software, the outline of the part of the robotic arm that is about to collide is highlighted in yellow, a warning sound is played, the collision indicator light turns yellow, and the collision status is displayed as "too close." Furthermore, the hand controller will provide damping force in the direction of the impending collision to remind the operator not to continue moving the handle in that direction, thus prompting the operator to move the handle in the opposite direction to avoid the collision.

[0067] The collision alarm is triggered as follows: In the virtual simulation human-computer interaction software, the outline of the part of the robotic arm that has collided is highlighted in red, a collision alarm sound plays, the collision indicator light turns red, and the collision status is displayed as "collision has occurred." Furthermore, the hand controller will provide feedback of a strong repulsive force to remind the operator to quickly move the handle in the opposite direction.

[0068] Incorporating human-computer interaction functions that meet practical application needs, such as incremental control modules, remote operation mapping parameter adjustment modules, and robotic arm collision detection and early warning modules, into the system design can improve the system's safety and practicality.

[0069] Example 3

[0070] A multimodal shared teleoperation method for a three-armed space robot is provided, using the system described in Example 1. The lower-level computer of the three-armed space robot multimodal teleoperation system integrates three control methods: pose control, force control, and autonomous control. The pose control includes three modes: hand controller control, voice control, and keyboard interaction control. All of the above multimodal teleoperation control methods can be combined with autonomous control to achieve multimodal shared teleoperation.

[0071] The principle of pose control is as follows:

[0072] Step 1: The system is powered on and initializes the parameters of each robotic arm, establishes IP communication, and obtains the current pose information and kinematic parameters of each robotic arm. The base of each robotic arm is fixed as the task space coordinate system, and a forward kinematic pose relationship transformation is performed based on this. The Lagrangian method is used to establish the dynamic model of each robotic arm. Since the space robot is in a weightless environment, the gravity term is omitted, and nonlinear disturbance terms such as joint motor friction are simplified. The calculation formula is as follows:

[0073]

[0074] Step 2: Real-time reading of hand controller operation commands, voice commands, and interactive commands output by the master remote operating system. First, these commands are parsed into the Cartesian target pose x of the robotic arm end effector through master-slave operation space mapping. d Next, obtain the joint angle q of the robotic arm at this time. t The Cartesian pose x of the robotic arm's end effector at this moment is obtained by solving forward kinematics. t Finally, we obtain the error term e(t) = x d -x t ;

[0075] Step 3: Input e(t) into the PID controller to iterate and obtain the pose u(t), the calculation formula of which is as follows:

[0076]

[0077] The new robot arm joint angles q are obtained by inverse kinematics solution of the iterative pose u(t). t+1 Then, in the next cycle, x is obtained again through forward kinematics. t+1 Compare it with the target pose x d Error calculations are performed and input into the PID controller to form a control closed loop. After N iterations, the position control command q is output. pr This greatly improves the control precision of the robotic arm, and at the same time, it can improve the system performance such as steady-state error, response speed, and overshoot by adjusting the number of iterations and various PID parameters.

[0078] Regarding the force control aspect, the principle is as follows:

[0079] Step 1: Perform the same initialization as Step 1 for pose control;

[0080] Step 2: The operator sets the target six-dimensional force signal F of the work arm through the master remote control system. d Simultaneously, the force data signal F from the force sensor at the end of the boom is read. t , will F d With F t The error term is calculated and input into the PID controller to obtain the iterative force signal F. e The force control command q is obtained through the dynamic model. f ;

[0081] Step 3: In the next loop, the force data signal F from the force sensor will be processed again. t+1 With F d Perform error calculation and input it into the PID controller, combined with the Jacobian matrix J of the robotic arm at this time. T (q), similar to pose control, forms a control closed loop and iteratively obtains new force control commands q. f+1 This enables precise force control of the working arm.

[0082] The specific principles of autonomous control are as follows:

[0083] Step 1: Take the pose of the end effector of the robotic arm at this time as the root node of an extended random tree, determine the pose of the target based on the recognition result of the target recognition algorithm, and take it as the root node of a second extended random tree;

[0084] Step 2: Expand the two extended random trees alternately in both directions with the same step size and random sampling, adding child nodes alternately until the two trees meet. Then the path planning algorithm converges.

[0085] Step 3: Once the algorithm converges, backtracking along the root node at the point where the two extended random trees meet will find the effective path. Perform inverse kinematics on each child node along the effective path to obtain the joint angle values ​​of the robotic arm in space, and output the autonomous command q. a .

[0086] The calculation formula for the shared control algorithm is as follows:

[0087] q c =Sq h +(IS)q a

[0088] In the formula, S = diag[s1,s2,s3,s4,s5,s6] T ,s i∈[0,1], the S matrix is ​​a six-dimensional diagonal weight matrix, representing the dimension and weight of the autonomous control commands mapped in the joint space of the robotic arm, I is a 6-dimensional identity matrix, q h q a q c Both are 6×1 matrices, q c Indicates a fusion command;

[0089] The value of S can be determined by the dynamic weight allocator:

[0090] 1) When the target recognition algorithm does not find the target object, or the target recognition result is C i Below the recognition threshold C L At this time, the dynamic weight allocator sets S to the identity matrix. At this time, the operator relies entirely on the hand controller or keyboard to control the position and force of the working arm and observe the position of the arm through voice control.

[0091] 2) When the target recognition result C i Greater than or equal to threshold C L At the same time, based on the type of task, the interactive commands set by the operator in the main remote operating system are used to calculate the value of S by the dynamic weight allocator, so as to realize the joint human-machine control of the position, posture and contact force of the robotic arm. For example, the operator can use voice to control the position of the observation arm, and the robot can autonomously control the posture of the observation arm to automatically align it with the target object to obtain a better local perspective. Or the operator can control the position and posture of the working arm, and the robot can autonomously control the force applied to the environment by the end of the working arm.

[0092] 3) When the target recognition result C i Greater than or equal to threshold C L When S is set to a zero matrix, the robot can fully autonomously control the pose and force of the robotic arm;

[0093] The specific implementation steps of the shared control algorithm are as follows:

[0094] Step 1: Before performing the actual operation, select the position control command q through interactive instructions according to the operation mode set by the operator. pr Or force control command q f The operator's teleoperation command q h ;

[0095] Step 2: Based on the target recognition result C i Depending on the type of task, the operator sends the main terminal interaction command to the dynamic weight allocator to calculate the values ​​of each diagonal element of the weight matrix S, thereby realizing the dynamic update of the shared control weights during the task process.

[0096] Step 3: Based on the calculation result S of the dynamic weight allocator, assign the operator's teleoperation command q...h With the robot's autonomous command q a The final fusion command q is obtained through fusion. c .

[0097] Example 4

[0098] This embodiment is a practical application example of the system and method of the present invention, a multimodal shared teleoperation system for a three-armed space robot, wherein the structure of the three-armed space robot is as follows: Figure 2 As shown, the three-armed space robot is fixed to the large robotic arm outside the space station cabin via the connecting bearing 5. Its power supply line is connected to the robot platform 3 via the connecting bearing 5, and its control circuit and communication module are installed inside the robot platform 3. Figure 3 This is a schematic diagram of the force feedback hand controller.

[0099] Inside the spacecraft cabin, the operator uses two force feedback hand controllers 17 to control the two working arms 1 and 4 of the external robot to perform tasks. The first button 18 on the force feedback hand controller controls the opening and closing of the grippers on working arms 1 and 4. A microphone array is used to collect the operator's voice signals, which are then used to control the observation arm 6 via voice commands. The binocular camera 7 mounted at the end of the observation arm 6 provides a better local perspective. The operator also acquires visual information about the robot's surroundings through the structured light camera 2 mounted on the head of the robot platform 3 and the binocular camera 7 mounted at the end of the observation arm 6. This information is transmitted via a communication module to the main virtual simulation human-machine interaction software and used for target recognition in the autonomous control module.

[0100] Incremental control implementation: When the operator presses the second button 19 of the force feedback hand controller, the position and posture information of the force feedback hand controller handle 17 will be sent to the hand controller drive module. When the button 19 is released, no information will be sent. When the master remote control system receives the signal that the button 19 has been released, it records the position and posture information of the task space at the end of the working arms 1 and 4. At this time, the operator can move the force feedback hand controller handle 17 to a suitable position, while the working arms 1 and 4 do not receive a signal and remain stationary. When the operator presses the button 19, the working arms 1 and 4 continue to move according to the operator's control.

[0101] In this invention, the virtual simulation human-computer interaction software is developed using Unity, such as... Figure 4 As shown, it includes a real-time rendered 3D robot model, teleoperation mapping parameter adjustment, collision detection and early warning functions, and can provide operators with real-time multimodal feedback information including vision, touch and hearing, as well as a rich graphical interactive interface, and output interactive commands.

[0102] The operator can adjust the position ratio mapping parameter and force mapping parameter in real time through the remote mapping parameter adjustment panel 9 on the interactive interface. The position ratio mapping parameter is the ratio of the motion scale of the force feedback hand controller handle 17 to the motion scale of the robotic arm. The larger the parameter, the larger the scale of movement of the robotic arm. When the parameter is 0, the robotic arm will not move. The force mapping parameter can adjust the magnitude of the feedback force provided by the hand controller when the end of the robotic arm comes into contact with the environment. The larger the parameter, the stronger the feedback force provided by the hand controller. When the parameter is 0, the force feedback magnitude will be 0.

[0103] In practical operation, the operator uses the various sub-menus in the main menu panel 11 to complete operations such as setting the robotic arm control mode, setting shared operation weights, setting force control target values, reading robotic arm parameters, and controlling the robotic arm pose via keyboard interaction. The interactive commands are sent to the slave-end pose control algorithm, force control algorithm, and shared control algorithm through communication mode, thereby realizing multimodal shared teleoperation of the three-armed space robot. Simultaneously, the operator can obtain the real-time position and status of each arm of the robot from the robotic arm important information display interface 14 in the virtual simulation human-computer interaction software. Using the robotic arm collision detection and early warning module developed based on the Unity collision body mechanism, the operator can detect the risk of collisions between arms and between the arm and the robot platform during the robotic arm's movement in real time. Early warnings can be issued through multimodal feedback methods such as auditory, visual, and repulsive force provided by the hand controller. The collision warning is divided into two stages: warning before a collision and alarm during a collision. This prevents collisions during the robotic arm's operation and ensures the safety of the system. Figure 3 As shown, the end of the robot's right working arm 4 is about to collide with the end of the left working arm 1. The outline of the parts about to collide is highlighted: right working arm collision warning 8, left working arm collision warning 10. At the same time, the human-machine interaction software plays warning music, and the collision indicator lights for the left and right arms on the robot arm's important information display interface 14 turn yellow, indicating that the collision status is "too close". Furthermore, the hand controller will provide damping force in the direction of the impending collision to remind the operator not to continue moving the handle in that direction, thus prompting the operator to move the handle in the opposite direction to avoid the collision.

[0104] In the event of a collision during operation, the observation arm 6 and the robot platform 3 will be notified of the collision, with the outline of the collided area highlighted: observation arm collision alarm 12, robot platform collision alarm 15. Simultaneously, the human-machine interface software will play a collision alarm sound, and the observation arm collision indicator light on the robotic arm's important information display interface 14 will turn red, indicating a collision has occurred. Furthermore, the hand controller will provide strong feedback of a repulsive force to alert the operator to quickly move the handle in the opposite direction. The operator can press the emergency stop button 16 to stop data communication between the master and slave ends and terminate all movement commands of the robotic arm, preventing further collision damage.

[0105] The specific implementation of multimodal shared teleoperation includes the following steps:

[0106] Step 1: Before actual operation, the operator sets the operating mode of both working arms through the main menu 11 of the virtual simulation human-machine interaction software, and sends interactive commands to the shared control. If pose control is selected, the position control command q is output. pr The operator's teleoperation command q h If force control is selected, the force control command q will be output. f The operator's teleoperation command q h ;

[0107] Step 2: Based on the video feedback from the visual unit and the target recognition result C i In addition to the type of task, the operator sends the main terminal interaction command to the dynamic weight allocator to calculate the values ​​of each diagonal element of the weight matrix S, thereby realizing the dynamic update of the shared control weights during the task process. The update steps are as follows:

[0108] 1) When the target recognition algorithm does not find the target object, or the target recognition result is C i Below the recognition threshold C L When the dynamic weight allocator sets S to the identity matrix, the operator relies entirely on the hand controller or keyboard to control the position and force of the working arm. The position of the arm 6 is observed through voice control. The latest voice commands and the completion status of the commands will be displayed on the voice command display panel 13.

[0109] 2) When the target recognition result C i Greater than or equal to threshold C L At the same time, according to the type of task, the interactive instructions set by the operator in the main remote operating system are used to calculate the value of S by the dynamic weight allocator, so as to realize the joint human-machine control of the position, posture and contact force of the robotic arm. For example, the operator can use voice to control the position of the observation arm 6, and the robot can autonomously control the posture of the observation arm 6 to automatically align it with the target object to obtain a better local perspective. Or the operator can control the position and posture of the working arm, and the robot can autonomously control the force applied to the environment by the end of the working arm.

[0110] 3) When the target recognition result C i Greater than or equal to threshold C L When S is set to a zero matrix, the robot can fully autonomously control the pose and force of the robotic arm;

[0111] Step 3: Based on the calculation result S of the dynamic weight allocator, assign the operator's teleoperation command q... h With the robot's autonomous command q a The final fusion command q is obtained through fusion. c Assign teleoperation command qh With autonomous command q a The dimensions and weights mapped onto the joint space of the robotic arm, and the formulas for achieving human-machine joint control of the robotic arm's position, posture, and contact force fusion are as follows:

[0112] q c =Sq h +(IS)q a .

[0113] In summary, the method of this invention utilizes two working arms of a three-armed space robot, making operation more convenient and flexible. Voice control of the observation arm 6 provides a better local perspective. By employing a hybrid control strategy, the method allows the operator to directly operate the robot to exercise their judgment and decision-making abilities while ensuring that the robot has a certain degree of autonomy. This enables the robot to assist the operator in completing complex extravehicular activity tasks, allowing the operator to focus on controlling the robotic arm to perform important and delicate operations, while the robot autonomously completes other simpler tasks. This reduces the operator's workload and improves the control accuracy and efficiency of remote operation.

[0114] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0115] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A multimodal shared teleoperation system for a three-armed space robot, characterized in that: It includes at least a master remote operating system, a communication module, and a slave robot system; The master remote operating system includes at least two force feedback hand controllers located in the left and right hands, a microphone array, and host computer software. The host computer software includes at least virtual simulation human-computer interaction software, a hand controller driver module, and a voice recognition module. The slave robot system includes a working arm, an observation arm, a force sensor, a vision unit, and lower-level software. Two working arms are provided, each equipped with a gripper at its end. The observation arm has a binocular camera at its end. The lower-level software includes at least a force control algorithm, a pose control algorithm, a target recognition algorithm, an autonomous control algorithm, and a shared control algorithm. The vision unit provides visual information about the surroundings of the slave robot to the master teleoperating system and is used for target recognition in the autonomous control algorithm. The communication module is used to construct a short-to-medium range, low-latency wireless local area network to realize wireless communication between the master remote operating system and the slave robot system.

2. The multimodal shared teleoperation system for a three-armed space robot as described in claim 1, characterized in that: In the master-end teleoperation system: The force feedback hand controller is used to collect the pose information input by the operator, receive force sensor data, and provide three-dimensional force feedback to the operator. The microphone array is used to collect the operator's audio signals. The virtual simulation human-computer interaction software includes a real-time rendered 3D robot model, teleoperation mapping parameter adjustment, collision detection and early warning functions, provides feedback information and an interactive interface for the operator, and outputs interactive commands. The hand controller driver module is used to calculate the operator's input pose information and output it as an operation command. The voice recognition module is used to parse and output the operator's voice commands.

3. The multimodal shared teleoperation system for a three-armed space robot as described in claim 1, characterized in that: In the lower-level machine software: The force control algorithm is used to control the interaction force between the working arm and the environment. Based on the force signal setpoint in the interactive command output by the master remote operating system and the feedback data from the force sensor, it outputs force control commands. ; The pose control algorithm is used to control the pose of the working arm and the observation arm. It can read the hand controller operation commands, voice commands, and interactive commands output by the master remote operating system and output position control commands. ; The target recognition algorithm is used to identify target objects in the environment and obtain their pose and contour. The autonomous control module: based on the target recognition result Autonomous path planning is performed, and bidirectional random search tree method is used to generate autonomous robot commands. ; The shared control algorithm: selects position control instructions based on interactive instructions. or force control command Remote operation commands as the operator and remote operation commands With the robot's autonomous commands Fusion obtains fusion command Based on a dynamic weight allocator, the output commands are dynamically allocated in the joint space of the robotic arm through interactive instructions, thereby enabling human-machine joint control of the robotic arm's position, posture, and contact force.

4. A multimodal shared teleoperation system for a three-armed space robot as described in claim 2 or 3, characterized in that: The master teleoperation system also includes an incremental control module, a teleoperation mapping parameter adjustment module, and a robotic arm collision detection and early warning module. The incremental control module enables and controls the movement of the robotic arm via a button on the force feedback hand controller handle; The teleoperation mapping parameter adjustment module is used to adjust the master-slave position ratio mapping parameters and force mapping parameters, and to adjust the movement step size of the robotic arm and the magnitude of the feedback force provided by the force feedback hand controller; The robotic arm collision detection and early warning module is used to detect the risk of collisions between arms or between the arm and the robot platform during the movement of the robotic arm, and to issue an early warning.

5. A multimodal shared teleoperation method for a three-armed space robot using the system described in claim 1, characterized in that: When the operator controls the slave robot, which has two working arms and one observation arm, through the master teleoperation system, the slave computer outputs control commands for three methods—pose control, force control, and autonomous control—to the shared control algorithm to achieve control of the robotic arms. Specifically: When using pose control, the instructions output by the master teleoperation system are read in real time, and the instructions are parsed into the Cartesian target pose of the robotic arm end effector through master-slave operation space mapping. Then obtain the joint angle of the robotic arm at this time. The Cartesian pose of the robotic arm's end effector at this moment is obtained by solving forward kinematics. And obtain the error term. ;Will The pose is obtained by iterative input into the PID controller. Iterate the pose The new joint angles of the robotic arm are obtained through inverse kinematics. Then, in the next cycle, we obtain the result again through forward kinematics. ,Will With target pose Error calculations are performed and input into the PID controller to form a control closed loop. After the next iteration, output the bit control instruction. ; When force control is used, the operator sets the target six-dimensional force signal of the working arm through the master remote control system. Read the force data signal from the force sensor at the end of the boom at this time. ,Will and The error term is calculated and input into the PID controller to obtain the iterative force signal. Force control commands are obtained through dynamic models. In the next cycle, the force data signal from the force sensor will be used again. and Perform error calculation and input it into the PID controller, taking into account the Jacobian matrix of the robotic arm at this time. Similar to pose control, it forms a control closed loop and iteratively obtains new force control commands. ; When using autonomous control, the pose of the robotic arm's end effector is used as the root node of an extended random tree. The target's pose is determined based on the target recognition algorithm's results and used as the root node of a second extended random tree. Both extended random trees are then expanded alternately in both directions with the same step size and random sampling, adding child nodes alternately until the two trees meet, at which point the path planning algorithm converges. Once converged, the effective path is found by backtracking along the root node at the meeting point of the two extended random trees. Inverse kinematics is then performed on each child node along the effective path to obtain the values ​​of the robotic arm's joint angles in space, and the output is the autonomous command. .

6. A multimodal shared teleoperation method for a three-armed space robot as described in claim 5, characterized in that: Among the three methods of integrating pose control, force control, and autonomous control, the shared control algorithm includes the following steps: S1: Select position control commands via interactive instructions based on the operator's set operation mode. or force control command Remote operation commands as the operator ; S2: Based on the target recognition results Depending on the type of task, the operator sends the main terminal interaction command to the dynamic weight allocator to calculate the weight matrix. The values ​​of each diagonal element are used to dynamically update the shared control weights during the operation process; S3: Based on the calculation results of the dynamic weight allocator, the operator's remote operation commands are... With the robot's autonomous commands The final fusion command is obtained through fusion. .

7. A multimodal shared teleoperation method for a three-armed space robot as described in claim 6, characterized in that: The weight matrix S in step S2 is determined by the dynamic weight allocator: When the target recognition algorithm does not find the target object, or the target recognition result is... Below the recognition threshold At that time, the dynamic weight allocator is set. The unit matrix is ​​used, and the operator uses a hand controller or keyboard to interactively control the position and force of the working arm, and observes the position of the arm through voice control. When the target recognition result Greater than or equal to the recognition threshold At that time, based on the type of task, the interactive commands set by the operator in the main remote operating system are calculated by the dynamic weight allocator. The value of is used to enable human-machine joint control of the position, posture, and contact force of the robotic arm; When the target recognition result Greater than or equal to the recognition threshold At that time, and set When the matrix is ​​zero, the robot can fully autonomously control the pose and force of the robotic arm.

Images