Multi-modal shared teleoperation system and method for three-arm space robot

Abstract

Disclosed in the present invention are a multi-modal shared teleoperation system and method for a three-arm space robot. The system at least comprises a master-side teleoperation system, a communication module and a slave-side robot system, wherein the master-side teleoperation system at least comprises two force feedback hand controllers, a microphone array and upper computer software, and the slave-side robot system comprises two operating arms each equipped with a gripper at the end, an observation arm having a binocular camera mounted at the end, a vision unit, a force sensor and lower computer software. The method comprises: an operator controlling two operating arms of an extravehicular robot to execute a task, and controlling an observation arm to acquire a better local field of view. In the method, a multi-modal teleoperation method comprising pose control, voice control and force control is fused with autonomous control of a robot by means of a shared control algorithm, and thus, human-robot collaborative control over the position, orientation and contact force of a robotic arm can be realized on the basis of the requirements of the operator, and the robot autonomously executes other relatively simple tasks, thereby reducing the operation burden of operators, and improving the control efficiency.

Description

A three-arm space robot multimodal shared teleoperation system and method Technical Field

[0001] The present invention belongs to the technical field of space teleoperation control, and mainly relates to a three-arm space robot multimodal shared teleoperation system and method. Background Art

[0002] my country's space industry has developed rapidly in recent years. As my country's space station enters a stable on-orbit operation phase, astronauts are increasingly performing extravehicular missions, such as maintaining equipment and replacing consumable fuel. However, these missions face challenges such as long preparation times, frequent operations, high workload intensity, and a wide range of operations. Furthermore, these missions present certain safety risks, leading to the need for space robots to replace astronauts in extravehicular missions. However, these missions require complex and sophisticated operations, such as on-orbit assembly, installation, and replacement of solar cells. Traditional single-arm robots are unable to meet these requirements, necessitating the development of space robots with dual or even multiple arms that can coordinate with each other to carry out these tasks.

[0003] Furthermore, due to the complex lighting environment and diverse mission requirements of extravehicular operations (EVO), it is difficult to develop a fully autonomous space robotic system suitable for these tasks based on current automation and sensor technologies. The current main solution is to develop a teleoperated robotic system directly controlled by an operator to perform complex space operations. This places a high demand on the operator's operational experience, especially when controlling a robotic system with two or more robotic arms. This places a significant strain on the operator's physical and mental abilities, requiring the operator to multitask, simultaneously controlling the position, posture, and contact force of multiple robotic arms with the environment, while also maintaining constant attention to the various feedback signals from the robots. Therefore, shared human-robot teleoperation has become an effective solution to these problems.

[0004] Summary of the Invention

[0005] The present invention is aimed at the deficiencies of the existing technology and discloses a three-arm space robot multimodal shared teleoperation system and method, which comprises at least a master-end teleoperation system, a communication module and a slave-end robot system. The master-end teleoperation system comprises at least two force feedback hand controllers located on the left and right hands, a microphone array and a host computer software. The slave-end robot system comprises two working arms equipped with grippers at the ends, an observation arm with a binocular camera installed at the end, a visual unit, a force sensor and a slave computer software. The operator uses two force feedback hand controllers in the spacecraft cabin to control the two working arms of the robot outside the cabin to perform tasks, and uses voice to control the two working arms of the robot outside the cabin to perform tasks. The command-controlled observation arm obtains a better local perspective; this method integrates the multimodal teleoperation control method including posture control, voice control, and force control with the robot's autonomous control through a shared control algorithm. It can dynamically allocate the dimensions and weights of the output command mapping of each control method in the robot arm's joint space according to the operator's needs, thereby achieving joint human-machine control of the robot arm's position, posture, and contact force. This allows the operator to focus on controlling the robot arm to perform important and difficult tasks, while the robot autonomously performs other simpler tasks, reducing the operator's operating burden and improving control efficiency. The master-side teleoperation system provides multimodal feedback including force, vision, and hearing, which can enhance the operator's sense of presence, control accuracy, and comfort in unstructured environments.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is: a three-arm space robot multimodal shared teleoperation system, comprising at least a master teleoperation system, a communication module and a slave robot system;

[0007] The master-end teleoperation system includes at least two force feedback hand controllers located on the left and right hands, a microphone array, and a host computer software, wherein the host computer software includes at least virtual simulation human-computer interaction software, a hand controller driving module, and a speech recognition module;

[0008] The slave robot system includes a working arm, an observation arm, a force sensor, a vision unit, and lower computer software. The working arms are provided with two, each equipped with a gripper at the end; a binocular camera is provided at the end of the observation arm; the lower computer software includes at least a force control algorithm, a posture control algorithm, a target recognition algorithm, an autonomous control algorithm, and a shared control algorithm; the vision unit is used to provide visual information around the slave robot to the master teleoperation system and is used for target recognition in the autonomous control algorithm;

[0009] The communication module is used to build a medium- and short-range low-latency wireless local area network to achieve wireless communication between the master-end teleoperation system and the slave-end robot system.

[0010] As an improvement of the present invention, in the master-side teleoperation system:

[0011] The force feedback hand controller is used to collect posture information input by the operator, receive force sensor data and provide three-dimensional force feedback to the operator;

[0012] The microphone array is used to collect the operator's audio signal;

[0013] The virtual simulation human-computer interaction software includes a real-time rendered three-dimensional robot model, remote operation mapping parameter adjustment, collision detection and warning functions, provides feedback information and an interactive interface for the operator, and outputs interactive instructions;

[0014] The hand controller driving module is used to calculate the operator's input posture information and output it as an operation instruction;

[0015] The speech recognition module is used to analyze and output the operator's speech instructions.

[0016] As an improvement of the present invention, in the lower computer software:

[0017] The force control algorithm is used to control the force between the working arm and the environment, and outputs the force control instruction q according to the force signal setting value in the interactive instruction output by the master end teleoperation system and the feedback data of the force sensor. f ;

[0018] The posture control algorithm is used to control the posture of the working arm and the observation arm, and can read the hand controller operation instructions, voice instructions and interactive instructions output by the master end remote operation system, and output the position control instruction q pr ;

[0019] The target recognition algorithm is used to identify the target object in the environment and obtain its position and outline;

[0020] The autonomous control module performs autonomous path planning based on the target recognition results and uses a bidirectional random search tree method to generate the robot's autonomous instructions q a ;

[0021] The shared control algorithm: selects the position control instruction q according to the interactive instruction pr Or force control instruction q f As the operator's teleoperation command q h and compare it with the robot's autonomous command q a Fusion gets the fusion instruction q c Based on the dynamic weight allocator, the dimensions and weights of the output command mappings of each method in the joint space of the robot arm are dynamically allocated through interactive instructions, so that the human and the machine can jointly control the position, posture and contact force of the robot arm.

[0022] As another improvement of the present invention, the master-side teleoperation system further includes an incremental control module, a teleoperation mapping parameter adjustment module, and a robotic arm collision detection and warning module;

[0023] The incremental control module enables and controls the movement of the robotic arm through the buttons on the force feedback hand controller handle;

[0024] The teleoperation mapping parameter adjustment module is used to adjust the master-slave position ratio mapping parameters and force mapping parameters, and to adjust the motion step length of the robotic arm and the feedback force provided by the force feedback hand controller;

[0025] The robot arm collision detection and warning module is used to detect the risk of collision between arms or between the arm and the robot platform during the movement of the robot arm, and to issue a warning.

[0026] As another improvement of the present invention, when an operator controls a slave robot having two working arms and one observation arm through a master remote operating system, its slave computer outputs control instructions for three methods, namely, posture control, force control, and autonomous control, to a shared control algorithm to realize control of the robot arms, wherein:

[0027] When using posture control, the command output by the master teleoperation system is read in real time, and the command is parsed into the Cartesian target posture x of the end of the manipulator through the master-slave operation space mapping. d , and then obtain the joint angle q of the robot arm at this time t , the Cartesian position x of the end of the robot arm is solved by forward kinematics t , and get the error term e(t)=x d -x t ; Input the error term e(t) into the PID controller to iteratively obtain the posture u(t), and obtain the new robot arm joint angle q by inverse kinematic solution of the iterative posture u(t) t+1 ; Then in the next cycle, we get x again through forward kinematics t+1 , and compare it with the target pose x d Calculate the error and input it into the PID controller to form a closed control loop. After N iterations, the position control instruction q is output. pr ;

[0028] When force control is used, the operator sets the target six-dimensional force signal F of the working arm through the master-side teleoperation system. d , read the force data signal F of the force sensor at the end of the working arm at this time t , F d With F t Make the error term and input it into the PID controller to get the iterative force signal F e , the force control instruction q is obtained through the dynamic model f; In the next cycle, the force data signal F of the force sensor is again t+1 With F d Calculate the error and substitute it into the PID controller, combined with the Jacobian matrix J of the robot at this time T (q), the same as the posture control to form a closed control loop, iteratively obtain the new force control instruction q f+1 ;

[0029] When using autonomous control, the posture of the end of the robot arm is used as the root node of an extended random tree. The posture of the target is determined according to the recognition result of the target recognition algorithm and used as the root node of the second extended random tree. The two extended random trees are alternately extended in both directions with the same step size and random sampling, and child nodes are added alternately until the two trees meet. The path planning algorithm converges. After the algorithm converges, a valid path can be found by backtracking along the root node at the intersection of the two extended random trees. The kinematic inverse solution of a series of child nodes on the valid path is performed to obtain the value of the robot arm joint angle space, and the output is the autonomous command q a .

[0030] As a further improvement of the present invention, the algorithm for shared control among the three methods of integrated posture control, force control and autonomous control includes the following steps:

[0031] S1: According to the operation mode set by the operator, select the position control instruction q through the interactive instruction pr Or force control instruction q f As the operator's teleoperation command q h ;

[0032] S2: According to the target recognition result C i According to the type of the task, the operator sends the master-side interactive instruction to the dynamic weight allocator to calculate the values ​​of each diagonal element of the weight matrix S, so as to realize the dynamic update of the shared control weight during the operation;

[0033] S3: According to the calculation result S of the dynamic weight distributor, the operator's teleoperation instruction q h With the robot's autonomous instructions q a Fusion gets the final fusion instruction q c .

[0034] As a further improvement of the present invention, the weight matrix S in step S2 is determined by a dynamic weight allocator:

[0035] When the target recognition algorithm does not find the target object, or the target recognition result C i Below the recognition threshold C LWhen , the dynamic weight allocator sets S to the unit matrix. At this time, the operator uses the hand controller or keyboard to interactively control the position and force of the working arm, and uses voice control to observe the position of the arm;

[0036] When the target recognition result C i Greater than or equal to threshold C L When the operator sets the interactive instructions in the master teleoperation system according to the type of task, the dynamic weight distributor calculates the value of S, and the position, posture and contact force of the manipulator are controlled by the human and the machine together.

[0037] When the target recognition result C i Greater than or equal to threshold C L When , and S is set to a zero matrix, the robot fully controls the position and force of the manipulator.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] (1) Compared with traditional single-arm robots, three-arm robots have the advantages of higher degrees of freedom, larger workspace, and more flexible multi-arm collaborative operations, which can meet the needs of extravehicular operations. In addition, the observation arm can provide a better local perspective, which facilitates the operator to perform refined operations.

[0040] (2) Using a hybrid control strategy, the operator is allowed to directly operate to exert his judgment and decision-making ability, while ensuring that the robot has a certain degree of autonomy and can assist the operator to complete complex extravehicular operations;

[0041] (3) By integrating the multimodal teleoperation control method including posture control, voice control, and force control with the robot's autonomous control through a shared control algorithm, the output instructions of each control method can be dynamically allocated to the dimensions and weights of the robot's joint space according to the operator's needs, so that the human and the machine can jointly control the position, posture, and contact force of the robot. This allows the operator to focus on controlling the robot to perform important and difficult tasks, while the robot autonomously performs other simpler tasks, reducing the operator's operating burden and improving control efficiency.

[0042] (4) The master-side teleoperation system provides multimodal feedback including force, vision, and hearing, which can improve the operator's sense of presence, control accuracy, and comfort in unstructured environments;

[0043] (5) Adding human-computer interaction functions that meet actual application needs, such as incremental control modules, remote operation mapping parameter adjustment modules, and robotic arm collision detection and warning modules, to the system design can improve the safety and practicality of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] FIG1 is a block diagram of a multimodal shared teleoperation system for a three-arm space robot according to the present invention;

[0045] FIG2 is a schematic diagram of the mechanical structure of a three-arm space robot in Example 4 of the present invention;

[0046] FIG3 is a schematic structural diagram of a force feedback hand controller according to a fourth embodiment of the present invention;

[0047] FIG4 is a schematic diagram of a host-side virtual simulation human-computer interaction software interface in Example 4 of the present invention;

[0048] In the figure: 1-working left arm; 2-structured light camera; 3-robot platform; 4-working right arm; 5-connecting bearing; 6-observation arm; 7-binocular camera; 8-working right arm collision warning; 9-teleoperation mapping parameter adjustment panel; 10-working left arm collision warning; 11-main menu panel; 12-observation arm collision alarm; 13-voice command display panel; 14-robot arm important information display interface; 15-robot platform collision alarm; 16-emergency stop button; 17-force feedback hand controller handle; 18-force feedback hand controller first button; 19-force feedback hand controller second button. DETAILED DESCRIPTION

[0049] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0050] Example 1

[0051] A three-arm space robot multimodal shared teleoperation system, as shown in Figure 1, includes at least a master-end teleoperation system, a communication module, and a slave-end robot system; the communication module is a router and master-slave communication software, which builds a medium- and short-range low-latency wireless local area network to achieve wireless communication between the master and slave ends.

[0052] The master-side teleoperation system includes two force feedback hand controllers for the left and right hands, a microphone array, and a host computer software. The force feedback hand controller uses the Geomagic Touch hand controller, which is used to collect the operator's input Cartesian space posture information, receive force sensor data, and feed back three-dimensional force to the operator; the microphone array is used to collect the operator's audio signal and has the functions of noise reduction and echo elimination; the host computer software includes virtual simulation human-computer interaction software, a hand controller driver module, and a voice recognition module; Among them,

[0053] The virtual simulation human-computer interaction software is developed using Unity and includes a real-time rendered three-dimensional robot model, remote operation mapping parameter adjustment, collision detection and warning functions. It can provide the operator with real-time multi-modal feedback information including vision, touch, and hearing, as well as a rich graphical interactive interface, and output interactive instructions; the hand controller driver module is used to solve the operator's input posture information and output it as operation instructions, and has incremental control functions; the voice recognition module is a trained offline neural network used to parse and output the operator's voice instructions.

[0054] The slave robot system includes two working arms equipped with tools at the end, an observation arm with a binocular camera installed at the end, a force sensor, a vision unit and a lower computer software. All the robotic arms are six-degree-of-freedom robotic arms, of which the two working arms are respectively installed on the shoulders of the robot, and the grippers are installed at the end to perform working tasks; the observation arm is installed at the waist of the robot, and a binocular camera is installed at the end to provide the operator with a local perspective outside the vision unit; the force sensors are respectively installed on the wrists of the two working arms, and six-dimensional force sensors are used to collect and feedback the force information generated during the contact between the tool at the end of the working arm and the environment, as well as the force information feedback in the force control function of the robotic arm; the vision unit consists of a structured light camera installed on the head of the robot and a binocular camera installed at the end of the observation arm, which is used to provide visual information of the robot's surrounding environment to the master end and is used for target recognition of the autonomous control algorithm; the lower computer software includes force control algorithm, posture control algorithm, target recognition algorithm, autonomous control algorithm, and shared control algorithm; wherein,

[0055] The force control algorithm is used to control the force between the working arm and the environment. It outputs the force control instruction q according to the force signal setting value in the interactive instruction output by the master teleoperation system and the feedback data of the force sensor. f ;

[0056] The posture control algorithm is used to control the posture of the working arm and the observation arm. It can read the hand controller operation instructions, voice instructions and interactive instructions output by the master teleoperation system, and output the position control instructions q pr ;

[0057] The target recognition algorithm is used to identify the target object in the environment and obtain its posture and outline. First, the perspective of the structured light camera on the robot head and the perspective of the binocular camera on the observation arm are spliced ​​based on the point cloud of the RGB image texture; then the DBSCAN algorithm is used to perform cluster analysis to retain point cloud data with similar density; then the DIC algorithm is used to match image feature points to obtain the point cloud outline and posture of the target object; finally, the recognition result C i Perform similarity detection with offline trained templates and test images to determine whether the target object is detected;

[0058] The autonomous control module performs autonomous path planning based on the target recognition results and uses a bidirectional random search tree method to generate the robot's autonomous instructions q a ;

[0059] The shared control algorithm selects the position control instruction q according to the interactive instruction pr Or force control instruction q f As the operator's teleoperation command q h and compare it with the robot's autonomous command q a Fusion gets the final fusion instruction q c ,The dynamic weight allocator can dynamically allocate the dimensions and weights of the output instruction mapping of each method in the ,robot joint space through interactive instructions according to the needs of the operator, ,and achieve joint human-machine control of the position, posture and contact force of the ,robot.

[0060] Using this system, the operator can use two force feedback hand controllers in the spacecraft cabin to control the two working arms of the extravehicular robot to perform tasks, and use voice commands to control the observation arm to obtain a better local perspective; at the same time, through a shared control algorithm, the multimodal remote operation control method including posture control, voice control, and force control is integrated with the robot's autonomous control, reducing the operator's operating burden and improving control efficiency.

[0061] Example 2

[0062] The difference between this embodiment and embodiment 1 is that the master-side teleoperation system further includes an incremental control module, a teleoperation mapping parameter adjustment module, and a robotic arm collision detection and warning module;

[0063] In the incremental control module, when the operator presses the button on the force feedback hand controller's handle, the force feedback hand controller's position information is sent to the hand controller driver module; when the button is released, the information is not sent. When the master teleoperation system receives the button release signal, it records the position information of the current task space of the end of the manipulator. At this time, the operator can move the force feedback hand controller's handle to a suitable position, while the manipulator does not receive the signal and is therefore fixed. When the operator presses the button, the manipulator continues to move according to the operator's control. Therefore, the incremental control module uses a mouse-like interaction method. When the operator releases the button on the force feedback hand controller's handle, the end of the manipulator will maintain the current position unchanged. The operator can move the force feedback hand controller's handle to a suitable position and press the button again to continue controlling the manipulator's movement. This solves the problem of limited control range due to the large difference in the working space range of the master force feedback hand controller and the slave manipulator. In addition, when the operator feels uncomfortable with the grip of the force feedback hand controller, he can quickly make adjustments, improving the operator's control accuracy and comfort.

[0064] In the remote operation mapping parameter adjustment module, due to the high difficulty of space operation tasks, the position control of the robotic arm usually requires both rapid movement in a large range and precise operation in a small range, and the ability to flexibly adjust the size of the force feedback. Therefore, it supports the astronauts in the cabin to adjust the position scale mapping parameters and force mapping parameters in real time. The position scale mapping parameter is the ratio of the motion scale of the force feedback hand controller handle to the motion scale of the robotic arm. The larger the parameter, the larger the scale of the robotic arm movement. When the parameter is 0, the robotic arm will not be able to move. The force mapping parameter can adjust the size of the feedback force provided by the hand controller when the end of the robotic arm contacts the environment. The larger the parameter, the stronger the feedback force provided by the hand controller. When the parameter is 0, the force feedback size will be 0;

[0065] The principles and interaction methods of the robotic arm collision detection and warning module are as follows:

[0066] The robotic arm collision detection and warning module, developed based on Unity's collision body mechanism, can detect in real time the risk of collisions between robotic arms and between the arms and the robot platform during movement. It can also issue warnings through multimodal feedback, including auditory, visual, and repulsive force provided by the hand controller. Collision warning is divided into two stages: pre-collision warning and alarm during collision.

[0067] Pre-collision warnings include: In the virtual simulation human-machine interaction software, the outline of the impending collision section of the robotic arm is highlighted in yellow, a warning music plays, the collision indicator turns yellow, and the collision status is displayed as "too close." The hand controller also applies damping force in the direction of the impending collision, prompting the operator to move the handle in the opposite direction to avoid the collision.

[0068] The collision alarm is displayed as follows: In the virtual simulation human-machine interaction software, the outline of the part of the robot arm that has collided is highlighted red, a collision warning music plays, the collision indicator light turns red, and the collision status is displayed as collided. The hand controller also responds with a strong repulsive force to remind the operator to quickly move the handle in the opposite direction.

[0069] Adding human-computer interaction functions that meet actual application needs, such as incremental control modules, remote operation mapping parameter adjustment modules, and robotic arm collision detection and warning modules, to the system design can improve the safety and practicality of the system.

[0070] Example 3

[0071] A method for multimodal shared teleoperation of a three-arm space robot is disclosed. The system described in Example 1 is used, wherein the lower computer of the three-arm space robot multimodal teleoperation system integrates three control methods: posture control, force control, and autonomous control. Posture control includes three modes: hand controller control, voice control, and keyboard interactive control. The above multimodal teleoperation control methods can be combined with autonomous control to achieve multimodal shared teleoperation.

[0072] The posture control principle is as follows:

[0073] Step 1: The system is powered on to initialize the parameters of each robotic arm, establish IP communication, obtain the current posture information and kinematic parameters of each robotic arm, and use the base of each robotic arm as the task space coordinate system to perform forward kinematic posture relationship transformation. The Lagrangian method is used to establish the dynamic model of each robotic arm. Since the space robot is in a weightless environment, the gravity term is omitted, and nonlinear disturbance terms such as joint motor friction are simplified. The calculation formula is as follows:

[0074] Step 2: Read the hand controller operation instructions, voice instructions and interactive instructions output by the master-side teleoperation system in real time. First, the above instructions are parsed into the Cartesian target position x of the end of the robot arm through the master-slave operation space mapping. d ; Then get the joint angle q of the robot arm at this time t , the Cartesian position x of the end of the robot arm is solved by forward kinematics t ; Finally, we get the error term e(t)=x d -x t ;

[0075] Step 3: Input e(t) into the PID controller and iterate to obtain the posture u(t). The calculation formula is as follows:

[0076] The iterative pose u(t) is obtained by inverse kinematics to obtain the new manipulator joint angle q t+1 ; Then in the next cycle, we get x again through forward kinematics t+1 , and compare it with the target pose x d Calculate the error and input it into the PID controller to form a closed control loop. After N iterations, the position control instruction q is output. pr , which greatly improves the control accuracy of the robotic arm. At the same time, it can improve the system's steady-state error, response speed, overshoot and other system performance by adjusting the number of iterations and various PID parameters.

[0077] Regarding force control, the principle is as follows:

[0078] Step 1: Perform the same initialization as the first step of posture control;

[0079] Step 2: The operator sets the target six-dimensional force signal F of the working arm through the master-side teleoperation system d At the same time, read the force data signal F of the force sensor at the end of the working arm t , F d With F t Make the error term and input it into the PID controller to get the iterative force signal F e , the force control instruction q is obtained through the dynamic model f ;

[0080] Step 3: In the next cycle, the force data signal F of the force sensor is again t+1 With F d Calculate the error and substitute it into the PID controller, combined with the Jacobian matrix J of the robot at this time T (q), the same as the posture control to form a closed control loop, iteratively obtain the new force control instruction q f+1 , completing precise force control of the working arm.

[0081] The principles of autonomous control are as follows:

[0082] Step 1: Use the current position of the robotic arm as the root node of an extended random tree. Determine the position of the target based on the recognition result of the target recognition algorithm and use it as the root node of the second extended random tree.

[0083] Step 2: Alternately expand the two extended random trees in both directions using random sampling with the same step size, adding child nodes alternately until the two trees meet, and the path planning algorithm converges;

[0084] Step 3: When the algorithm converges, trace back along the root node at the intersection of the two extended random trees to find a valid path. Perform inverse kinematics on a series of child nodes on the valid path to obtain the values ​​in the joint angle space of the robot arm, and output them as autonomous instructions q a .

[0085] The calculation formula of the shared control algorithm is as follows: q c =Sq h +(IS)q a

[0086] Where S = diag[s1,s2,s3,s4,s5,s6] T ,s i ∈[0,1], the S matrix is ​​a six-dimensional diagonal weight matrix, which represents the dimension and weight of the autonomous control command mapping in the manipulator joint space, I is a six-dimensional unit matrix, q h ,q a ,q c Both are 6×1 matrices, qc Indicates fusion instruction;

[0087] The value of S can be determined by a dynamic weight allocator:

[0088] 1) When the target recognition algorithm does not find the target object, or the target recognition result C i Below the recognition threshold C L When , the dynamic weight allocator sets S to the unit matrix. At this time, the operator relies entirely on the hand controller or keyboard to interactively control the position and force of the working arm, and the position of the observation arm is controlled by voice control.

[0089] 2) When the target recognition result C i Greater than or equal to threshold C L When performing a task, the operator sets the interactive instructions in the master-side teleoperation system according to the type of task, and the dynamic weight distributor calculates the value of S, so that the human and the robot can jointly control the position, posture and contact force of the robotic arm. For example, the operator can use voice to control the position of the observation arm, and the robot can autonomously control the posture of the observation arm to automatically align it with the target object to obtain a better local view. Alternatively, the operator can control the position and posture of the working arm, and the robot can autonomously control the force exerted by the end of the working arm on the environment.

[0090] 3) When the target recognition result C i Greater than or equal to threshold C L When , and S is set to zero matrix, the robot fully controls the position and force of the manipulator;

[0091] The specific implementation steps of the shared control algorithm are as follows:

[0092] Step 1: Before the actual operation, according to the operation mode set by the operator, select the position control instruction q through the interactive instruction pr Or force control instruction q f As the operator's teleoperation command q h ;

[0093] Step 2: Based on the target recognition result C i According to the type of the task, the operator sends the master-side interactive instruction to the dynamic weight allocator to calculate the values ​​of each diagonal element of the weight matrix S, so as to realize the dynamic update of the shared control weight during the operation;

[0094] Step 3: According to the calculation result S of the dynamic weight distributor, the operator's teleoperation instruction q h With the robot's autonomous instructions q a Fusion gets the final fusion instruction q c .

[0095] Example 4

[0096] This embodiment is a practical application example of the system and method of the present invention. It involves a three-arm space robot multimodal shared teleoperation system. The structure of the three-arm space robot is shown in Figure 2. The three-arm space robot is fixed to a large robotic arm outside the space station cabin via a connecting bearing 5. Its power supply line is connected to the robot platform 3 via the connecting bearing 5. Its control circuit and communication module are installed inside the robot platform 3. Figure 3 is a schematic diagram of the structure of the force feedback hand controller.

[0097] Inside the spacecraft cabin, the operator uses two force feedback hand controller handles 17 to control the two working arms 1 and 4 of the extravehicular robot to perform tasks. The first button 18 of the force feedback hand controller controls the opening and closing of the grippers of the working arms 1 and 4. A microphone array collects the operator's voice signals, and voice commands are used to control the observation arm 6, which uses a binocular camera 7 mounted at its end to obtain a better local view. The operator obtains visual information of the robot's surroundings using a structured light camera 2 mounted on the head of the robot platform 3 and a binocular camera 7 mounted at the end of the observation arm 6. This information is transmitted to the host virtual simulation human-computer interaction software via a communication module and used for target recognition in the autonomous control module.

[0098] Specific implementation of incremental control: When the operator presses the second button 19 of the force feedback hand controller, the position information of the force feedback hand controller handle 17 is sent to the hand controller drive module. When the button 19 is released, the position information is not sent. When the master-side teleoperation system receives the signal that the button 19 has been released, it records the position information of the task space at the end of the working arms 1 and 4. At this time, the operator can move the force feedback hand controller handle 17 to the appropriate position, while the working arms 1 and 4 do not receive the signal and are therefore fixed. When the operator presses the button 19, the working arms 1 and 4 continue to move according to the operator's control.

[0099] In the present invention, the virtual simulation human-computer interaction software is developed using Unity, as shown in Figure 4. It includes a real-time rendered three-dimensional robot model, remote operation mapping parameter adjustment, collision detection and warning functions, and can provide the operator with real-time multi-modal feedback information including visual, tactile, and auditory feedback, as well as a rich graphical interactive interface, and output interactive instructions.

[0100] The operator adjusts the position-ratio mapping parameter and force-ratio mapping parameter in real time via the teleoperation mapping parameter adjustment panel 9 on the interactive interface. The position-ratio mapping parameter is the ratio of the motion scale of the force feedback hand controller handle 17 to the motion scale of the robotic arm. A larger parameter results in a larger scale of robotic arm movement. When the parameter is 0, the robotic arm will not move. The force-ratio mapping parameter adjusts the feedback force provided by the hand controller when the robotic arm's end contacts the environment. A larger parameter results in a stronger feedback force. When the parameter is 0, the force feedback is zero.

[0101] During specific operations, the operator uses the various submenus in the main menu panel 11 to complete interactive commands such as setting the manipulator control mode, setting shared operation weights, setting force control target values, reading manipulator parameters, and interactively controlling the manipulator's posture using the keyboard. These interactive commands are then sent to the slave's posture control algorithm, force control algorithm, and shared control algorithm via communication mode, thereby enabling multimodal shared teleoperation of the three-arm spatial robot. Simultaneously, the operator can obtain the real-time position and status of each robot arm using the manipulator's important information display interface 14 within the virtual simulation human-machine interaction software. The manipulator collision detection and warning module, developed based on the Unity collision body mechanism, detects the risk of collisions between the manipulator arms and between the arm and the robot platform during manipulator movement in real time. This module can provide multimodal feedback, including auditory, visual, and repulsive force provided by the hand controller, to issue warnings. Collision warnings are divided into two stages: pre-collision warnings and collision alarms. This prevents manipulator arm collisions during operation and ensures system safety. As shown in Figure 3, the end of the robot's right working arm 4 is about to collide with the end of its left working arm 1. The outline of the impending collision is highlighted: right arm collision warning 8, left arm collision warning 10. Simultaneously, the human-machine interaction software plays warning music, and the left and right arm collision indicators on the robot's important information display interface 14 turn yellow, indicating a collision status of "too close." The hand controller also applies damping force in the direction of the impending collision, prompting the operator to stop moving the handle in that direction and instead move the handle in the opposite direction to avoid a collision.

[0102] During operation, if a collision occurs, the observation arm 6 and the robot platform 3 will indicate a collision. The outline of the collided portion will be highlighted: the observation arm collision alarm 12 and the robot platform collision alarm 15 will be activated. Simultaneously, the human-machine interaction software will play a collision warning sound, and the observation arm collision indicator on the robot arm's important information display interface 14 will turn red, indicating the collision status has been displayed as "collision occurred." The hand controller will also provide a strong repulsive force to prompt the operator to quickly move the handle in the opposite direction. The operator can click the emergency stop button 16 to stop data communication between the master and slave terminals and terminate all movement commands of the robot arm, preventing further collision damage.

[0103] The specific implementation of multimodal shared teleoperation includes the following steps:

[0104] Step 1: Before the actual operation, the operator sets the operation mode of the two working arms through the main menu 11 of the virtual simulation human-computer interaction software, and sends the interactive instructions to the shared control. If posture control is selected, the position control instruction q is output. pr As the operator's teleoperation command q h If force control is selected, the force control instruction q is output f As the operator's teleoperation command qh ;

[0105] Step 2: Based on the video feedback of the visual unit and the target recognition result C i The operator sends the master-side interactive instruction to the dynamic weight allocator to calculate the values ​​of each diagonal element of the weight matrix S, and realizes the dynamic update of the shared control weight during the operation. The update steps are as follows:

[0106] 1) When the target recognition algorithm does not find the target object, or the target recognition result C i Below the recognition threshold C L When , the dynamic weight allocator sets S to the unit matrix. At this time, the operator relies entirely on the hand controller or keyboard to interactively control the position and force of the working arm. The position of the observation arm 6 is controlled by voice. The latest voice command and the completion status of the command are displayed on the voice command display panel 13.

[0107] 2) When the target recognition result C i Greater than or equal to threshold C L When the operator sets the interactive instructions in the master-side teleoperation system according to the type of task, the dynamic weight allocator calculates the value of S, so that the human and the robot can jointly control the position, posture and contact force of the robot arm. For example, the operator can use voice to control the position of the observation arm 6, and the robot can autonomously control the posture of the observation arm 6 to automatically align it with the target object to obtain a better local perspective. Alternatively, the operator can control the position and posture of the working arm, and the robot can autonomously control the force exerted by the end of the working arm on the environment.

[0108] 3) When the target recognition result C i Greater than or equal to threshold C L When , and S is set to zero matrix, the robot fully controls the position and force of the manipulator;

[0109] Step 3: According to the calculation result S of the dynamic weight distributor, the operator's teleoperation instruction q h With the robot's autonomous instructions q a Fusion gets the final fusion instruction q c , assign teleoperation command q h With autonomous instruction q a The dimension and weight of the robot arm joint space are mapped to achieve the fusion formula of human-machine joint control of the robot arm's position, posture and contact force as follows: c =Sq h +(IS)q a .

[0110] In summary, the method of the present invention uses the two working arms of the three-arm space robot, which is more convenient to operate and more flexible to use. It obtains a better local perspective through voice control of the observation arm 6 and utilizes a hybrid control strategy, which allows the operator to directly operate to exert his judgment and decision-making ability, while ensuring that the robot has a certain degree of autonomy, and can assist the operator to complete complex extravehicular operation tasks, so that the operator can focus on controlling the robotic arm to perform important and refined operations, and the robot can independently complete other simpler tasks, reducing the operator's operating burden and improving the control accuracy and control efficiency of remote operation.

[0111] Throughout this specification, references to terms such as "one embodiment," "example," or "specific example" indicate that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0112] It should be noted that the above content merely illustrates the technical idea of ​​the present invention and cannot be used to limit the scope of protection of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications all fall within the scope of protection of the claims of the present invention.

Claims

1. A three-arm space robot multimodal shared teleoperation system, characterized by: At least includes a master-end teleoperation system, a communication module and a slave-end robot system; The master-end teleoperation system includes at least two force feedback hand controllers located on the left and right hands, a microphone array, and a host computer software, wherein the host computer software includes at least virtual simulation human-computer interaction software, a hand controller driving module, and a speech recognition module; The slave robot system includes a working arm, an observation arm, a force sensor, a vision unit, and lower computer software. The working arms are provided with two, each equipped with a gripper at the end; a binocular camera is provided at the end of the observation arm; the lower computer software includes at least a force control algorithm, a posture control algorithm, a target recognition algorithm, an autonomous control algorithm, and a shared control algorithm; the vision unit is used to provide visual information around the slave robot to the master teleoperation system and is used for target recognition in the autonomous control algorithm; The communication module is used to build a medium- and short-range low-latency wireless local area network to achieve wireless communication between the master-end teleoperation system and the slave-end robot system.

2. A three-arm space robot multimodal shared teleoperation system according to claim 1, characterized in that: In the master-side teleoperation system: The force feedback hand controller is used to collect posture information input by the operator, receive force sensor data and provide three-dimensional force feedback to the operator; The microphone array is used to collect the operator's audio signal; The virtual simulation human-computer interaction software includes a real-time rendered three-dimensional robot model, remote operation mapping parameter adjustment, collision detection and warning functions, provides feedback information and an interactive interface for the operator, and outputs interactive instructions; The hand controller driving module is used to calculate the operator's input posture information and output it as an operation instruction; The speech recognition module is used to analyze and output the operator's speech instructions.

3. The three-arm space robot multimodal shared teleoperation system according to claim 1, characterized in that: In the lower computer software: The force control algorithm is used to control the force between the working arm and the environment, and outputs the force control instruction q according to the force signal setting value in the interactive instruction output by the master end teleoperation system and the feedback data of the force sensor. f ; The posture control algorithm is used to control the posture of the working arm and the observation arm, and can read the hand controller operation instructions, voice instructions and interactive instructions output by the master end remote operation system, and output the position control instruction q pr ; The target recognition algorithm is used to identify the target object in the environment and obtain its position and outline; The autonomous control module performs autonomous path planning based on the target recognition results and uses a bidirectional random search tree method to generate the robot's autonomous instructions q a ; The shared control algorithm: selects the position control instruction q according to the interactive instruction pr Or force control instruction q f As the operator's teleoperation command q h and compare it with the robot's autonomous command q a Fusion gets the fusion instruction q c Based on the dynamic weight allocator, the dimensions and weights of the output command mappings of each method in the joint space of the robot arm are dynamically allocated through interactive instructions, so that the human and the machine can jointly control the position, posture and contact force of the robot arm.

4. A three-arm space robot multimodal shared teleoperation system according to claim 2 or 3, characterized in that: The master-side teleoperation system further includes an incremental control module, a teleoperation mapping parameter adjustment module, and a robotic arm collision detection and warning module; The incremental control module enables and controls the movement of the robotic arm through the buttons on the force feedback hand controller handle; The teleoperation mapping parameter adjustment module is used to adjust the master-slave position ratio mapping parameters and force mapping parameters, and to adjust the motion step length of the robotic arm and the feedback force provided by the force feedback hand controller; The robot arm collision detection and warning module is used to detect the risk of collision between arms or between the arm and the robot platform during the movement of the robot arm, and to issue a warning.

5. A multimodal shared teleoperation method for a three-arm space robot using the system of claim 1, characterized in that: When an operator controls a slave robot with two working arms and one observation arm through a master remote operating system, the slave computer outputs control instructions for three methods, namely posture control, force control, and autonomous control, to the shared control algorithm to control the robot arms. When using posture control, the command output by the master teleoperation system is read in real time, and the command is parsed into the Cartesian target posture x of the end of the manipulator through the master-slave operation space mapping. d , and then obtain the joint angle q of the robot arm at this time t , the Cartesian position x of the end of the robot arm is solved by forward kinematics t , and get the error term e(t)=x d -x t ; Input the error term e(t) into the PID controller to iteratively obtain the posture u(t), and obtain the new robot arm joint angle q by inverse kinematic solution of the iterative posture u(t) t+1 ; Then in the next cycle, we get x again through forward kinematics t+1 , and compare it with the target pose x d Calculate the error and input it into the PID controller to form a closed control loop. After N iterations, the position control instruction q is output. pr ; When force control is used, the operator sets the target six-dimensional force signal F of the working arm through the master-side teleoperation system. d , read the force data signal F of the force sensor at the end of the working arm at this time t , F d With F t Make the error term and input it into the PID controller to get the iterative force signal F e , the force control instruction q is obtained through the dynamic model f ; In the next cycle, the force data signal F of the force sensor is again t+1 With F d Calculate the error and substitute it into the PID controller, combined with the Jacobian matrix J of the robot at this time T (q), the same as the posture control to form a closed control loop, iteratively obtain the new force control instruction q f+1 ; When using autonomous control, the posture of the end of the robot arm is used as the root node of an extended random tree. The posture of the target is determined according to the recognition result of the target recognition algorithm and used as the root node of the second extended random tree. The two extended random trees are alternately extended in both directions with the same step size and random sampling, and child nodes are added alternately until the two trees meet. The path planning algorithm converges. After the algorithm converges, a valid path can be found by backtracking along the root node at the intersection of the two extended random trees. The kinematic inverse solution of a series of child nodes on the valid path is performed to obtain the value of the robot arm joint angle space, and the output is the autonomous command q a .

6. A three-arm space robot multi-modal shared teleoperation method according to claim 5, characterized in that: In the three integrated methods of posture control, force control, and autonomous control, the shared control algorithm includes the following steps: S1: According to the operation mode set by the operator, select the position control instruction q through the interactive instruction pr Or force control instruction q f As the operator's teleoperation command q h ; S2: According to the target recognition result C i According to the type of the task, the operator sends the master-side interactive instruction to the dynamic weight allocator to calculate the values ​​of each diagonal element of the weight matrix S, so as to realize the dynamic update of the shared control weight during the operation process; S3: According to the calculation result S of the dynamic weight distributor, the operator's teleoperation instruction q h With the robot's autonomous instructions q a Fusion gets the final fusion instruction q c .

7. A three-arm space robot multi-modal shared teleoperation method according to claim 6, characterized in that: The weight matrix S in step S2 is determined by a dynamic weight allocator: When the target recognition algorithm does not find the target object, or the target recognition result C i Below the recognition threshold C L When , the dynamic weight allocator sets S to the unit matrix. At this time, the operator uses the hand controller or keyboard to interactively control the position and force of the working arm, and uses voice control to observe the position of the arm; When the target recognition result C i Greater than or equal to threshold C L When the operator sets the interactive instructions in the master teleoperation system according to the type of task, the dynamic weight distributor calculates the value of S, and the position, posture and contact force of the manipulator are controlled jointly by the human and the machine. When the target recognition result C i Greater than or equal to threshold C L When , and S is set to a zero matrix, the robot fully controls the position and force of the manipulator.

Images