Safety shared control system based on performance of teleoperator

Abstract

A safety shared control system based on the performance of a teleoperator, the system comprising a master-end teleoperation system, a slave-end robotic arm system and a communication module, wherein a master end comprises the measurement of an electroencephalogram signal of a teleoperator, an operation input of a hand controller, and upper computer software, and an upper computer comprises a man-robot interaction interface, a safety simulation for keeping a robot safe, and a performance-of-teleoperator model; and a slave end comprises a visual camera, a robotic arm and lower computer software, and the lower computer software comprises a target recognition algorithm, an autonomous controller, and a shared controller for dynamically allocating a man-robot weight. By means of safety simulation, a safety control coefficient before a teleoperator actually performs operations is confirmed, a safety control method under different pieces of operation experience is implemented, and the safe operation of a teleoperated robot is ensured; and after a long-term operation, an electroencephalogram signal of the teleoperator is measured to dynamically update a man-robot shared control weight on the basis of the performance of the teleoperator, thereby improving the operation capability of a teleoperation system.

Description

A safe shared control system based on operator performance Technical Field

[0001] The present invention belongs to the technical field of robot teleoperation systems, and in particular relates to a safety sharing control system based on operator performance. Background Art

[0002] Teleoperation systems are a key area of ​​robotics development. Their simple control methods allow operators to remotely control the robot's movements over the internet, making them widely used in environments that are inaccessible or hazardous to humans. However, due to varying levels of operator proficiency, the same task can yield different results when performed by different operators. Furthermore, communication latency and data loss inevitably present challenges, making it difficult for operators to perform delicate operations. On the other hand, when robots operate autonomously, people hope to achieve fully automated control—that is, without human intervention, the robot can perceive environmental information and autonomously complete its tasks, with the operator only needing to remotely supervise the robot's operation. However, current levels of automation and sensor technology make it difficult to develop fully autonomous robotic systems suitable for complex environments and diverse tasks. Therefore, shared human-robot control has become an effective approach to address these issues.

[0003] Existing shared control methods for teleoperated human-robot shared control systems primarily include semi-autonomous control and hybrid control. In semi-autonomous control, the state variables controlled by the autonomous controller and the operator are separated, with the operator and robot each performing distinct tasks. This control approach is relatively simple, but the degree of shared control is limited, preventing the full utilization of their respective strengths. In hybrid control, the operator generates input commands through human-robot interaction, while the robot's autonomous controller generates autonomous decision-making commands. A specific arbitration mechanism is employed to maximize the integration of the advantages of both, providing an optimized fused control input. This control approach can allocate optimal control rights to the operator and robot based on different operating environments, but its effectiveness is largely limited by the arbitration mechanism, which is a key component of shared control methods. Furthermore, operators with varying levels of experience can directly impact the teleoperation system's operational capabilities and may even cause damage to the robot itself. Therefore, safety protection strategies tailored to varying levels of experience should be a key consideration in the design of shared control methods.

[0004] Summary of the Invention

[0005] To address the above-mentioned issues, the present invention discloses a safety shared control system based on operator performance. Before actual operation, the operator's safety control coefficient is confirmed through safety simulation, thereby realizing a safety control method under different operating experiences and protecting the safe operation of the teleoperated robot. After long-term operation, the operator's electroencephalogram (EEG) signal is measured to realize the dynamic update of the human-machine shared control weight based on the operator's performance (PoT), thereby effectively integrating human decision-making ability and robot operation ability, and improving the operation capability of the teleoperation system.

[0006] To achieve the above object, the technical solution of the present invention is as follows:

[0007] A safe shared control system based on operator performance, comprising a master-end teleoperation system, a slave-end manipulator system, and a communication module;

[0008] The master-side teleoperation system includes the operator's EEG signal measurement, hand controller operation input, and host computer software. The host computer includes a human-computer interaction interface (GUI), safety simulation for protecting the robot's safety, and a PoT model.

[0009] The operator's EEG signal measurement is performed using an Emotiv EPOC 14-channel EEG instrument to collect EEG signals and obtain raw EEG signals;

[0010] The hand controller operation input is to use the Geomagic Touch X hand controller to perform operation input and obtain the operator's operation instruction U h ;

[0011] The human-machine interaction interface includes information feedback output such as terminal vision and robot status, as well as interactive operations such as starting a task, selecting manual / shared / autonomous mode, and ending a task;

[0012] The safety simulation is used to confirm the safety control factor K before the operator actually operates. s , the specific steps are as follows:

[0013] (a) Constructing a simulation environment, including a robotic arm with an end gripper, a target object, and a target position;

[0014] (b) Performing a grasping task, the operator uses a hand controller to manipulate the robotic arm in the simulation environment to grasp the target object to the target location, and three operation performance indicators are counted: operation time, path length, and path smoothness;

[0015] (c) Operator Performance PoT in Computational Security Simulation SIM , its calculation formula is: PoTSIM =∑W i P i

[0016] Where i = 1, 2, 3, P i is the normalized operation performance index of the i-th, W i is the weighted coefficient corresponding to the i-th operation performance indicator;

[0017] (d) Output safety control factor Ks = PoT SIM , Ks∈[0,1];

[0018] The PoT model uses EEG signals to indirectly obtain the operator's performance PoT during long-term work in real time through a BP (back propagation) neural network model. BP The neural network consists of an input layer, a hidden layer, and an output layer. The operator's 14-channel EEG signal is used as input, and the model outputs PoT BP ,In the model training process, data from performing the crawling task in ,security simulation is used;

[0019] The slave-end robotic arm system includes a robotic arm, a visual camera, and lower-computer software. The lower-computer software includes a target recognition algorithm, an autonomous controller for generating autonomous instructions for the robot, and a shared controller for dynamically allocating human-machine control weights.

[0020] The robotic arm is a slave entity that replaces humans to perform tasks in scenes that are inaccessible or harmful to humans;

[0021] The visual camera provides visual information of the slave's operating environment to the master, and is used for target recognition by the robot's autonomous controller.

[0022] The target recognition algorithm is used to identify 3D targets in space and determine their position and pose. It uses the LineMod algorithm for identification and positioning. The algorithm's workflow can be divided into two phases: template rendering and image matching. During the template rendering phase, RGB-D images of the target object are acquired at multiple angles and scales. During the image matching phase, a sliding window is used to measure the similarity between the offline trained template and the test image to determine whether the target object has been detected.

[0023] The autonomous controller performs autonomous path planning based on the target recognition results and generates the robot's autonomous instructions Ur using a rapidly expanding random tree (RRT) algorithm. This algorithm uses the search starting point as the root node and then generates a random expansion tree by adding child nodes through random sampling. When the child nodes of the random tree enter the target area, the path from the starting position to the target position is obtained.

[0024] The shared controller integrates the operator's operating instructions and the robot's autonomous instructions to obtain the robot's final shared instruction Uc. Its calculation is divided into two stages: the safety protection period and the post-long operation period. The safety protection period is based on the operation performance of the safety simulation, and the post-long operation period is based on the operation performance during the actual operation based on the EEG signal. The specific steps are as follows:

[0025] 1) Safety protection period, calculated according to the safety control coefficient Ks of safety simulation: K h =K s

[0026] 2) After a long period of operation, the operator's performance according to the PoT model BP Calculation can be obtained:

[0027] Where H and L are the upper and lower thresholds of operator performance, respectively.

[0028] 3) Output shared control instructions: U c =K h U h +(1-K h )U r , K h ∈[0,1].

[0029] The communication module realizes the communication between the master-end teleoperation system and the slave-end robotic arm system.

[0030] The beneficial effects of the present invention are:

[0031] 1. The present invention confirms the operator's safety control coefficient through safety simulation before actual operation, realizes safety control methods under different operating experience, and can protect the safe operation of teleoperated robots under different operating experience;

[0032] 2. The present invention uses the operator's EEG signals to monitor the operator's performance in long-term operations in real time, effectively arbitrates the operator's control weight based on the operator's performance, and realizes the dynamic update of human-machine shared control weight. It can effectively integrate human decision-making ability and robot operation ability, and improve the operation ability of the remote operation system. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] FIG1 is an overall block diagram of a safety shared control system based on operator performance.

[0034] Figure 2 is a diagram of the security simulation environment.

[0035] Figure 3 is the flow chart of the RRT algorithm.

[0036] Figure 4 shows the PoT model based on BP neural network. DETAILED DESCRIPTION

[0037] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0038] This embodiment provides a safe shared control system based on operator performance, including a master-side teleoperation system and a slave-side robotic arm system. The overall system framework is shown in Figure 1. In this invention, a human-machine interaction interface (GUI) serves as the interface between the operator and the robot, displaying slave-side visual information, robot status information, and safe simulation environment information, as well as facilitating task interactions such as starting a task, selecting manual, shared, or autonomous modes, and ending a task. The specific implementation includes the following steps: (1) Before performing actual operations, the operator performs the target object grasping task in the safety simulation through the hand controller to confirm the operator's safety control coefficient. This safety control coefficient will be used to allocate the operator's control weight in the dynamic weight distributor; (2) After completing the safety simulation, the operator will input operation instructions through the hand controller to control the actual operation of the slave robot arm under the safety control coefficient. The slave end operation environment information is fed back to the master end display through visual feedback. The lower computer will also perform target recognition based on the visual information provided by the camera. The autonomous controller will complete the trajectory planning and generate the robot's autonomous instructions. The shared controller will share and fuse the operation instructions and autonomous instructions based on the control weight; (3) After a long period of operation, by inputting the EEG signal into the PoT model in the upper computer, the PoT model calculates the operator's PoT and sends it to the dynamic weight distributor of the slave end, realizing the dynamic update of the shared control weight based on the operator's performance, and updating the operator's shared control weight under long-term operation.

[0039] (1) Safety simulation confirms the safety control factor:

[0040] Before the actual operation, since different operators have different proficiency in the teleoperation system, in order to ensure the safety of the robot, a safety simulation is performed to obtain the operator's safety control coefficient K before the actual operation. s Specifically, step (1) includes:

[0041] (a) A safety simulation environment is constructed using Matlab, as shown in Figure 2, which includes a robotic arm with an end fixture, a target object, and a target position.

[0042] (b) Performing a grasping task, the operator uses a hand controller to manipulate the robotic arm in the simulation environment to grasp the target object to the target location, and three operation performance indicators are counted: operation time, path length, and path smoothness;

[0043] (c) Operator Performance PoT in Computational Security Simulation SIM , its calculation formula is: PoT SIM =∑W i P i

[0044] Where i = 1, 2, 3, P i is the normalized operation performance index of the i-th, W i is the weighted coefficient corresponding to the i-th operation performance indicator;

[0045] (d) Output safety control factor Ks = PoT SIM , Ks∈[0,1];

[0046] (2) Shared control in actual operation:

[0047] In actual operation, the robot's input instructions include two parts: the operator's operation instructions U h and the robot's autonomous instructions U r ,The operation command comes from the input of the master hand controller, and the autonomous command is the result of path planning by the autonomous controller based on the target recognition result. Specifically, step (2) includes:

[0048] (a) The LineMod algorithm identifies 3D objects in space and determines their pose. The algorithm's workflow can be divided into two stages: template rendering and image matching. In the template rendering stage, RGB-D images of the target object are collected at multiple angles and scales. In the image matching stage, the offline trained template and the test image are similarly measured using a sliding window to determine whether the target object has been detected.

[0049] (b) The RRT algorithm performs path planning and generates autonomous instructions. The algorithm flow is shown in Figure 3;

[0050] (c) Operational instructions and autonomous instructions are shared and integrated as follows: c =K s U h +(1-K s )U r , K s ∈[0,1]

[0051] (3) After a long period of operation, dynamic update of shared control weight based on PoT is realized:

[0052] After a long period of operation, the operator's operating experience and operating status will change to a certain extent. Therefore, it is necessary to further evaluate the changes in the operator's performance online to dynamically update the human-machine shared control weight. By using the Emotiv EPOC 14-channel electroencephalogram (EEG) device to collect EEG signals, a 14-channel EEG signal is obtained. The BP neural network is used to model the PoT. The model is shown in Figure 4. Specifically, step (3) includes:

[0053] (a) EEG signal acquisition: An Emotiv EPOC 14-channel EEG instrument is used to acquire 14-channel raw EEG signals. The software development kit (SDK) provided by Emotiv performs signal filtering and feature extraction on the raw signals to obtain the 14-channel EEG signals required by the PoT model.

[0054] (b) BP neural network modeling includes input layer, hidden layer, and output layer. The operator's 14-channel EEG signal is used as input, and the output layer data comes from the grasping task performed in the safety simulation.

[0055] When calculating the neuron transfer of each layer:

[0056] Where, and is the excitation and bias of the jth neuron in the i-th layer, is the weight from the kth neuron in the j-1th layer to the ith neuron in the jth layer. The cost function of the output layer is as follows:

[0057] Where PoT and PoT* are the actual output and expected output of the output layer respectively. The weight update function is as follows:

[0058] Where t is the number of iterations, η is the learning rate of the neural network, and δ i is the learning error of the i-th neuron in the current layer, O k is the output of the kth neuron in the upper layer. Output of the neural network PoT BP As follows: PoT BP =f n (x p )

[0059] Where x p is the 14-channel EEG signal vector, f n is the trained neural network model.

[0060] (c) Calculate the shared control weight K based on PoT h :

[0061] Where H and L are the upper and lower thresholds of operator performance, respectively, as follows:

[0062] ① When K h =0, that is, the operator's performance is poor and below the lower threshold, the operation mode is autonomous control mode, and the output command of the shared controller is the autonomous command of the robot arm. At this time, the operator does not participate in the control of the robot arm, and the robot arm performs autonomous control based on visual information.

[0063] ②When 0<K h When K is less than 1, the operator's performance is good and between the upper and lower thresholds. The operation mode is human-machine shared control mode, and the output instructions of the shared controller are the shared fusion of the operator's operation instructions and the robot's autonomous instructions. In this mode, when K h = 0.5, the operator and the robot each account for half of the control weight. h As the value continues to increase, the control weight of the operator continues to increase.

[0064] ③When K h =1, that is, the operator's performance is excellent and higher than the upper threshold, the operation mode is manual control mode, and the output instructions of the shared controller are the operator's operation instructions. At this time, the robot arm is completely controlled by the operator, that is, the operator controls the robot arm through the hand controller.

[0065] (d) Operational instructions and autonomous instructions are shared and integrated as follows: c =K h U h +(1-K h )U r , K h ∈[0,1]

[0066] (e) The robotic arm receives the shared control instruction U c , and reach the final target location.

[0067] It should be noted that the above content merely illustrates the technical idea of ​​the present invention and cannot be used to limit the scope of protection of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications all fall within the scope of protection of the claims of the present invention.

Claims

1. A safety shared control system based on operator performance, characterized in that: It includes a master-end teleoperation system, a slave-end robotic arm system and a communication module; The master-end teleoperation system includes the operator's EEG signal measurement, hand controller operation input and host computer software, and the host computer includes a human-computer interaction interface, a safety simulation for protecting the robot's safety, and a PoT model; The slave-end robotic arm system includes a robotic arm, a visual camera, and a lower-level computer software, wherein the lower-level computer software includes a target recognition algorithm, an autonomous controller for generating autonomous instructions for the robot, and a shared controller for dynamically allocating human-machine control weights; The communication module realizes the communication between the master-end teleoperation system and the slave-end robotic arm system.

2. The operator performance-based safety shared control system according to claim 1, characterized in that: The EEG signal measurement of the operator in the master-side teleoperation system is performed by using an Emotiv EPOC 14-channel EEG instrument to collect EEG signals and obtain raw EEG signals; The hand controller operation input is to use the Geomagic Touch X hand controller to perform operation input and obtain the operator's operation instruction U h ; The human-machine interaction interface includes information feedback output from the terminal vision and robot status, as well as interactive operations for starting tasks, manual / shared / autonomous mode selection, and ending tasks; The safety simulation is used to confirm the safety control factor K before the operator actually operates. s , the specific steps are as follows: (a) constructing a simulation environment, including a robotic arm with an end fixture, a target object, and a target position; (b) Performing the grasping task, the operator uses the hand controller to manipulate the robotic arm in the simulation environment to grasp the target object to the target position, and three operation performance indicators are counted: operation time, path length, and path smoothness; (c) Operator Performance PoT in Computational Security Simulation SIM , its calculation formula is: can SIM =∑W i P i Where i = 1, 2, 3, P i is the i-th normalized operation performance index, W i is the weighted coefficient corresponding to the i-th operation performance indicator; (d) Output safety control factor Ks = PoT SIM , Ks∈[0,1]; The PoT model uses EEG signals to indirectly obtain the operator's performance PoT during long-term work in real time through a back-propagation neural network model. BP The neural network includes an input layer, a hidden layer, and an output layer. The operator's 14-channel EEG signal is used as input, and the model outputs PoT BP ,In the model training process, the data of performing the grasping task in the safety simulation is used.

3. The operator performance-based safety shared control system according to claim 1, characterized in that: The robot arm in the slave robot arm system is a slave end body that replaces humans to reach scenes that are inaccessible or harmful to humans to perform tasks; The visual camera provides visual information of the slave end's working environment to the master end, and is used for target recognition by the robot arm's autonomous controller; The target recognition algorithm is used to identify the 3D target object in space and obtain its position and posture. The LineMod algorithm is used for recognition and positioning. The process of the algorithm can be divided into two stages: template rendering and image matching. In the template rendering stage, RGB-D images of the target object are collected at multiple angles and multiple scales. In the image matching stage, the offline training template and the test image are similarly measured by window sliding to determine whether the target object is detected; The autonomous controller performs autonomous path planning based on the target recognition result and generates the robot autonomous instruction Ur. The algorithm uses the fast expansion random tree method. The algorithm uses the search starting point as the root node and then generates a random expansion tree by adding child nodes through random sampling. When the child node of the random tree enters the target area, the path from the starting point to the target position is obtained. The shared controller integrates the operator's operating instructions and the robot's autonomous instructions to obtain the robot's final shared instructions Uc. The calculation is divided into two stages: the safety protection period and the long-term operation period. The safety protection period is based on the operation performance of the safety simulation, and the long-term operation is based on the operation performance of the actual operation based on the EEG signal. The specific steps are as follows: 1) Safety protection period, calculated according to the safety control coefficient Ks of safety simulation: K h =K s 2) After a long period of operation, the operator's performance according to the PoT model BP The calculation can be obtained: In the formula, H and L are the upper and lower thresholds of the operator’s performance, respectively; 3) Output shared control instructions: IN c =K h IN h +(1-K h )IN r ,K h ∈[0,1].

4. A safety shared control system based on operator performance, characterized in that: The specific implementation of the control system includes the following steps: (1) Before the actual operation, the operator uses the hand controller to perform the target object grasping task in the safety simulation to confirm the operator's safety control factor. This safety control factor will be used to allocate the operator's control weight in the dynamic weight allocator; (2) After completing the safety simulation, the operator will input the operation instructions through the hand controller to control the actual operation of the slave robot arm under the safety control factor. The slave operation environment information is fed back to the master display through visual feedback, and the lower computer will also The autonomous controller uses this information to complete trajectory planning and generate autonomous commands for the robot. The shared controller will share and fuse the operation commands and autonomous commands based on the control weights. (3) After long-term operation, the EEG signal is input into the PoT model in the upper computer. The PoT model calculates the operator's PoT and sends it to the dynamic weight allocator on the slave end, thereby realizing the dynamic update of the shared control weight based on the operator's performance and updating the operator's shared control weight under long-term operation.

Images