A safety sharing control system based on operator performance

Abstract

This invention discloses a safety-sharing control system based on operator performance, comprising a master-end teleoperation system, a slave-end robotic arm system, and a communication module. The master end includes operator EEG signal measurement, hand controller operation input, and host computer software, which includes a human-machine interface, safety simulation for robot safety, and an operator performance model. The slave end includes a vision camera, a robotic arm, and lower-level software, which includes a target recognition algorithm, an autonomous controller, and a sharing controller for dynamically allocating human-machine weights. This invention uses safety simulation to confirm the safety control coefficients before actual operation, realizing safety control methods under different operating experiences and protecting the safe operation of the teleoperated robot. After prolonged operation, by measuring the operator's EEG signals, it achieves dynamic updates of the human-machine shared control weights based on operator performance, improving the operational capabilities of the teleoperation system.

Description

Technical Field

[0001] This invention belongs to the field of robot teleoperation system technology, specifically relating to a safe shared control system based on operator performance. Background Technology

[0002] Teleoperation systems, as an important direction in robotics development, are widely used in scenarios inaccessible or harmful to humans due to their simple control method, allowing operators to remotely control robots via a network. However, varying operator skill levels lead to different results for the same task, and issues such as communication latency and data loss inevitably exist, making it difficult for operators to perform precise operations. On the other hand, when robots operate autonomously, the ideal is fully automated control—the robot that can perceive environmental information and autonomously complete tasks without human intervention, requiring only remote supervision from the operator. However, given current levels of automation and sensor technology, it is difficult to develop fully autonomous robot systems suitable for complex environments and diverse tasks. Therefore, human-robot shared control has become an effective method to address these problems.

[0003] For teleoperated human-machine shared control systems, existing shared control methods mainly include semi-autonomous control and hybrid control. In semi-autonomous control, the state variables controlled by the autonomous controller and the operator are separate, with each undertaking different tasks. This control method is relatively simple, but the degree of sharing is limited, failing to fully leverage the advantages of each. In hybrid control, the operator generates input commands through human-machine interaction, and the robot's autonomous controller generates autonomous decision commands. Through a certain arbitration mechanism, the advantages of both are integrated as much as possible, providing an optimized fused control input. This control method can allocate better control rights to the operator and robot according to different operating environments. However, the effectiveness of this control method is largely limited by the arbitration mechanism, and a reasonable arbitration mechanism is a key part of the shared control method. Furthermore, operators with different operating experience will directly affect the operational capabilities of the teleoperated system and may even cause damage to the robot itself. Therefore, safety protection strategies under different operating experience should also be an important consideration in the design of shared control methods. Summary of the Invention

[0004] To address the aforementioned issues, this invention discloses a safety-sharing control system based on operator performance. Before actual operation, safety simulation is used to confirm the operator's safety control coefficients, enabling safety control methods under different operating experiences and protecting the safe operation of the teleoperated robot. After prolonged operation, by measuring the operator's electroencephalogram (EEG), dynamic updates of the human-machine shared control weights based on operator performance (PoT) are achieved, thereby effectively integrating human decision-making capabilities and robot operating capabilities, and improving the operational capabilities of the teleoperated system.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows:

[0006] A safety-sharing control system based on operator performance includes a master-end teleoperation system, a slave-end robotic arm system, and a communication module;

[0007] The master-end teleoperation system includes operator EEG signal measurement, hand controller operation input, and host computer software. The host computer includes a human-computer interaction interface (GUI), a safety simulation for protecting the robot's safety, and a Potentially Tunneling (PoT) model.

[0008] The operator's EEG signal measurement was performed using an Emotiv EPOC 14-channel EEG analyzer to acquire raw EEG signals.

[0009] The hand controller operation input is performed using the Geomagic Touch X hand controller to obtain the operator's operation command U. h ;

[0010] The human-computer interaction interface includes feedback output of information such as end vision and robot status, as well as interactive operations such as starting a task, selecting manual / shared / autonomous mode, and ending a task.

[0011] The safety simulation is used to confirm the safety control coefficient K before the operator actually performs the operation. s The specific steps are as follows:

[0012] (a) Construct a simulation environment, including a robotic arm with an end effector, a target object, and a target location;

[0013] (b) Perform the grasping task. The operator uses a hand controller to manipulate the robotic arm in the simulation environment to grasp the target object to the target position. Three operation performance indicators are statistically analyzed: operation time, path length, and path smoothness.

[0014] (c) Calculating Operator Performance (PoT) in Safety Simulation SIM Its calculation formula is:

[0015] PoT SIM =∑W i P i

[0016] In the formula, i = 1, 2, 3, P i W is the i-th normalized operational performance metric. i is the weighting coefficient corresponding to the i-th performance indicator;

[0017] (d) Output safety control coefficient Ks = PoT SIM Ks∈[0,1];

[0018] The PoT model utilizes electroencephalogram (EEG) signals to indirectly acquire, in real-time, operator performance (PoT) during prolonged work sessions via a backpropagation (BP) neural network model. BP The neural network consists of an input layer, hidden layers, and an output layer. The operator receives 14 channels of EEG signals as input, and the model outputs a PoT. BP During model training, data from security simulations used to perform grabbing tasks were employed.

[0019] The slave robotic arm system includes a robotic arm, a vision camera, and lower-level software. The lower-level software includes a target recognition algorithm, an autonomous controller for generating autonomous robot commands, and a shared controller for dynamically allocating human-machine control weights.

[0020] The robotic arm is a slave unit that performs tasks in places where humans cannot reach or where it is harmful to humans.

[0021] The vision camera provides visual information about the working environment of the slave end to be fed back to the master end, as well as target recognition for the autonomous controller of the robotic arm.

[0022] The target recognition algorithm is used to identify 3D target objects in space and obtain their poses. It employs the LineMod algorithm for recognition and localization, and the algorithm's process can be divided into two stages: template rendering and image matching. In the template rendering stage, RGB-D images of the target object are acquired from multiple angles and scales. In the image matching stage, the similarity between the offline-trained template and the test image is measured by sliding a window to determine whether the target object has been detected.

[0023] The autonomous controller performs autonomous path planning based on the target recognition results, generates robot autonomous commands Ur, and adopts the Rapidly-exploring Random Tree (RRT) method. This algorithm takes the starting position of the search as the root node, and then generates a random expansion tree by adding child nodes through random sampling. When a child node of the random tree enters the target area, the path from the starting position to the target position is obtained.

[0024] The shared controller integrates the operator's commands and the robot's autonomous commands to obtain the robot's final shared command Uc. Its calculation is divided into two phases: a safety protection period and a post-long-term operation period. The safety protection period is based on the operational performance during safety simulation, while the post-long-term operation period is based on the operational performance during actual operation using EEG signals. The specific steps are as follows:

[0025] 1) The safety protection period can be calculated based on the safety control coefficient Ks from the safety simulation:

[0026] K h =K s

[0027] 2) After prolonged operation, based on the operator performance of the PoT model. BP Calculation yields:

[0028]

[0029] In the formula, H and L are the upper and lower thresholds of the operator's performance, respectively.

[0030] 3) Output shared control commands:

[0031] U c =K h U h +(1-K h )U r K h ∈[0,1].

[0032] The communication module enables communication between the master remote operating system and the slave robotic arm system.

[0033] The beneficial effects of this invention are:

[0034] 1. This invention confirms the operator's safety control coefficient through safety simulation before actual operation, realizes a safety control method under different operating experience, and can protect the safe operation of remotely operated robots under different operating experience.

[0035] 2. This invention uses the operator's EEG signals to monitor the operator's performance in real time during long-term operations. Based on the operator's performance, the control weight of the operator is effectively arbitrated, realizing the dynamic update of the human-machine shared control weight. It can effectively integrate human decision-making ability and robot operation ability, and improve the operation capability of the remote operating system. Attached Figure Description

[0036] Figure 1 This is a general block diagram of a safety-sharing control system based on operator performance.

[0037] Figure 2 This is a diagram of a safe simulation environment.

[0038] Figure 3 This is a flowchart of the RRT algorithm.

[0039] Figure 4 This is a PoT model based on a BP neural network. Detailed Implementation

[0040] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0041] This embodiment provides a safety-sharing control system based on operator performance, including a master-end teleoperation system and a slave-end robotic arm system. The overall framework of the system is as follows: Figure 1 As shown. In this invention, the human-machine interface (GUI) serves as the interaction interface between the operator and the robot, used to display slave-side visual information, robot status information, safety simulation environment information, and for task interaction, such as starting a task, selecting manual / shared / autonomous mode, and ending a task. The specific implementation includes the following steps:

[0042] (1) Before actual operation, the operator performs the target object grasping task in the safety simulation through the hand controller to confirm the operator's safety control coefficient. This safety control coefficient will be used by the dynamic weight allocator to allocate the operator's control weight; (2) After the safety simulation is completed, the operator will input operation instructions through the hand controller and control the slave robot arm to actually operate under the safety control coefficient. The slave operation environment information is visually fed back to the master display. The slave computer will also perform target recognition based on the visual information provided by the camera. The autonomous controller will use this to complete trajectory planning and generate robot autonomous instructions. The shared controller will share and fuse the operation instructions and autonomous instructions according to the control weight; (3) After long-term operation, the EEG signal is input into the PoT model in the host computer. The PoT model calculates the operator's PoT and sends it to the dynamic weight allocator of the slave to realize the dynamic update of the shared control weight based on the operator's performance and update the operator's shared control weight under long-term operation.

[0043] (1) Safety simulation confirms the safety control coefficient:

[0044] Before actual operation, due to varying levels of proficiency with the teleoperation system among different operators, a safety simulation was conducted to ensure robot safety and obtain the safety control coefficient K before the operator's actual operation. s Specifically, step (1) includes:

[0045] (a) Constructing a secure simulation environment using Matlab, such as Figure 2 As shown, it includes a robotic arm with an end effector, a target object, and a target position;

[0046] (b) Perform the grasping task. The operator uses a hand controller to manipulate the robotic arm in the simulation environment to grasp the target object to the target position. Three operation performance indicators are statistically analyzed: operation time, path length, and path smoothness.

[0047] (c) Calculating Operator Performance (PoT) in Safety Simulation SIM Its calculation formula is:

[0048] PoT SIM =∑W i P i

[0049] In the formula, i = 1, 2, 3, P i W is the i-th normalized operational performance metric. i is the weighting coefficient corresponding to the i-th performance indicator;

[0050] (d) Output safety control coefficient Ks = PoT SIM Ks∈[0,1];

[0051] (2) Shared control in actual operation:

[0052] In actual operation, the robot's input instructions consist of two parts: the operator's operation instructions U... h And the robot's autonomous commands U r The operation instructions come from the input of the master hand controller, and the autonomous instructions are the result of the autonomous controller's path planning based on the target recognition results. Specifically, step (2) includes:

[0053] (a) The LineMod algorithm identifies 3D target objects in space and obtains their poses. The algorithm can be divided into two stages: template rendering and image matching. In the template rendering stage, RGB-D images of the target object are acquired from multiple angles and scales. In the image matching stage, the similarity between the offline trained template and the test image is measured by sliding a window to determine whether the target object has been detected.

[0054] (b) The RRT algorithm performs path planning and generates autonomous instructions. The algorithm flow is as follows: Figure 3 As shown;

[0055] (c) Operational instructions and autonomous instructions are shared and integrated, as follows:

[0056] U c =K s U h +(1-K s )U r K s ∈[0,1]

[0057] (3) After a long period of operation, implement dynamic updates of shared control weights based on PoT:

[0058] After prolonged operation, the operator's experience and operational status may change, necessitating further online evaluation of these changes to dynamically update the human-machine shared control weights. By employing an Emotiv EPOC 14-channel EEG analyzer to acquire 14 channels of EEG signals, a backpropagation (BP) neural network was used to model the Potentially Oriented Tract (PoT), as shown in the model below. Figure 4 As shown, specifically, step (3) includes:

[0059] (a) EEG signal acquisition: Emotiv EPOC 14-channel EEG instrument was used to acquire EEG signals and obtain 14 channels of raw EEG signals. The software development kit (SDK) provided by Emotiv was used to filter the raw signals and extract features to obtain the 14 channels of EEG signals required by the PoT model.

[0060] (b) The model is based on a BP neural network, which includes an input layer, a hidden layer, and an output layer. The operator's 14-channel EEG signal is used as the input, and the output layer data comes from the grasping task performed in the security simulation.

[0061] When calculating neuron transmission in each layer:

[0062]

[0063] In the formula, and For the activation and bias of the j-th neuron in the i-th layer, Let be the weights from the k-th neuron in layer (j-1) to the i-th neuron in layer j. The cost function of the output layer is as follows:

[0064]

[0065] In the formula, PoT and PoT* represent the actual output and expected output of the output layer, respectively. The weight update function is as follows:

[0066]

[0067] In the formula, t is the number of iterations, η is the learning rate of the neural network, and δ i O represents the learning error of the i-th neuron in the current layer. k This represents the output of the k-th neuron in the upper layer. The output PoT of the neural network. BP as follows:

[0068] PoT BP =f n (x p )

[0069] In the formula, x p f is a 14-channel EEG signal vector. n This is the trained neural network model.

[0070] (c) Calculate the shared control weight K based on PoT. h :

[0071]

[0072] In the formula, H and L are the upper and lower thresholds of the operator's performance, respectively, as follows:

[0073] ① When K h When the threshold is 0, it means the operator's performance is poor and below the lower threshold. The operation mode is autonomous control mode, and the output command of the shared controller is the autonomous command of the robotic arm. At this time, the operator does not participate in the control of the robotic arm, and the robotic arm performs autonomous control based on visual information.

[0074] ② When 0 < K h When K < 1, it indicates good operator performance, falling between the upper and lower thresholds. The operation mode is a human-machine shared control mode, where the shared controller outputs a fusion of operator commands and robotic arm autonomous commands. In this mode, when K h When K = 0.5, the operator and the robotic arm each account for half of the control weight. As K... h As the value increases, the control weight held by the operator also increases.

[0075] ③ When K h When the threshold is 1, it means the operator performs well and exceeds the upper threshold. The operation mode is manual control mode, and the output command of the shared controller is the operator's operation command. At this time, the robotic arm is completely controlled by the operator, that is, the operator controls the robotic arm through the hand controller.

[0076] (d) Operational instructions and autonomous instructions are shared and integrated, as follows:

[0077] U c =K h U h +(1-K h )U r K h ∈[0,1]

[0078] (e) The robotic arm receives the shared control command U c And reach the final target location.

[0079] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A safety sharing control system based on operator performance, characterized in that: Includes a master teleoperation system, a slave robotic arm system, and a communication module; The master-end teleoperation system includes the operator's electroencephalogram (EEG) signal measurement, hand controller operation input, and host computer software. The host computer includes a human-computer interaction interface, a safety simulation for protecting the robot's safety, and a Potentially Tunneling (PoT) model. The slave robotic arm system includes a robotic arm, a vision camera, and lower-level software. The lower-level software includes a target recognition algorithm, an autonomous controller for generating autonomous robot commands, and a shared controller for dynamically allocating human-machine control weights. The communication module enables communication between the master teleoperation system and the slave robotic arm system. The measurement of the operator's brainwave signals in the master-end remote operating system is achieved by using an electroencephalogram (EEG) device to collect brainwave signals and obtain raw brainwave signals. The hand controller operation input is obtained by using the hand controller to input the operation command U from the operator. h ; The human-computer interaction interface includes information feedback output from the end vision and robot status, as well as interactive operations such as starting a task, selecting manual / shared / autonomous mode, and ending a task. The safety simulation is used to confirm the safety control coefficient K before the operator actually performs the operation. s The specific steps are as follows: (a) Construct a simulation environment, including a robotic arm with an end effector, a target object, and a target location; (b) Perform the grasping task. The operator uses a hand controller to manipulate the robotic arm in the simulation environment to grasp the target object to the target position. Three operation performance indicators are statistically analyzed: operation time, path length, and path smoothness. (c) Calculating Operator Performance (PoT) in Safety Simulation SIM Its calculation formula is: can SIM =ΣW i P i In the formula, i = 1, 2, 3, P i W is the i-th normalized operational performance metric. i is the weighting coefficient corresponding to the i-th performance indicator; (d) Output safety control coefficient Ks = PoT SIM Ks∈[0,1]; The PoT model utilizes electroencephalogram (EEG) signals to indirectly acquire, in real-time, operator performance (PoT) during prolonged work sessions through a backpropagation neural network model. BP The neural network consists of an input layer, hidden layers, and an output layer. The operator's electroencephalogram (EEG) signals are used as input, and the model outputs a PoT (Potentially Transmitted Twist). BP During model training, data from security simulations used to perform grabbing tasks were employed. The robotic arm described in the slave robotic arm system is a slave entity that replaces humans to perform tasks in scenarios that are inaccessible or harmful to humans. The vision camera provides visual information about the working environment of the slave end to be fed back to the master end, as well as target recognition for the autonomous controller of the robotic arm. The target recognition algorithm is used to identify 3D target objects in space and obtain their poses. The LineMod algorithm is used for recognition and localization. The algorithm process is divided into two stages: template rendering and image matching. In the template rendering stage, RGB-D images of the target objects are acquired from multiple angles and at multiple scales. In the image matching stage, the similarity between the offline trained template and the test image is measured by sliding a window to determine whether the target object has been detected. The autonomous controller performs autonomous path planning based on the target recognition results, generates robot autonomous commands Ur, and adopts the fast expanding random tree method. This algorithm takes the starting position of the search as the root node, and then generates a random expanding tree by randomly sampling and adding child nodes. When the child node of the random tree enters the target area, the path from the starting position to the target position is obtained. The shared controller integrates the operator's commands and the robot's autonomous commands to obtain the robot's final shared command Uc. Its calculation is divided into two phases: a safety protection period and a post-long-term operation period. The safety protection period is based on the operational performance during safety simulation, while the post-long-term operation period is based on the operational performance during actual operation using EEG signals. The specific steps are as follows: 1) The safety protection period is calculated based on the safety control coefficient Ks obtained from the safety simulation: K h =K s 2) After prolonged operation, based on the operator performance of the PoT model. BP The calculation yielded: In the formula, H and L are the upper and lower thresholds of the operator's performance, respectively; 3) Output shared control commands: IN c =K h IN h +(1-K h )IN r ,K h ∈[0,1].

2. The application method of a safety sharing control system based on operator performance according to claim 1, characterized in that: Includes the following steps: (1) Before the actual operation, the operator performs the target object grabbing task in the safety simulation through the hand controller to confirm the operator's safety control coefficient. This safety control coefficient will be used by the dynamic weight allocator to allocate the operator's control weight. (2) After the safety simulation is completed, the operator will input the operation command through the hand controller and control the actual operation of the slave robot arm under the safety control coefficient. The slave operation environment information is displayed to the master end through visual feedback. The slave machine will also perform target recognition based on the visual information provided by the camera. The autonomous controller will use this to complete trajectory planning and generate robot autonomous commands. The shared controller will share and fuse the operation command and autonomous command according to the control weight. (3) After a long period of operation, the EEG signal is input into the PoT model in the host computer. The PoT model calculates the operator's PoT and sends it to the dynamic weight allocator at the slave end to realize the dynamic update of the shared control weight based on the operator's performance and update the shared control weight of the operator under long-term operation.

Images