A robot teleoperation collision avoidance method based on human arm motion prediction

Abstract

The application claims a kind of robot teleoperation collision avoidance method based on human arm motion prediction, comprising the following processes: establishing human arm model, human arm model is used to predict the motion trajectory of slave human arm, and is sent to three-dimensional graphics visualization scene. Establish robot model, robot model simulates the motion trajectory of teleoperation robot according to remote operation master end data, and is sent to three-dimensional graphics visualization scene. Establish three-dimensional graphics visualization scene, three-dimensional graphics visualization scene is used to display the motion trajectory predicted by human arm model, and display the motion trajectory simulated by robot model. The human arm model of the method simulates the motion trajectory of human arm according to the feedback of slave visual module, the motion trajectory of human arm and robot that the operator of master end sees through three-dimensional graphics visualization scene is the real-time motion trajectory of slave human arm and robot, so that the action of operator can be continuous, and the safety threat to slave end environment due to time delay can be avoided.

Description

Technical Field

[0001] This invention relates to the field of teleoperation technology, and more specifically, to a collision avoidance method for robot teleoperation based on human arm motion prediction. Background Technology

[0002] Network-based robot teleoperation technology plays a crucial role in projects such as telemedicine, aerospace exploration, and deep-sea exploration. In teleoperation, the master end is the operator's control end, and the slave end is the controlled end of the robot being operated. When the slave robot needs to coordinate with a collaborator, the physical distance between the master operator and the slave robot / collaborator, coupled with network latency, makes it difficult for the operator to accurately perceive the slave environment. This leads to a "motion-wait" state, causing discontinuous movement or system instability, resulting in operational failure and posing a significant safety threat to the slave environment. Existing methods cannot address the safety issues posed by dynamic elements such as human arm movements to robot teleoperation systems.

[0003] Therefore, how to solve the problem of motion discontinuity caused by visual delay for operators is the technical problem that this application aims to solve. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the purpose of this invention is to provide a robot teleoperation collision avoidance method based on human arm motion prediction. This method uses a three-dimensional graphical visualization scene to simulate the motion of the human arm and the robot in real time, thus solving the safety problems caused by dynamic objects such as human arm motion to the robot teleoperation system in existing technologies.

[0005] To achieve the above objectives, the following technical solution is provided:

[0006] A collision avoidance method for robot teleoperation based on human arm motion prediction includes the following steps:

[0007] St10. Establish a human arm model. The human arm model is used to predict the motion trajectory of the human arm at the end and send it to the 3D graphics visualization scene.

[0008] St20. Establish a robot model. The robot model simulates the motion trajectory of the teleoperated robot based on the data from the teleoperation master terminal and sends it to the 3D graphics visualization scene.

[0009] St30. Establish a 3D graphical visualization scene. The 3D graphical visualization scene is used to display the predicted motion trajectory of the human arm model and the simulated motion trajectory of the robot model.

[0010] In summary, the above technical solution has the following beneficial effects: The human arm model in this method simulates the human arm's motion trajectory based on feedback from the slave-end vision module. Correspondingly, the slave end is equipped with a vision module, such as a vision sensor or infrared sensor coupled to the human arm model, for detecting the human arm's motion. To reduce motion discontinuity caused by network latency, the human arm model predicts the human arm's motion trajectory through feedback from the vision module, allowing the operator at the master end to see the predicted trajectory in a 3D graphical visualization scene. The robot model simulates its motion trajectory based on data from the teleoperated master end. The motion trajectories of the human arm and robot seen by the operator at the master end through the 3D graphical visualization scene are the real-time motion trajectories of the human arm and robot at the slave end, ensuring continuous operator actions and avoiding safety threats to the slave-end environment due to latency. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the modular framework of a robot teleoperation collision avoidance method based on human arm motion prediction;

[0012] Figure 2 A schematic diagram illustrating the predicted trajectory of human arm movement;

[0013] Figure 3 This is a schematic diagram of the arm angle;

[0014] Figure 4 This is a schematic diagram of a 12-dimensional simulation of multi-joint pose.

[0015] Figure 5 This is a schematic diagram of the 8-dimensional simulation used in this invention;

[0016] Figure 6 This is a flowchart illustrating a robot teleoperation collision avoidance method based on human arm motion prediction.

[0017] Figure 7 This is a schematic diagram of the hand trajectory prediction results;

[0018] Figure 8 This is a schematic diagram of the arm angle prediction results;

[0019] Figure 9 This is a schematic diagram comparing the relative errors of hand movement trajectories.

[0020] Figure labels: 10, human arm model; 11, kinematic model; 12, hand trajectory prediction model; 13, anthropomorphic arm configuration prediction model; 20, robot model; 30, 3D graphics visualization scene; 40, video communication module. Detailed Implementation

[0021] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings. In the following embodiments, many details are described to facilitate a better understanding of the present application. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other materials or methods. In some cases, certain operations related to the present application are not shown or described in the specification. This is to avoid obscuring the core parts of the present application with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; they can fully understand the related operations based on the description in the specification and general technical knowledge in the art.

[0022] Furthermore, the features, operations, or characteristics described in the specification can be combined in any suitable manner to form various embodiments. At the same time, the steps or actions in the method description can be rearranged or adjusted in a manner obvious to those skilled in the art. Therefore, the various orders in the specification and drawings are only for the clear description of a particular embodiment and do not imply a necessary order, unless otherwise stated that a particular order must be followed.

[0023] A collision avoidance method for robot teleoperation based on human arm motion prediction includes the following steps:

[0024] St10, establish a human arm model 10. The human arm model 10 is used to predict the motion trajectory of the human arm at the end and send it to the three-dimensional graphics visualization scene 30.

[0025] like Figure 1 As shown, the human arm model 10 includes a kinematic model 11, a hand trajectory prediction model 12, and an anthropomorphic arm configuration prediction model 13. The kinematic model 11 is used to connect the bones of the human arm model 10 through joint rotation. The hand trajectory prediction model 12 is used to predict the hand movement trajectory. The anthropomorphic arm configuration prediction model 13 is used to calculate the human arm configuration corresponding to the hand trajectory point based on the hand movement trajectory and bone data.

[0026] Kinematic model 11 is a 7-DOF kinematic model of the SRS structure. Kinematic model 11 treats each bone of the human arm as a rigid body link and connects them through rotational joints to establish a typical 7-DOF kinematic model of the SRS structure (shoulder swing and lifting, upper arm rotation, elbow flexion, forearm rotation, wrist flexion and swing), with joints including shoulder joint S, elbow joint E and wrist joint W.

[0027] St11, predict hand movement trajectory using hand trajectory prediction model 12.

[0028] Predicting hand movement trajectories involves the following processes:

[0029] St111: Collect hand movement trajectory data and form an observation dataset.

[0030] Specifically, the slave end is equipped with a visual module such as a visual sensor or infrared sensor coupled to the human arm model 10 for detecting the movement of the human arm. For example, the Vicon infrared motion capture system can collect real-time images of the scene and the movement state of the human body, and transmit them to the hand trajectory prediction model 12 via the network.

[0031] St112 and hand trajectory prediction model 12 predict hand movement trajectories using the observed dataset and form a prediction dataset. The hand movement trajectory data uses normal symbols and symbols with ∧ to represent observed data and predicted data, respectively. The observed dataset consists of hand movement trajectory points from N frames prior to a certain moment, and is represented as follows:

[0032]

[0033] In equation (1), X k is the observation dataset at time k; [;] is the time series of the model input; N is a natural constant representing the range (number of frames) of the observation dataset, 6 indicates that the hand movement trajectory includes 6 dimensions, and 6N indicates 6*N frames of dimensions.

[0034] The prediction dataset consists of hand motion trajectory points M frames after a certain moment. The prediction dataset is represented as follows:

[0035]

[0036] In equation (2), Let be the predicted dataset at time k; [;] is the time series output by the model; M is a natural constant representing the range (number of frames) of the predicted dataset; For the prediction of the Mth frame at time k.

[0037] The predicted time for the prediction dataset is between 200ms and 600ms. Preferably, the prediction dataset time is 400ms. Since the maximum measured delay of the remote ultrasonic robot is 200ms, the motion prediction time is set to 400ms. The observation dataset of the first 1000ms is used as input, and the prediction dataset of the last 400ms is output.

[0038] St113, the hand trajectory prediction model 12 combines the predicted data obtained each time with the observation dataset to form a new observation dataset, and then predicts the hand movement trajectory using the new observation dataset. Specifically, at a certain moment, after the hand trajectory prediction model 12 predicts one frame of data, it combines the predicted data of that frame with the N frames of observation dataset and uses it as the input of the hand trajectory prediction model 12 at that moment, until the M frames of prediction dataset are completed. Due to the time-varying nature of human arm movement, the hand trajectory prediction model 12 can be represented by a time-varying function f:

[0039]

[0040] like Figure 2 As shown, human arm movements are complex and highly nonlinear, with strong temporal relationships. Therefore, an N-to-1 RNN (Recurrent Neural Network) hand trajectory prediction model 12 is used to acquire N frames of historical hand movement trajectories and predict and output one frame of hand movement trajectory. The hand trajectory prediction model 12 continuously adds the newly predicted frame of hand movement trajectory as input, and then predicts the next frame, repeating the prediction until a complete set of M frames is obtained. The advantage of this hand trajectory prediction model 12 is that it allows for greater flexibility in online adaptation; once new observations are available, the model can be adaptively updated.

[0041] St12, the anthropomorphic arm configuration prediction model 13 is used to calculate the predicted arm configuration for the hand movement trajectory, thus obtaining the predicted movement trajectory of the arm from the end. The hand movement trajectory consists of several frames of hand movement trajectory points. This method first predicts the hand movement trajectory, and then calculates the arm configuration of each frame of hand movement trajectory points to complete the arm prediction. The prediction process for one frame requires data from a total of eight dimensions: six dimensions of the hand movement trajectory, upper arm length, and forearm length. For example... Figure 4 As shown, Figure 4 Chinese k e k w k and h k These represent the observed spatial position sequences of the shoulder joint (S), elbow joint (E), wrist joint (W), and hand, respectively. and To predict the position sequence, if a single frame's localization prediction is completed using shoulder, elbow, wrist, and hand coordinates, 12 dimensions need to be calculated. This results in high training and computation costs, poor prediction progress, and the inability to directly output the angles of each joint in the human arm. Furthermore, additional hardware is required on the slave end to obtain the real-time positions of the shoulder joint (S), elbow joint (E), and wrist joint (W). Figure 5As shown, this method reduces the dimensionality of input and output through the anthropomorphic arm configuration prediction model 13. It only requires the spatial positions of the hand (x, y, z), the attitude angles of the end effector rotation around the x-axis (α), y-axis (β), and z-axis (γ), and considers the influence of the upper and lower arm lengths on the motion prediction progress, thus reducing the computational load, shortening the prediction time, and improving the prediction accuracy. Furthermore, the hand trajectory prediction model 12 is based on a recurrent neural network (RNN), and the anthropomorphic arm configuration prediction model 13 is based on a multilayer perceptron (MLP).

[0042] Skeletal data includes upper arm length, lower arm length, and arm angle. For example... Figure 3 As shown, a seven-DOF human arm model has countless arm configurations when the hand pose is fixed, i.e., countless inverse solutions. Let the plane determined by the three points of shoulder joint S, elbow joint E, and wrist joint W be the arm plane. When the angle of joint q3 in one of the inverse solutions is 0, we let the elbow joint be E0. The arm plane S-E0-W is the reference plane, and the arm angle is defined as the angle between the reference plane and the arm plane.

[0043] St121, the anthropomorphic arm configuration prediction model 13 predicts the arm angle based on the hand movement trajectory, upper arm length, and lower arm length, and then obtains the human arm configuration corresponding to the hand trajectory through the inverse kinematics of the arm angle. The data of upper arm length, lower arm length, and hand movement trajectory are processed to obtain the input vector m = [x, y, z, α, β, γ, l1, l2]. T And predict the arm angle corresponding to each frame's hand motion trajectory point. The anthropomorphic arm configuration prediction model 13 can be represented as follows:

[0044] Specifically, the anthropomorphic arm configuration prediction model 13 uses MLP to predict the arm configuration based on the arm angle value corresponding to the hand motion trajectory in each frame. The mathematical model is as follows:

[0045]

[0046] In equation (4), Let l1 and l2 be the predicted arm angle value of the human arm at the point of the hand's motion trajectory at time k, and l1 and l2 represent the lengths of the upper arm and lower arm, respectively. MLP represents the learned [h] k [l1,l2] T to arm corner The mapping relationship is established. Learning methods, including but not limited to Gaussian processes, linear regression, and neural networks, are employed to learn the hand pose at the end of the human arm and the upper and lower arm length vectors *m* and arm angles. The mapping relationship between them is used to establish a humanoid arm configuration prediction model 13.

[0047] St20, establish robot model 20. Robot model 20 simulates the motion trajectory of the remote-operated robot based on the remote operation master data and sends it to the three-dimensional graphic visualization scene 30.

[0048] like Figure 6 As shown, the teleoperation master data includes the robot's Cartesian pose data. The robot model 20 uses the robot's inverse kinematics to solve the Cartesian pose data in real time, thereby obtaining the robot's joint angles. The robot's joint angles are then used as inputs to drive the simulated teleoperated robot's motion.

[0049] St30, establish a three-dimensional graphic visualization scene 30, which is used to display the motion trajectory predicted by the human arm model 10 and the motion trajectory simulated by the robot model 20.

[0050] St31. Humans and robots are modeled using 3D software. The models of humans and robots are assembled using nodes. The models are then imported into a 3D graphical visualization scene 30 to form a virtual robot and a virtual human arm.

[0051] The data of St32 and human arm model 10 are imported into the 3D graphics visualization scene 30 in real time to control the real-time movement of the human model.

[0052] The data of St33 and robot model 20 are imported into the 3D graphics visualization scene 30 in real time to control the real-time movement of the robot model.

[0053] The 3D graphical visualization scene 30 is connected to the video communication module 40, which is used to acquire environmental video from the slave end. By combining the predicted display of the 3D visualization scene of the human arm and robot with the ordinary video communication scene, the visual presence of the operator at the master end of the remote control system is enriched, allowing the operator to predict collisions in advance and effectively avoid them. Safe control commands are then sent to the slave robot, avoiding the collision safety hazards caused by time delays leading to lag in visual feedback.

[0054] In this method, the human arm model 10 simulates the human arm's motion trajectory based on feedback from the slave-end vision module. Correspondingly, the slave end is equipped with a vision module, such as a vision sensor or infrared sensor, coupled to the human arm model 10 to detect the human arm's motion. To reduce motion discontinuity caused by network latency, the human arm model 10 predicts the human arm's motion trajectory through feedback from the vision module, allowing the operator at the master end to see the predicted trajectory on the 3D graphical visualization scene 30. The robot model 20 simulates its motion trajectory based on data from the teleoperated master end. The motion trajectories of the human arm and robot seen by the operator at the master end through the 3D graphical visualization scene 30 are the real-time motion trajectories of the human arm and robot at the slave end, ensuring continuous operator action and avoiding safety threats to the slave-end environment due to latency.

[0055] The predictive effectiveness of this invention was evaluated through the following experiments. A Vicon infrared motion capture system was constructed to collect motion trajectory data sequences of the patient's right arm and hand during scanning, with a collection period of 40ms (1 frame). Sixteen subjects were randomly selected, including 8 males and 8 females. The subjects were aged 18-38 years. The height of the male subjects ranged from 165-193cm, and their weight ranged from 52-101kg; the height of the female subjects ranged from 151-180cm, and their weight ranged from 39-70kg.

[0056] The scanning bed was placed at the center of the infrared motion capture system's field of view. Subjects, dressed in black motion capture suits, lay supine on the operating table. Light reflection markers were fixed at the shoulder, elbow, wrist, and hand. The subjects lay upright, simulating possible arm movements during the scan. During data collection, the starting and ending points of each movement could be freely adjusted. The duration of each movement was maintained between 1 and 3 seconds. Twenty sets of arm movement data were collected from each subject at different starting and ending points, resulting in 320 sets of arm movement trajectory data sequences from 16 subjects. Static joint data was collected to learn the arm angles corresponding to the hand positions, training a humanoid arm configuration prediction model. Subjects gripped the robotic arm's end handle and dragged it with zero force to change hand positions, maintaining the most comfortable arm configuration. Arm configurations corresponding to each gripping position were collected, including the shoulder joint, elbow joint, wrist joint, hand position, and upper and lower arm lengths. Eighty sets of data were determined for each subject, resulting in a total of 1280 sets of data. Linear was chosen as the activation function for the first hidden layer of the prediction model, ReLU was chosen as the activation function for the second hidden layer, and Adam was chosen as the optimizer. The model was iterated 3000 times, and a static joint data training set was input. The anthropomorphic arm configuration prediction model 13 was obtained by training with Python 3.6.

[0057] Changes between hand trajectory prediction data and observed data are as follows Figure 7As shown, the maximum absolute error of the predicted hand trajectory position is 3.95 mm, and the predicted trajectory is basically consistent with the observed trajectory.

[0058] Input the hand trajectory prediction dataset into the anthropomorphic arm configuration prediction model 13, and obtain the comparison results between the predicted values ​​and the actual values, such as... Figure 8 As shown, the average error between the predicted arm angle and the actual arm angle is 0.0532 rad.

[0059] Compared to the prediction using a 12-dimensional human arm model (model 10), which has a high input / output dimension, the model could not achieve optimal results using the same training parameters as the prediction model in this paper. Therefore, the number of hidden layer neurons in this model was adjusted to 1024, and RMSE was used as the loss function. After 10,000 iterations, the loss function was approximately 0.2, resulting in a trained prediction model. When inputting the observed dataset for prediction, the maximum absolute error of the predicted hand trajectory position was 4.841 mm. The relative motion error of the M-th frame is defined as the ratio of the error at that moment to the motion amplitude, used to measure the performance of the prediction model. The results are as follows: Figure 9 As shown. When predicting within 10 frames, the relative motion error of the human arm model 10 of the present invention is within 8%, and for prediction results of more than 25 frames, it remains within 10%. The relative motion of the human arm model 10 of the present invention is superior to that of the 12-dimensional human arm model 10.

[0060] This invention simplifies the input and output dimensions of the prediction model by incorporating human arm kinematics. It also uses the lengths of the upper and lower arms as the prediction input, optimizing the impact of different arm length parameters on motion prediction accuracy. In a 400ms motion prediction, the average absolute error of the arm angle prediction is 0.052 rad, and the relative motion error of the hand trajectory prediction position is within 8%. Compared to a 12-dimensional prediction scheme, this achieves a more effective human arm motion prediction over a longer period. Based on this, a human-computer collision avoidance method based on human arm motion prediction is proposed. By constructing a virtual simulation display of the patient's arm movement state and using a 3D graphical visualization scene for real-time collision detection, this method avoids human-computer collisions caused by delays in visual presence, thus improving the safety of the teleoperation system.

[0061] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principle of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for collision avoidance in robot teleoperation based on human arm motion prediction, characterized in that, Includes the following processes: A human arm model (10) is established, which is used to predict the motion trajectory of the end human arm and send it to the three-dimensional graphics visualization scene (30); A robot model (20) is established, which simulates the motion trajectory of the teleoperated robot based on the teleoperation master terminal data and sends it to the three-dimensional graphic visualization scene (30); A three-dimensional graphic visualization scene (30) is established, which is used to display the motion trajectory predicted by the human arm model (10) and the motion trajectory simulated by the robot model (20); The human arm model (10) includes a kinematic model (11), a hand trajectory prediction model (12), and an anthropomorphic arm configuration prediction model (13); The kinematic model (11) is used to connect the bones of the human arm model (10) through joint rotation, and the kinematic model (11) is a 7-DOF kinematic model (11) with an SRS structure; The hand trajectory prediction model (12) is used to predict the hand movement trajectory, and the prediction of the hand movement trajectory includes the following process: collecting hand movement trajectory data and forming an observation dataset; the hand trajectory prediction model (12) predicts the hand movement trajectory through the observation dataset and forms a prediction dataset. The anthropomorphic arm configuration prediction model (13) is used to calculate the human arm configuration corresponding to the hand trajectory point based on the hand movement trajectory and bone data. After the hand movement trajectory is predicted by the hand trajectory prediction model (12), the human arm configuration predicted by the anthropomorphic arm configuration prediction model (13) is used to calculate the human arm configuration of the predicted hand movement trajectory, thereby obtaining the predicted movement trajectory of the human arm from the end. The hand trajectory prediction model (12) is an N-to-1 structured RNN model. The hand trajectory prediction model (12) combines the predicted data obtained each time with the observation dataset as a new observation dataset, and then predicts the hand movement trajectory through the new observation dataset. The predicted time of the prediction dataset is between 200ms and 600ms. The anthropomorphic arm configuration prediction model (13) is established based on multilayer perceptron (MLP), and the skeletal data includes upper arm length, lower arm length and arm angle. The anthropomorphic arm configuration prediction model (13) predicts the arm angle based on the hand movement trajectory, upper arm length and lower arm length, and then obtains the human arm configuration corresponding to the hand trajectory through the inverse kinematics of the arm angle.

2. The human arm motion prediction based robot teleoperation collision avoidance method of claim 1, wherein, The teleoperation master data includes the robot's Cartesian pose data. The robot model (20) uses the robot's inverse kinematics to solve the Cartesian pose data in real time, thereby obtaining the robot's joint angles. The robot's joint angles are used as input to drive the simulated teleoperated robot's movement.

3. The human arm motion prediction based robot teleoperation collision avoidance method of claim 1, wherein, Humans and robots are modeled using 3D software. The models of humans and robots are assembled using nodes and then imported into the 3D graphics visualization scene (30). The data of the human arm model (10) is imported into the three-dimensional graphics visualization scene (30) in real time to control the real-time movement of the human model; Data of the robot model (20) is imported into a three-dimensional graphics visualization scene (30) in real time to control real-time movement of the model of the robot.

4. The human arm motion prediction based robot teleoperation collision avoidance method of claim 3, wherein, The three-dimensional graphics visualization scene (30) comprises a video communication module (40) for collecting environment video from an end.

Images