A human vision-driven control method for an upper limb assistive robot

Abstract

The present invention is applicable to the field of assistive robot control, and provides a human vision-driven upper limb assistive robot control method, the method comprising: a 3D gaze point estimation step, a head posture estimation step, a gaze-based intention recognition step, a timing intention optimization step, and an execution step. The method of the present invention combines artificial intelligence machine learning technology with human behavior laws, uses a deep neural network to extract the intrinsic relationship between gaze and intention and perform intention recognition, and after obtaining gaze point position information and grasping intention information, the assistive robot uses gaze to complete trajectory planning and realize auxiliary functions.

Description

Technical Field

[0001] The present invention belongs to the technical field of assistive robot control, and in particular relates to a human vision-driven upper limb assistive robot control method. Background Art

[0002] Upper limb function is an important ability in human daily life, but some elderly people or people with upper limb impairment have limited upper limb function and cannot independently complete daily functions such as grasping. Assistive robots are a solution to compensate and enhance human upper limb function. In order to effectively use these assistive robots, an effective assistive robot control method is needed that can achieve: 1. Effectively identify the user's grasping intention, judge the user's action and execute it; 2. Accurately locate the position of the user's target and complete the grasping.

[0003] At present, the commonly used upper limb assistive robot control schemes include electromyography (EMG) control methods, electroencephalogram (EEG) control methods, and methods based on other limb movements. Among them, the EMG method / EEG method uses electromyography / electroencephalogram to identify the user's movement intention, such as identifying different gestures and different grasping movements, and then uses the recognition results to control auxiliary equipment such as intelligent prostheses to complete the movement. This type of method can be used to identify the user's grasping intention and movement intention. However, this type of method based on physiological electrical signals often faces the problems of noise interference, inaccurate physiological electrical signals, and low acceptance. In addition, this method can only identify the user's movement intention and realize the movement, but cannot locate the target's position information, and cannot transmit the target's position information to the upper limb assistive robot to complete the control. The method based on limb movements uses the user's healthy limbs to control the auxiliary robot equipment, such as using the feet to control the robot's movement direction and movement. However, this method is limited by the freedom of a person's healthy limbs. When controlling a robot to move in space and achieve grasping, control signals of multiple degrees of freedom are often required, which requires a large number of the user's healthy limbs and is difficult to achieve.

[0004] The human eye is the main means for people to obtain information, and the motor function of the human eye can be preserved even in patients with severe hemiplegia. In addition, human vision can locate the position of the target, reflect human intentions and guide hand movements. In addition, for users, controlling eye movements is simpler than controlling limb movements, so using human vision to assist robot control is an effective solution, which can significantly improve control efficiency and has a higher degree of market acceptance. Summary of the invention

[0005] In view of the above problems, the purpose of the present invention is to provide a human vision-driven upper limb assistive robot control method, aiming to solve the technical problems existing in the existing control methods based on physiological electrical signals and limb movements.

[0006] The present invention adopts the following technical solution:

[0007] The human vision-driven upper limb assistive robot control method is implemented based on an eye tracker, a depth camera, and a nine-axis inertial sensor, and the control method comprises the following steps:

[0008] Step S1, obtaining the user's sight direction through an eye tracker, and the intersection of the user's sight direction and the point cloud captured by the depth camera is the user's 3D gaze point;

[0009] Step S2: estimating the user's head posture through the nine-axis inertial sensor, obtaining the 3D gaze point coordinates of the user under different head postures and the scene image captured by the scene camera of the eye tracker;

[0010] Step S3, taking the 3D gaze point coordinates and the scene image as input, and the user's grasping intention and grasping target as output, to train the intention recognition neural network;

[0011] Step S4: input the scene image and 3D gaze point coordinates output by the eye tracker in real time into the intention recognition neural network to obtain continuous intention recognition results, describe the intention recognition results as a sequence, and use the long short-term memory network to fuse the past intention recognition results and the current intention recognition results to identify the most likely intention of the current user;

[0012] Step S5: Control the upper limb assistive robot to perform corresponding actions according to the most likely intention of the user.

[0013] Furthermore, the depth camera shaft is installed on the top of the eye tracker, and the inertial sensor is integrated into the eye tracker or the depth camera.

[0014] Furthermore, the specific process of step S1 is as follows:

[0015] The coordinates of two points on the user's line of sight measured in the eye tracker scene camera coordinate system. The line connecting these two points is the user's line of sight.

[0016] The calibration relationship between the depth camera and the eye tracker scene camera is used to convert the user's line of sight in the scene camera coordinate system from the scene camera coordinate system to the depth camera coordinate system. The intersection of the line of sight in the depth camera coordinate system and the point cloud captured by the depth camera is the user's 3D gaze point.

[0017] Furthermore, in step S2, the acceleration, angular velocity and magnetic field strength data measured by the nine-axis inertial sensor are used to obtain the system state equation and observation equation, and the head trajectory of the user is obtained by using the Kalman filter.

[0018] Furthermore, in step S3, the scene image is subjected to target detection and saliency detection to output a saliency grayscale map, the 3D gaze point coordinates are transformed to generate a gaze map, and the saliency grayscale map and the gaze map are used as inputs of the intent recognition neural network.

[0019] Furthermore, in step S4, the training process of the LSTM network is as follows: using the current intention recognition result and the intention recognition results of multiple previous moments as input, and the user's true intention as output to train the LSTM network,

[0020] The beneficial effects of the present invention are: using the method of the present invention, the user does not need to be trained, and the user's vision can directly provide the upper limb assistive robot system with target position information and the user's grasping intention; the method combines artificial intelligence machine learning technology with human behavior laws, uses deep neural networks to extract the intrinsic relationship between gaze and intention and perform intention recognition, and after obtaining the gaze point position information and grasping intention information, the assistive robot uses gaze to complete trajectory planning and realize auxiliary functions. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a magnetic flow chart of a human vision-driven upper limb assistive robot control method provided by an embodiment of the present invention;

[0022] Figure 2 It is a schematic diagram of a device integrating an eye tracker, a depth camera, and a nine-axis inertial sensor;

[0023] Figure 3 is a schematic diagram of 3D gaze estimation provided by an embodiment of the present invention;

[0024] Figure 4 is a schematic diagram of the network structure of intent recognition provided by an embodiment of the present invention;

[0025] Figure 5 It is a schematic diagram of a temporal fusion intention model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0027] In the application of upper limb assistive robots, effective human-computer interaction and intention recognition methods determine the accuracy of the equipment and the comfort of the user. The current control methods for upper limb assistive robots mainly include methods based on physiological electrical signals and limb movements. Although the method based on physiological electrical signals can effectively identify the user's grasping intention, it cannot provide the position information of the target. The method based on limb movements cannot effectively control assistive robots with multiple degrees of freedom. The user's vision can effectively provide the position information of the target and potentially reflect the grasping intention. Therefore, the invention proposes a vision-driven upper limb assistive robot control scheme, which can solve the problems existing in the existing control methods based on physiological electrical signals and limb movements.

[0028] In order to illustrate the technical solution of the present invention, a specific embodiment is provided below for illustration.

[0029] Figure 1 The flow chart of the human vision-driven upper limb assistive robot control method provided by an embodiment of the present invention is shown. For the sake of convenience, only the part related to the embodiment of the present invention is shown.

[0030] like Figure 1 As shown, the human vision-driven upper limb assistive robot control method provided in this embodiment includes the following steps:

[0031] Step S1, 3D gaze point estimation step: obtain the user's gaze direction through an eye tracker, and the intersection of the user's gaze direction and the point cloud captured by the depth camera is the user's 3D gaze point.

[0032] The control method of the embodiment of the present invention is implemented based on an eye-tracker, a depth camera and a nine-axis inertial sensor. Figure 2 As shown, A, B, and C are respectively an eye tracker, a depth camera, and a nine-axis inertial sensor (IMU). The rotation axis of the depth camera B is installed on the top of the eye tracker A, and the inertial sensor C is integrated into the eye tracker A or the depth camera B. In the figure, the inertial sensor is integrated into the eye tracker. The eye tracker described in this embodiment is a binocular eye tracker.

[0033] The eye tracker can provide 2D gaze point coordinates based on the scene camera, but lacks the depth information of the object, so other sensors that can obtain the position of the object information are needed. Therefore, the present invention combines the depth camera and the nine-axis inertial sensor with the eye tracker to estimate the user's 3D gaze point.

[0034] When implementing this step, refer to Figure 3As shown in the schematic diagram. In the figure, the Camera coordinate system is the camera coordinate system, the Pixel coordinate system is the pixel coordinate system, and the Image physical coordinate system is the image physical coordinate system. G is the 3D gaze point in the camera coordinate system, Zc is the optical axis of the scene camera, and G' is the pixel coordinate in the image pixel coordinate system. g is the distance between G' and G. This step includes the following process:

[0035] S11. Coordinates of two points on the user's line of sight measured in the eye tracker scene camera coordinate system, where the line connecting the two points is the user's line of sight direction.

[0036] The pixel coordinates of the user’s gaze point measured by the eye tracker are ( p u gi , p v gi ). The user's line of sight can be represented by a straight line formed by two points. You can choose any two points and set the length of the line of sight to be 0m and 1m respectively. e p0, e p1. The coordinates of the two points are:

[0037]

[0038] e z g0 =0m, e z g1 =1m

[0039] in and are the x-axis and y-axis focal lengths of the eye tracker scene camera, Represents the pixel coordinate center of the image, and γ is the distortion parameter of the camera.

[0040] S12. Use the calibration relationship between the depth camera and the eye tracker scene camera to convert the user's line of sight direction in the scene camera coordinate system from the scene camera coordinate system to the depth camera coordinate system. The intersection of the line of sight direction in the depth camera coordinate system and the point cloud captured by the depth camera is the user's 3D gaze point.

[0041] The calibration relationship between the depth camera and the eye tracker scene camera can be measured by hardware design and is expressed as The view direction can be converted from the scene camera coordinate system to the depth camera coordinate system using the transformation matrix:

[0042]

[0043] Among them, the upper left subscript indicates the coordinate system to which the coordinate point belongs, and the two points are converted from the scene camera coordinate system to the depth camera coordinate system.

[0044] The intersection of the line of sight and the point cloud captured by the depth camera is the user's 3D gaze point, expressed as d p g Using this method, the three-dimensional coordinates of the user's gaze point in the camera coordinate system can be measured.

[0045] Step S2, head posture estimation step: estimate the user's head posture through a nine-axis inertial sensor (IMU), obtain the user's 3D gaze point coordinates under different head postures and the scene image captured by the scene camera of the eye tracker.

[0046] Since the user's 3D gaze point measured in step S1 is located in the depth camera coordinate system, and during the use of the eye tracker, the user's gaze point will change in the world coordinate system as the user's head moves, this step requires estimating the user's head posture.

[0047] In the upper limb assistive robot application scenario, the user wears an integrated device including a binocular eye tracker, a depth camera and a 9-axis IMU on his head. The 9-axis IMU is a general term for a magnetometer, accelerometer and gyroscope, which can measure magnetic field strength, acceleration and angular velocity data.

[0048] In the world coordinate system, the head posture is expressed as quaternion:

[0049] x=[ E q k ]=[q0,q1,q2,q3]

[0050] The upper left subscript E represents the earth coordinate system, and the time domain differential equation of the quaternion is:

[0051]

[0052]

[0053] where w x ,w y ,w z It is the three-axis angular velocity measured by the IMU, where b represents the IMU body coordinate system, which has been set before the IMU device leaves the factory, and k represents the kth moment of sampling, the same below.

[0054] Therefore, the state equation of the system can be constructed:

[0055] x(k)=Ax(k-1)+o(k)

[0056] in o(k) represents the process noise of the system, Δt is the sensor sampling period, and I is the identity matrix.

[0057] The gravity acceleration component in the IMU body coordinate system is:

[0058] in Represents the rotation matrix between the world coordinate system E and the IMU body coordinate system b. |g| represents the absolute value of the gravitational acceleration under the z-axis of the world coordinate system (the sky points to the earth).

[0059] Then the Jacobian matrix

[0060] Then the observation equation 1 of the system is:

[0061] m acc (k)=H1x(k)+v1(k)

[0062] m acc represents the acceleration measurement result, and v1(k) represents the acceleration measurement noise.

[0063] The components of the magnetic field strength in the IMU body coordinate system are:

[0064] |mag| represents the measured value of magnetic force in the world coordinate system.

[0065] Then the Jacobian matrix

[0066] Then the observation equation 2 of the system is:

[0067] m mag (k)=H2x(k)+v2(k)

[0068] m mag represents the magnetometer measurement result, v2(k) represents the magnetic measurement noise;

[0069] The state equation of the system can be established based on the aforementioned quaternion time-domain differential equation, and the two observation equations of the system can be established based on the above two Jacobian matrices. Finally, based on the state equation of the system and the two observation equations, the user's head posture can be obtained using the two-stage Kalman filter head trajectory.

[0070] The specific steps are as follows:

[0071] 1. Computational system prior state estimation

[0072] 2. Calculate the prior estimated covariance Where Q1 is the process noise covariance

[0073] 3. Calculate H1(k)

[0074] 4. Calculate Kalman gain Where R1 is the measurement noise covariance

[0075] 5. Calculate h1(k)

[0076] 6. Calculate the posterior state estimate

[0077] 7. Calculate the posterior error estimate covariance matrix P1(k) = (I-K1(k)H1(k))P - (k)

[0078] 8. Mag measurement results are achieved in the same way, and finally

[0079]

[0080] After obtaining the 3D gaze point coordinates of the camera coordinate system and the posture of the user's head, the 3D gaze point coordinates of the user under any head movement can be estimated. This coordinate can provide position information for the auxiliary robot system for grasping assistance.

[0081] Step S3, gaze-based intention recognition step: take the 3D gaze point coordinates and scene image as input, and the user's grasping intention and grasping target as output, and train the intention recognition neural network.

[0082] After obtaining the target's location information, it is necessary to determine whether the user has the intention to grasp the target. Only when there is the intention to grasp the target should the assistive robot take action to assist.

[0083] This step uses the 3D gaze point coordinates and the scene image obtained by the scene camera as input, and the user's grasping intention and grasping target as output to train the intention recognition neural network. The structure of the recognition neural network is as follows Figure 4 As shown:

[0084] The scene image is subjected to target detection and saliency detection to output a saliency grayscale map ( Figure 4 The first picture in the upper left of the middle), the gaze point is transformed to generate the gaze map ( Figure 4 The method of converting the fixation point into the fixation map is:

[0085] Let img be an image of size W×H, then:

[0086]

[0087]

[0088] Where (u gi ,vgi ) represents the coordinates of the gaze point, (x, y) represents the pixel of the image. σ is a set hyperparameter that can be adjusted.

[0089] Since the gaze image has the same frequency as the scene camera, a gaze image should correspond to multiple gaze points, so the gaze image g t for:

[0090] The saliency grayscale map is directly implemented using the existing algorithm (Hou X, Zhang L. Saliency detection: A spectral residual approach [C] / / 2007 IEEE Conference on computer vision and pattern recognition. Ieee, 2007: 1-8.)

[0091] The generated saliency grayscale map and gaze map are used as the input of the intention recognition neural network, and the true intention is used as the output. The intention recognition neural network is trained using the cross entropy loss function and the stochastic gradient descent method. After the intention recognition neural network training is completed, the scene image and gaze point output by the eye tracker in real time can be used to judge the user's grasping target and grasping intention in real time.

[0092] Step S4, intention timing optimization step: input the scene image and 3D gaze point coordinates output by the eye tracker in real time into the intention recognition neural network to obtain continuous intention recognition results, describe the intention recognition results as a sequence, and use the long short-term memory network to fuse the past intention recognition results and the current intention recognition results to identify the most likely intention of the current user.

[0093] In upper limb assistance applications, people's movements are continuous, and stable intention recognition results are required to effectively control the assistive robot to complete grasping assistance tasks. Not every intention recognition result will be executed.

[0094] When implementing this step, a time series optimization method for intent is adopted. Specifically, the intention recognition results of the user at different times are described as a sequence, and the long short-term memory network (LSTM) is used to fuse the past intention recognition results and the current intention recognition results to identify the most likely intention of the current user.

[0095] like Figure 5 As shown:

[0096] The current intent recognition result and the intent recognition results at multiple previous moments are used as input, and the user's true intention is used as output to train the LSTM model. After the model is trained, the model is used to fuse the temporal information of the intent for intent recognition.

[0097] This method can effectively improve the stability of intention recognition and avoid identifying incorrect grasping intentions due to sudden human gaze behavior, which can affect the results of intention recognition.

[0098] Step S5, execution step: controlling the upper limb assistive robot to perform corresponding actions according to the most likely intention of the user.

[0099] In specific implementation, the user's 3D gaze point is used to obtain the grasping target, and the motion trajectory of the upper limb assistive robot is planned. The robot is guided to move with the identified grasping target trajectory. There is no need to use complex algorithms to generate the motion trajectory of the robotic arm. When the robot moves to the target position according to the user's vision, the gaze-based grasping intention recognition and timing optimization are used to identify the user's grasping intention and complete the grasping action.

[0100] In the scheme of the present invention, a depth camera and an IMU are combined into an eye tracker, providing a method for measuring the user's 3D gaze point, and the gaze point can be converted into a gaze image, solving the problem that the gaze point and the image cannot be used for training the network at the same time. At the same time, a scene saliency grayscale image detection method using target detection and saliency detection is proposed, and combined with the intention recognition network model, the intention recognition based on gaze is realized, which can recognize the user's grasping intention and grasping target in real time.

[0101] In addition, in view of the possible instability of the output of the intention recognition neural network, which is unsafe for controlling the robot, the present invention also provides a method for integrating time series information to improve the stability and accuracy of intention recognition, thereby ensuring the accuracy of recognition. Finally, the use of gaze point planning to assist the robot arm / robot motion trajectory is realized, and the robot motion trajectory planning and control of the robot grasping action are completed by gaze.

[0102] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A human vision-driven upper limb assistive robot control method, characterized in that: The control method is implemented based on an eye tracker, a depth camera and a nine-axis inertial sensor, and the control method comprises the following steps: Step S1, obtaining the user's sight direction through an eye tracker, and the intersection of the user's sight direction and the point cloud captured by the depth camera is the user's 3D gaze point; Step S2: estimating the user's head posture through the nine-axis inertial sensor, obtaining the 3D gaze point coordinates of the user under different head postures and the scene image captured by the scene camera of the eye tracker; Step S3, taking the 3D gaze point coordinates and the scene image as input, and the user's grasping intention and grasping target as output, to train the intention recognition neural network; Step S4: input the scene image and 3D gaze point coordinates output by the eye tracker in real time into the intention recognition neural network to obtain continuous intention recognition results, describe the intention recognition results as a sequence, and use the long short-term memory network to fuse the past intention recognition results and the current intention recognition results to identify the most likely intention of the current user; Step S5: controlling the upper limb assistive robot to perform corresponding actions according to the most likely intention of the user; The specific process of step S1 is as follows: S11, coordinates of two points on the user's line of sight measured in the eye tracker scene camera coordinate system, where the line connecting the two points is the user's line of sight direction; The pixel coordinates of the user’s gaze point measured by the eye tracker are ( p u gi , p v gi ), set the lengths of the two points on the line of sight to be 0m and 1m respectively e p0, e p1, the coordinates of the two points are: e with g0 =0m, e with g1 =1m in and are the x-axis and y-axis focal lengths of the eye tracker scene camera, Represents the pixel coordinate center of the image, γ is the distortion parameter of the camera; S12, using the calibration relationship between the depth camera and the eye tracker scene camera to convert the user's sight direction in the scene camera coordinate system from the scene camera coordinate system to the depth camera coordinate system, and the intersection of the sight direction in the depth camera coordinate system and the point cloud captured by the depth camera is the user's 3D gaze point; Use the transformation matrix to transform the viewing direction from the scene camera coordinate system to the depth camera coordinate system: Among them, the upper left subscript indicates the coordinate system to which the coordinate point belongs. The two points are converted from the scene camera coordinate system to the depth camera coordinate system. The intersection of the line direction and the point cloud captured by the depth camera is the user's 3D gaze point, which is expressed as d p g ; In step S2, the acceleration, angular velocity and magnetic field strength data measured by the nine-axis inertial sensor are used to obtain the system state equation and observation equation, and the head trajectory of the user is obtained by using the Kalman filter. The specific process of step S2 is as follows: In the world coordinate system, the head posture is expressed as quaternion: x=[ E q k ]=[q0,q1,q2,q3] The upper left subscript E represents the earth coordinate system, and the time domain differential equation of the quaternion is: where w x ,w y ,w z is the three-axis angular velocity measured by the IMU, where b represents the IMU body coordinate system and k represents the kth moment of sampling; Therefore, the state equation of the system can be constructed: x(k)=Ax(k-1)+o(k) in o(k) represents the process noise of the system, Δt is the sensor sampling period, and I is the unit matrix; The gravity acceleration component in the IMU body coordinate system is: in E b R represents the rotation matrix between the world coordinate system E and the IMU body coordinate system b, |g| represents the absolute value of gravity acceleration under the z-axis of the world coordinate system; Then the Jacobian matrix Then the observation equation 1 of the system is: m acc (k)=H1x(k)+v1(k) m acc represents the acceleration measurement result, v1(k) represents the acceleration measurement noise; The components of the magnetic field strength in the IMU body coordinate system are: mag| represents the measured value of magnetic force in the world coordinate system; Then the Jacobian matrix Then the observation equation 2 of the system is: m mag (k)=H2x(k)+v2(k) m mag represents the magnetometer measurement result, v2(k) represents the magnetic measurement noise; The state equation of the system is established according to the aforementioned quaternion time-domain differential equation, and the two observation equations of the system are established according to the above two Jacobian matrices. Finally, based on the state equation of the system and the two observation equations, the head trajectory of the user is obtained using the two-stage Kalman filter; In the step S3, the scene image is subjected to target detection and saliency detection to output a saliency grayscale map, the 3D gaze point coordinates are transformed to generate a gaze map, the saliency grayscale map and the gaze map are used as inputs of the intent recognition neural network, the true intent is used as output, and the cross entropy loss function and the stochastic gradient descent method are used to train the intent recognition neural network; In step S4, the training process of the LSTM network is as follows: the current intention recognition result and the intention recognition results of multiple previous moments are used as input, and the user's true intention is used as output to train the LSTM network.

2. The human vision-driven upper limb assistive robot control method as claimed in claim 1, characterized in that: The depth camera shaft is installed on the top of the eye tracker, and the inertial sensor is integrated into the eye tracker or the depth camera.

Images