Eye movement characteristic driven cooperative mechanical arm teleoperation control system and method

Abstract

The invention discloses an eye movement characteristic driven cooperative mechanical arm teleoperation control system and method. The method comprises the following steps: calibrating an eye tracker and a global camera; constructing and training an action intention recognition model; the action intention recognition model takes eye movement features as data input and action intentions as data output, and establishes a relation model of data input and output; performing action intention online identification and prediction according to the eye movement features, and extracting action intention identification; recognizing and commanding the mechanical arm to execute the action according to the action intention. By means of the method, an operator can control the mechanical arm in a natural and visual mode, control modes do not need to be switched frequently, and the uncontrollable movement phenomenon, caused by unconscious eye jumping, of the mechanical clamping jaw can be effectively avoided. Meanwhile, according to the method, control over detailed postures of the mechanical clamping jaw during grabbing and placing is perfected, the control efficiency and convenience are improved for the disabled, especially people with upper limb dysfunction, and therefore the capacity of independently completing daily activities is enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot control and human-machine interaction, and in particular relates to a collaborative robot arm teleoperation control system and method driven by eye movement features. Background Art

[0002] People with upper limb dysfunction, such as patients with spinal cord injury, stroke, multiple sclerosis, amyotrophic lateral sclerosis, Duchenne muscular dystrophy, etc., find it difficult to independently control their upper limbs to complete daily living activities (such as grasping and placing objects). Using collaborative robotic arms to assist human hands in completing daily living activities can improve the independent living ability of these special groups of people.

[0003] At present, the control of collaborative robotic arms is mainly completed through joysticks or teaching pendants. The control process requires the operator to frequently switch between the translation mode, rotation mode and opening and closing mode of the robotic gripper in Cartesian space, which is inefficient, difficult and has a heavy cognitive load. In recent years, eye movement technology, as a non-invasive control method, has been initially applied in the fields of intelligent robots and medical assistance. It provides users with a more natural and intuitive way of interaction. The operator can control the movement of the robotic arm and the opening and closing of the robotic gripper through eye movement signals, which can greatly improve the control efficiency of the collaborative robotic arm. The core of eye movement control of the robotic arm movement is the control strategy and control algorithm.

[0004] Universities such as the University of California, Los Angeles, Imperial College London, and Southeast University in China have used eye movement signals to control robotic arms. The operator stares at the target object, and when the staring time exceeds a certain threshold, the pre-programmed robot action is triggered. However, this method is not a true eye-movement-driven robotic arm assisted operation. The movement of the robotic arm is completely dependent on pre-programming and is only effective in a structured environment. In addition, the operator does not have a strong sense of control during the operation, and is not suitable for free intention tasks.

[0005] The main reason that currently restricts eye-movement-driven robotic arm teleoperation technology is that the operator cannot autonomously control eye movements so that the line of sight is always at the front end of the gripper to steadily guide the movement of the gripper. There are a large number of involuntary saccades in eye movements (the line of sight jumps from one position to another). For example, during teleoperation, the line of sight will frequently saccade between the target object and the mechanical gripper. If the mechanical gripper is simply allowed to move following the intersection of the line of sight, the mechanical gripper will follow the involuntary saccades to make uncontrollable movements, which will have a great negative impact on the daily auxiliary tasks of the disabled.

[0006] Therefore, there is an urgent need for a robotic arm remote operation control system and method that can achieve natural, convenient and intuitive control, perfect detailed posture control, and avoid eye saccadic interference. Summary of the invention

[0007] In order to solve the above technical problems, the present invention provides a collaborative robot arm teleoperation control system and method driven by eye movement features.

[0008] The technical solution provided by the present invention is as follows:

[0009] First, it provides action intention recognition based on eye movement feature extraction and a shared control system for robotic arms, including:

[0010] A binocular visual eye tracker, including a scene camera, a pupil capture camera and a wireless module; used to capture the three-dimensional coordinates of the sight focus in the scene camera and transmit the data;

[0011] The QR code positioning board is connected to the binocular vision eye tracker;

[0012] Global cameras, including ordinary cameras and depth cameras, are used to shoot videos containing QR code positioning plates;

[0013] Collaborative robotic arms for picking up, moving and placing objects;

[0014] A tactile force feedback device for collecting tactile information from the operator;

[0015] The host computer connects the data to the global camera, collaborative robot arm and tactile force feedback device; it is used to analyze and process the data, identify the operator's action intention online and convert it into robot arm control instructions, which are sent to the collaborative robot arm to execute the action.

[0016] In a second aspect, a method for motion intention recognition by eye movement feature extraction and robot arm shared control is provided, wherein the method is implemented based on the motion intention recognition by eye movement feature extraction and robot arm shared control system described in the first aspect, and comprises the following steps:

[0017] Eye tracker and global camera calibration;

[0018] Constructing and training an action intention recognition model; the action intention recognition model uses eye movement features as data input and action intention as data output, and establishes a relationship model between data input and output;

[0019] Perform online recognition and prediction of action intention based on eye movement features, and extract action intention recognition;

[0020] Command the robot arm to perform actions based on action intention recognition.

[0021] In a possible implementation, both the scene camera and the global camera of the eye tracker are calibrated using a chessboard.

[0022] In one possible implementation, the method for constructing and training the action intention recognition model is as follows:

[0023] The operator remotely controls the collaborative robot arm through a tactile force feedback device to complete the actions of grabbing, moving and placing objects;

[0024] Obtaining a training data set, including eye movement feature data captured by a binocular vision eye tracker and real-time motion data of a collaborative robot arm collected by a host computer; the eye movement feature data includes sight focus position, sight focus moving speed, sight vector angular velocity, fixation duration, and scan amplitude and duration;

[0025] The training data set is divided into a large-range movement data set and a fine adjustment data set; the large-range movement data set is a data set of the robot arm after completing a relatively large displacement action; the fine adjustment data set is a data set of the robot arm after completing a relatively large displacement action and approaching a pre-grasping position or a pre-placement position, and then performing a small-range precise position and posture adjustment;

[0026] Taking eye movement feature data as input and real-time motion data as output, the Gaussian mixture model is used as the basic model for fitting and training, and action intention recognition models based on a large-range mobile dataset and a fine-tuned dataset are obtained respectively.

[0027] Further, the method for calculating the sight focus position is as follows:

[0028] The image of the QR code positioning plate captured by the global camera is used to obtain the two-dimensional image coordinates of the QR code vertices through image recognition and corner detection;

[0029] Calculate the position of the QR code positioning plate {B} relative to the global camera {C} C T B ;

[0030] According to the position transformation matrix between the eye tracker scene camera {S} and the QR code calibration plate {B} B T S Calculate the pose of the eye tracker scene camera relative to the global camera C T S ;

[0031] According to the transformation matrix between the global camera {C} and the robot base coordinate system {R} R T C Calculate the pose of the eye tracker scene camera relative to the global coordinate system R T S , and obtain the three-dimensional sight vector direction and sight focus coordinates in the global coordinate system.

[0032] Furthermore, in the action intention recognition model, the input includes: state variable x i =(τ i ,δ i )∈R N , where τ i =(τ x,i ,τ y,i ,τ z,i )∈R N represents the coordinate position of the end gripper of the collaborative robot arm in data i in the total data volume N during the collection process, δ i Represents the Euler angle parameters of the end gripper of the collaborative robot arm in data i in the total data volume N; time observation sequence o i =(g i ,ε i )∈R N , where g i =(g x,i ,g y,i ,g z,i ) represents the focus position of the operator's eyes in the global coordinate system in data i, ε i Represents eye movement behavior patterns of fixation and gaze shifting;

[0033] Through the joint probability distributions P1(o,x) and P2(o,x) of the time observation sequence and state variables in the large-scale mobile data set and the fine-tuning data set, the conditional probability distributions P1(x|o) and P2(x|o) based on the large-scale mobile data set and the fine-tuning data set are solved by Gaussian conditions respectively;

[0034] The Gaussian mixture model of a large-scale mobile dataset is expressed as:

[0035]

[0036] Among them, θ k is the parameter set of the Gaussian mixture model, including the weight ω of the kth Gaussian distribution k , mean vector μ k and the covariance matrix ∑ k Each portion is a multivariate Gaussian distribution, K is the number of Gaussian distributions;

[0037] Given eye movement data o i , robot end data x i The probability under the condition is:

[0038]

[0039] Since the observed variable o i is a continuous variable, the joint distribution probability can be calculated to calculate the marginal probability P1(oi |θ k ),for:

[0040] P1(o i |θ k )=∫P1(o i ,x i |θ k )dx i

[0041] During the training process, θ k Each Gaussian component in is randomly initialized, and the maximum expectation algorithm EM algorithm is used to calculate the posterior probability of each sample belonging to the Gaussian component. The posterior probability is used to update the parameters of each Gaussian component, and the iteration is repeated until convergence to obtain the observation condition o at a given real time t in the range movement mode. t ′The state variable x t ′;

[0042] Similarly, the observation condition o″ at a given real time t in the precise adjustment mode is obtained t The next state variable x″ t .

[0043] Furthermore, the method for performing online recognition and prediction of action intention based on eye movement features and extracting action intention recognition comprises the following steps:

[0044] Collect the operator's eye movement characteristics and update the time observation sequence in real time. t ;

[0045] The conditional distribution probability P1(x t |o t ) and P2(x t |o t ), enter one of the two data sets through the conditional threshold; determine the relationship between the distance between the mechanical gripper position and the intended grasping object and the distance threshold. If the distance is greater than the distance threshold, the eye movement signal is input into the conditional distribution model trained based on the large-scale mobile data set to calculate P1(x t ); if the distance is less than the distance threshold, the eye movement signal is input into the conditional distribution model trained based on the fine-tuning dataset to calculate P2(x t );

[0046] After the capture is completed, the relationship between the variance of the eye movement focus in the window and the fixation threshold is continuously calculated; if the variance is greater than the fixation threshold, the eye movement signal is input into P1(x t ); if the variance is less than the fixation threshold, then input P2(x t); According to the maximum a posteriori probability principle, the predicted value x of the operator's intention at the time t of the robot end moving speed, end posture, etc. is inferred t .

[0047] The third aspect is to provide an action intention recognition and robotic arm shared control device for eye movement feature extraction, including:

[0048] Calibration module, used for eye tracker and global camera calibration;

[0049] A model building and training module is used to build and train an action intention recognition model; the action intention recognition model uses eye movement features as data input and action intention as data output, and establishes a relationship model between data input and output;

[0050] The extraction module is used to perform online recognition and prediction of action intention based on eye movement features and extract action intention recognition;

[0051] The execution module is used to command the robot arm to perform actions according to the action intention recognition.

[0052] In a fourth aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for action intention recognition and robot arm shared control based on eye movement feature extraction as described above is implemented.

[0053] In a fifth aspect, a non-transitory computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method of action intention recognition and robot arm shared control as described above is implemented by extracting eye movement features.

[0054] The present invention has the following beneficial effects:

[0055] (1) The method provided by the present invention proposes a classification of "large-scale movement data set" and fine-adjustment data set in view of the differences in actions such as movement, placement and grasping. This not only meets the high efficiency of the robot arm's range of movement, but also can adapt to the object shape and environmental characteristics during grasping and placing. At the same time, the RGB-D depth camera can provide more accurate environmental information, providing a conditional threshold for switching between the "large-scale movement data set" and the fine-adjustment data set, reducing the complexity of use for operators.

[0056] (2) The method provided by the present invention associates eye movement features with the action intention of the robot arm by distinguishing and training the model, thereby realizing real-time recognition of action intention. The action intention recognition method based on eye movement features enables the operator to control the robot arm in a natural and intuitive way without the need to frequently switch control modes, and can also effectively avoid the uncontrollable movement of the mechanical gripper caused by involuntary eye saccades. At the same time, the method improves the detailed posture control of the mechanical gripper during grasping and placement, which improves the efficiency and convenience of control for the disabled group, especially those with upper limb dysfunction, thereby enhancing their ability to independently complete daily activities. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 1 is a schematic diagram of the system composition of the method for motion intention recognition and robot shared control based on eye movement feature extraction provided by an embodiment of the present invention; in the figure: 1-QR code positioning board {B}, 2-pupil capture camera, 3-scene camera {S}, 4-RGB sensor, 5-depth sensor, 6-global camera {C}, 7-mechanical gripper, 8-mechanical arm {R}, 9-host computer;

[0058] Figure 2 is a flow chart of a method for identifying action intention provided by an embodiment of the present invention;

[0059] Figure 3 is a flow chart of intention recognition-driven robotic arm sharing control provided by an embodiment of the present invention;

[0060] Figure 4 It is a flow chart of remote operation collection of action intention recognition model data set provided by an embodiment of the present invention;

[0061] Figure 5 is a posture diagram of a remote operation "move-grasp-place" process provided by an embodiment of the present invention;

[0062] Figure 6 It is a schematic diagram of the position of the remote operation "move-grab-place" process provided by an embodiment of the present invention;

[0063] Figure 7 is a schematic diagram of the definition of the Euler angle of the posture of the joystick of the tactile force feedback device provided by an embodiment of the present invention;

[0064] Figure 8 It is a structural schematic diagram of the motion intention recognition and robot shared control device based on eye movement feature extraction provided by the present invention;

[0065] Fig. 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0066] The present invention will be described in detail below in conjunction with the accompanying drawings. It should be noted that the described embodiments are only for the purpose of illustration and are not intended to limit the scope of the present invention.

[0067] like Figure 1 As shown, the motion intention recognition based on eye movement feature extraction and the robot shared control system include:

[0068] Binocular visual eye tracker, including a scene camera, a pupil capture camera and a wireless module; used to capture the three-dimensional coordinates of the sight focus in the scene camera and transmit the data;

[0069] The QR code positioning board is connected to the binocular vision eye tracker;

[0070] Global cameras, including ordinary cameras and depth cameras, are used to shoot videos containing QR code positioning plates;

[0071] Collaborative robotic arms for picking up, moving and placing objects;

[0072] A tactile force feedback device for collecting tactile information from the operator;

[0073] The host computer connects the data to the global camera, collaborative robot arm and tactile force feedback device; it is used to analyze and process the data, identify the operator's action intention online and convert it into robot arm control instructions, which are sent to the collaborative robot arm to execute the action.

[0074] Specifically, the binocular visual eye tracker is Tobii Glass 3 or a similar product;

[0075] Specifically, the global camera is a Kinect Azure depth camera or a similar product;

[0076] Specifically, the tactile force feedback is set to Geomagic Touch or a similar product.

[0077] In a possible implementation, the wireless module is a WIFI module, which sends the eye movement data to a host computer in real time via the TCP / IP protocol.

[0078] like Figure 2 As shown, a method for motion intention recognition and robot shared control based on eye movement feature extraction includes the following steps:

[0079] Step S100: calibrating the eye tracker and the global camera.

[0080] In a possible implementation, in step S100, the eye tracker calibration method is as follows:

[0081] Pupil capture camera calibration: Before actual operation, the operator wears the eye tracker and stares at the bull's eye on a calibration plate or card at a distance of half a meter to one meter. The eye tracker software performs relevant calculations for calibration. After calibration, the eye tracker can calculate the three-dimensional coordinates of the sight focus in the eye tracker scene camera coordinate system based on the pupil and corneal reflection positions detected by the pupil capture camera.

[0082] Scene camera calibration: Perform chessboard calibration on the eye tracker scene camera. Use the eye tracker scene camera to take more than 20 photos of the chessboard calibration board in different positions and postures, and import them into MATLAB to calculate the position transformation matrix between the eye tracker scene camera {S} and the QR code calibration board {B} B T S , so that in the actual shared control operation of the operator, the position of the line of sight focus in the global coordinate system (robotic arm base coordinate system) can be calculated through the posture of the calibration plate.

[0083] It should be noted that the global coordinate system is the coordinate system of the space where the collaborative robot arm is located.

[0084] Step S200, constructing and training an action intention recognition model; the action intention recognition model uses eye movement features as data input and action intention as data output, and establishes a relationship model between data input and output.

[0085] In a possible implementation, step S200 includes the following steps:

[0086] S210, the operator remotely controls the collaborative robot arm through a tactile force feedback device to complete the actions of grabbing, moving, and placing objects;

[0087] S220, obtaining a training data set, including eye movement feature data captured by a binocular vision eye tracker and real-time motion data of a collaborative robot arm collected by a host computer; the eye movement feature data includes a gaze focus position, a gaze focus moving speed, a gaze vector angular velocity, a fixation duration, and a scan amplitude and duration;

[0088] S230, dividing the training data set into a “large-scale movement data set” and a “fine adjustment data set”;

[0089] S240, taking the eye movement feature data as input and the real-time motion data as output, adopts the Gaussian mixture model as the basic model for fitting and training, and obtains the action intention recognition models based on the "large-range motion data set" and the "fine-tuning data set" respectively.

[0090] Further, in step S220, the method for calculating the sight focus position is as follows:

[0091] The image of the QR code positioning plate captured by the global camera is used to obtain the two-dimensional image coordinates of the QR code vertices through image recognition and corner detection;

[0092] Calculate the position of the QR code positioning plate {B} relative to the global camera {C} C T B ;

[0093] According to the position transformation matrix between the eye tracker scene camera {S} and the QR code calibration plate {B} B T S Calculate the pose of the eye tracker scene camera relative to the global camera C T S ;

[0094] According to the transformation matrix between the global camera {C} and the robot base coordinate system {R} R T C Calculate the pose of the eye tracker scene camera relative to the global coordinate system R T S , and obtain the three-dimensional sight vector direction and sight focus coordinates in the global coordinate system.

[0095] Further, in step S220, the calculation method of the sight focus moving speed, sight vector angular velocity, fixation duration, scanning amplitude and duration, etc. is as follows:

[0096] The distance between the coordinates of the line of sight focus of the two frames divided by the time difference between the two frames can be obtained to obtain the moving speed of the line of sight focus; the angle between the line of sight vectors of the two frames divided by the time difference between the two frames can be obtained to obtain the angular velocity of the line of sight vector; the fixation duration is the duration of a single fixation, the saccade amplitude is the angle swept by the line of sight vector during a single line of sight saccade, and the saccade duration is the duration of a single line of sight saccade. These eye movement characteristics are directly given by the eye tracker in real time.

[0097] Furthermore, in the action intention recognition model, the input includes: (1) state variable x i =(τ i ,δ i )∈R N , τ i =(τ x,i ,τ y,i ,τ z,i )∈R N represents the coordinate position of the end gripper of the collaborative robot arm in data i in the total data volume N during the collection process; Represents the Euler angle parameter of the end gripper of the collaborative robot in data i in the total data volume N, θ i represents the pitch angle, represents the roll angle, β iRepresents the yaw angle; Since the present invention adopts the common horizontal posture grasping method of the robot arm gripper, the pitch angle and roll angle of the end gripper are fixed, and the yaw angle is released, that is, θ i , The size is a suitable constant, β i As a variable, discard the pitch angle and roll angle, keep the yaw angle, and take δ i =β i ; (2) Time observation sequence o i =(g i ,ε i )∈R N , g t =(g x,i ,g y,i ,g z,i ) represents the focus position of the operator's eyes in the global coordinate system under data i, ε i Represents two eye movement behavior patterns: fixation and saccade. Fixation refers to an eye movement behavior pattern in which the eyes stay at a specific point, and saccade refers to an eye movement behavior pattern in which the eyes quickly move from one fixation point to another. The two are distinguished by the angular velocity of the gaze vector. A gaze vector angular velocity of less than 30° / s is a fixation, and greater than 30° / s is a saccade.

[0098] After the collection of the training data set for the action intention recognition model is completed, the training data set is divided into two parts by labeling through the playback experiment process, namely the large-scale mobile data set and the fine-tuning data set. Through the joint distribution probabilities P1(o,x) and P2(o,x) of the time observation sequence and state variables in the large-scale mobile data set and the fine-tuning data set, the conditional distribution probabilities P1(x|o) and P2(x|o) obtained by training based on the large-scale mobile data set and the fine-tuning data set are solved by Gaussian conditions respectively;

[0099] The mixed Gaussian mixture model calculated based on a large range of mobile data sets is expressed as:

[0100]

[0101] Among them, θ k is the parameter set of the Gaussian mixture model, including the weight ω of the kth Gaussian distribution k , mean vector μ k and the covariance matrix ∑ k Each portion is a multivariate Gaussian distribution, K is the number of Gaussian distributions;

[0102] Given a time observation sequence o i , the state variable x of the end of the robot i The probability under the condition is:

[0103]

[0104] Since the time observation sequence o i is a continuous variable, the joint distribution probability can be calculated to calculate the marginal probability P1(o i |θ k ),for:

[0105] P1(o i |θ k )=∫P1(o i ,x i |θ k )dx i

[0106] During the training process, θ k Each Gaussian component in is randomly initialized, and the Exceptional-Maximization (EM) algorithm is used to calculate the posterior probability of each sample belonging to the Gaussian component. The posterior probability is used to update the parameters of each Gaussian component. The iteration is repeated until convergence, and the time-view sequence o′ at a given real-time time t under the conditional distribution model trained based on a large-scale mobile data set is obtained. t The state variable x′ is t Similarly, we obtain the time observation sequence o″ at a given real time t under the conditional distribution model trained based on the precise mobile dataset t The next state variable x″ t .

[0107] Specifically, the steps of the EM algorithm are as follows:

[0108] E step (Expectation Step), for each time observation sequence o in each observation variable i and each state point x in each state variable i , calculate the posterior probability γ that belongs to each Gaussian distribution k ik :

[0109]

[0110] M step (Maximization Step), update the parameters of each Gaussian distribution, including the weight ω k , mean vector μ k and the covariance matrix ∑ k :

[0111]

[0112] Repeat the iterative E and M steps until the parameters converge.

[0113] In the above mixed Gaussian model, each Gaussian distribution The specific expression is:

[0114]

[0115] Where X is the joint vector, D refers to the dimension of the vector, and T is the transposed matrix. In the model of this embodiment, the value of D is 8.

[0116] According to the conditional distribution probability P1(x i |o i ,θ k ), the maximum a posteriori probability (MAP) criterion is used to infer the state variable x t :

[0117] x t =argmax(P1(x i |o i ,θ k ))

[0118] The observation condition o′ at a given real time t under the conditional distribution model trained based on a large-scale mobile data set can be obtained t The next state variable x′ t , and further make corresponding controls according to needs. The same is true for the conditional distribution model trained based on the precise adjustment of the data set.

[0119] Further, in step S220, the method for obtaining the training data set includes the following steps:

[0120] (1) The operator remotely controls the robotic arm through a tactile force feedback device to complete actions such as grabbing, moving, and placing objects.

[0121] (2) The host computer records the real-time motion data of the robot arm at a sampling frequency of 50 Hz, including the movement speed of the end of the robot arm (including speed direction and amplitude), the end posture (Euler angle), and the opening and closing of the gripper. In addition, during the remote operation process, the operator wears a binocular visual eye tracker throughout the process, and the host computer synchronously records the real-time eye movement data, including the focus position of the line of sight and the closing status of the eyes.

[0122] (3) The training data set is divided into two parts by replaying the experimental process and manually labeling: a large-scale movement data set and a precise adjustment data set. The reason for dividing the training set into two parts is that during the large-scale movement of the robotic arm and the adjustment process of the end of the robotic arm to complete the grasping and moving movements, there are significant differences in the human eye movement characteristics and the real-time motion data of the robotic arm. After the data set is divided, two paired samples of eye movement data and robotic arm motion data are formed. Group training is conducive to improving the matching degree of the conditional distribution model trained based on different data sets and the corresponding eye movement characteristics.

[0123] It should be noted that the tactile force feedback device includes a base, a connecting rod, a stylus that can move / rotate in six degrees of freedom, and a button on the stylus. The operator holds the stylus and moves it freely. The host computer continuously detects the position of the stylus and maps it to the end of the robot arm, controlling the translation and rotation of the end of the robot arm in three-dimensional space. At the same time, the buttons of the custom stylus pen allow the operator to control the opening and closing of the gripper at the end of the robot arm through the buttons on the stylus pen, realizing all-round remote operation of the robot arm.

[0124] For example, Figure 4 As shown in the figure, the operator holds the joystick of the tactile force feedback device, and the host computer continuously reads the XYZ coordinates of the tip of the tactile force feedback device joystick and the Euler angle ABC that describes the joystick posture at a sampling frequency of 50Hz. For the XYZ coordinates, a direct proportional mapping relationship is used, and the XYZ coordinates of the tip of the joystick are multiplied by the proportional coefficient and mapped to the end of the robotic arm. For the Euler angle ABC, an equal proportional mapping relationship is used, and the end posture of the tip of the joystick can be expressed by the pitch angle θ H , yaw angle γ H , Roll Angle To express, the three angles are defined as Figure 5 As shown. At time t0, the robot end and the tactile force feedback device are placed in an initial posture, that is, and At time t, the posture control signal of the robot end can be calculated by the following formula:

[0125] θ R (t) = θ R (t0)+(θ H (t)-θ H (t0)

[0126] γ R (t) = γ R (t0)+(γ H (t)-γ H (t0)

[0127]

[0128] Since the final grasping adopts a suitable horizontal posture, the posture control finally mapped to the host computer only uses the yaw angle γ R That's it.

[0129] The three-dimensional positions of the tactile force feedback device joystick and the end of the robotic arm at time t can be expressed as [X H (t)Y H (t)Z H (t)] and [X R (t)Y R (t)Z R (t)] indicates that the position control signal of the robot end can be calculated by the following formula:

[0130] X R (t) = k1X H (t),

[0131] Y R (t) = k2Y H (t),

[0132] Z R (t) = k3Z H (t),

[0133] Among them, k1, k2, k3 are proportional coefficients, [D XR D YR D ZR ] and [D XH D YH D ZH ] are the dimensions of the motion space at the end of the robotic arm and the end of the joystick of the tactile force feedback device in the three axes of XYZ.

[0134] After determining the teleoperation mapping relationship, the host computer continuously sends translation speed instructions and rotation speed instructions to the robot arm at a control frequency of 50Hz, and the operator completes the "move-grasp-place" process. Figure 6 As shown, there are 6 different gripping and placing positions of the robot arm gripper, for each different gripping and placing position. Figure 7 As shown, two suitable horizontal postures are selected to complete the grasping and placing actions, and the robot arm gripper is in the same initial posture when it is at the starting position.

[0135] Step S300, performing online recognition and prediction of action intention based on eye movement features, and extracting action intention recognition.

[0136] In a possible implementation, step S300 includes the following steps:

[0137] S310, collect the operator's eye movement characteristics and update the time observation sequence in real time t ;

[0138] S320, based on the large-scale mobile data set and the fine-tuning data set, respectively train the conditional distribution model to obtain the conditional distribution probability P1(x t |o t ) and P2(x t |o t ), before the grasping is completed, determine which conditional distribution model to use according to the distance threshold; judge the relationship between the distance between the position of the mechanical gripper and the intended grasped object and the distance threshold. If the distance is greater than the distance threshold, input the eye movement signal as a feature value into the conditional distribution model trained based on a large range of mobile data sets, and calculate P1(x t ); if the distance is less than the distance threshold, the eye movement signal is used as a feature value to input the conditional distribution model trained based on the fine-tuning dataset to calculate P2(x t );

[0139] S330, after the capture is completed, determine which conditional distribution model to use through the fixation threshold; continue to calculate the relationship between the variance of the eye movement fixation focus in the window and the fixation threshold. If the fixation variance is greater than the fixation threshold, input the eye movement signal as a feature value into the conditional distribution model trained based on a large-scale mobile data set, and calculate P1(x t ); If the gaze variance is less than the gaze threshold, the eye movement signal is used as a feature value to input the conditional distribution model trained based on the fine-tuning data set to calculate P2(x t ) According to the maximum a posteriori probability principle, the host computer predicts the operator's intention at time t. t (including gripper Euler angle posture, velocity direction and amplitude).

[0140] For example, see Figure 3 As shown, this embodiment selects the direction and amplitude of the translation and rotation speed of the end of the manipulator with the highest probability expected distribution according to the recognition result of the action intention, forms relevant control instructions, and sends them to the manipulator through the USB data cable and USB communication protocol, and the control frequency is 50Hz. At the same time, the host computer obtains the eye movement data through the eye tracker to determine whether the operator has closed his eyes. Since the time of blinking is usually in the range of 0.2s-0.4s, if the time of closing eyes exceeds 0.5S and the gripper is not closed, the manipulator gripper performs the "stationary 1 second-closed" operation; if the time of closing eyes exceeds 0.5S and the gripper is already closed, the manipulator gripper performs the "stationary 1 second-opened" operation.

[0141] The operator first inputs the eye movement signal into the large-scale mobile data set to move the robot gripper to the vicinity of the pre-grasping position. The global camera determines the distance between the position of the mechanical gripper and the intended grasping object. If the distance is less than the set distance threshold, the eye movement signal is input into the fine adjustment data set to make fine position and posture adjustments to allow the mechanical gripper to adapt to the shape of the object. Finally, the gripper enters the grasping position, closes the eyes, and triggers the gripper to "stand still for 1 second-close" to grasp the object. After completing the grasping action, the eye movement signal is re-input into the large-scale mobile data set to guide the robot gripper to the vicinity of the pre-placement position. The host computer continuously calculates whether the variance of the eye movement focus in the last 3s window is less than the gaze threshold. If the condition of less than the threshold is met, the eye movement signal is input into the fine adjustment data set to make fine position and posture adjustments to adapt to the surrounding environment of the placement. Finally, the gripper enters the placement position, closes the eyes, and triggers the gripper to "stand still for 1 second-open" to place the object.

[0142] Step S400, commanding the robot arm to perform an action according to the action intention recognition.

[0143] In one possible implementation, step S400 includes: obtaining the host computer's prediction value of the operator's intention at time t, and transmitting the predicted movement speed of the robot arm end (including speed direction and amplitude) and end posture (Euler angle) as execution commands to the collaborative robot arm through the host computer and USB communication protocol. The gripper robot arm executes the host computer command and performs action under the guidance of the operator's action intention.

[0144] The following is a description of the motion intention recognition and robotic arm sharing control device based on eye movement feature extraction provided by the present invention. The motion intention recognition and robotic arm sharing control device based on eye movement feature extraction described below and the motion intention recognition and robotic arm sharing control method based on eye movement feature extraction described above can be referenced to each other.

[0145] Figure 8 is a structural diagram of an action intention recognition and robotic arm shared control device for eye movement feature extraction provided by an embodiment of the present invention, such as Figure 8 As shown, it includes: a calibration module 81, a model building and training module 82, an extraction module 83 and an extraction module 83, wherein:

[0146] A calibration module 81, used for eye tracker and global camera calibration;

[0147] A model building and training module 82 is used to build and train a motion intention recognition model; the motion intention recognition model uses eye movement features as data input and motion intention as data output, and establishes a relationship model between data input and output;

[0148] Extraction module 83, used for online recognition and prediction of action intention based on eye movement features, and extraction of action intention recognition;

[0149] The extraction module 84 is used to command the robot arm to perform an action according to the action intention recognition.

[0150] Fig. 9 An example of a physical structure diagram of an electronic device is shown in FIG. Fig. 9 As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930 and a communication bus 940, wherein the processor 910, the communication interface 920 and the memory 930 communicate with each other through the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute the action intention recognition based on eye movement feature extraction and the robot sharing control method.

[0151] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0152] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the action intention recognition based on eye movement feature extraction and robot shared control methods provided by the above-mentioned methods.

[0153] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the action intention recognition and robot shared control methods based on eye movement feature extraction provided by the above-mentioned methods.

[0154] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0155] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0156] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. Motion intention recognition based on eye movement feature extraction and robot arm shared control system, characterized in that: include: Binocular visual eye tracker, including scene camera, pupil capture camera and wireless module; Used to capture the three-dimensional coordinates of the sight focus in the scene camera and transmit the data; The QR code positioning board is connected to the binocular vision eye tracker; Global cameras, including ordinary cameras and depth cameras, are used to shoot videos containing QR code positioning plates; Collaborative robotic arms for picking up, moving and placing objects; A tactile force feedback device for collecting tactile information from the operator; The host computer connects the data to the global camera, collaborative robot arm and tactile force feedback device; it is used to analyze and process the data, identify the operator's action intention online and convert it into robot arm control instructions, which are sent to the collaborative robot arm to execute the action.

2. A method for motion intention recognition based on eye movement feature extraction and shared control of a robotic arm, characterized in that: The method is implemented based on the action intention recognition and robot arm sharing control system based on the eye movement feature extraction of claim 1, and comprises the following steps: Eye tracker and global camera calibration; Constructing and training an action intention recognition model; the action intention recognition model uses eye movement features as data input and action intention as data output, and establishes a relationship model between data input and output; Perform online recognition and prediction of action intention based on eye movement features, and extract action intention recognition; Command the robot arm to perform actions based on action intention recognition.

3. The method for motion intention recognition and robot arm shared control based on eye movement feature extraction according to claim 2, characterized in that: The scene camera and the global camera of the eye tracker are calibrated by a chessboard.

4. The method for motion intention recognition and robot arm shared control based on eye movement feature extraction according to claim 2, characterized in that: The method for constructing and training the action intention recognition model is as follows: The operator remotely controls the collaborative robot arm through a tactile force feedback device to complete the actions of grabbing, moving and placing objects; Obtaining a training data set, including eye movement feature data captured by a binocular vision eye tracker and real-time motion data of a collaborative robot arm collected by a host computer; the eye movement feature data includes sight focus position, sight focus moving speed, sight vector angular velocity, fixation duration, and scan amplitude and duration; The training data set is divided into a large-range movement data set and a fine adjustment data set; the large-range movement data set is a data set of the robot arm after completing a relatively large displacement action; the fine adjustment data set is a data set of the robot arm after completing a relatively large displacement action and approaching a pre-grasping position or a pre-placement position, and then performing a small-range precise position and posture adjustment; Taking eye movement feature data as input and real-time motion data as output, the Gaussian mixture model is used as the basic model for fitting and training, and action intention recognition models based on a large-range mobile dataset and a fine-tuned dataset are obtained respectively.

5. The method for motion intention recognition and robot arm shared control based on eye movement feature extraction according to claim 4, characterized in that: The method for calculating the sight focus position is as follows: The image of the QR code positioning plate captured by the global camera is used to obtain the two-dimensional image coordinates of the QR code vertices through image recognition and corner detection; Calculate the position of the QR code positioning plate {B} relative to the global camera {C} C T B ; According to the position transformation matrix between the eye tracker scene camera {S} and the QR code calibration plate {B} B T S Calculate the pose of the eye tracker scene camera relative to the global camera C T S ; According to the transformation matrix between the global camera {C} and the robot base coordinate system {R} R T C Calculate the pose of the eye tracker scene camera relative to the global coordinate system R T S , and obtain the three-dimensional sight vector direction and sight focus coordinates in the global coordinate system.

6. The method for motion intention recognition and robot arm shared control based on eye movement feature extraction according to claim 4, characterized in that: In the action intention recognition model, the input includes: state variable x i =(τ i ,δ i )∈R N , where τ i =(τ x,i ,τ y,i ,τ z,i )∈R N represents the coordinate position of the end gripper of the collaborative robot arm in data i in the total data volume N during the collection process, δ i Represents the Euler angle parameters of the end gripper of the collaborative robot arm in data i in the total data volume N; time observation sequence o i =(g i ,ε i )∈R N , where g i =(g x,i ,g y,i ,g z,i ) represents the focus position of the operator's eyes in the global coordinate system in data i, ε i Represents fixation and gaze shifting eye movement behavior patterns; Through the joint probability distributions P1(o,x) and P2(i,x) of the time observation sequence and state variables in the large-scale mobile data set and the fine-tuning data set, the conditional probability distributions P1(x|o) and P2(x|o) based on the large-scale mobile data set and the fine-tuning data set are solved by Gaussian conditions respectively; The Gaussian mixture model of a large-scale mobile dataset is expressed as: Among them, θ k is the parameter set of the Gaussian mixture model, including the weight ω of the kth Gaussian distribution k , mean vector μ k and the covariance matrix ∑ k Each portion is a multivariate Gaussian distribution, K is the number of Gaussian distributions; Given eye movement data o i , robot end data x i The probability under the condition is: Since the observed variable o i is a continuous variable, the joint distribution probability can be calculated to calculate the marginal probability P1(o i |θ k ),for: P1(o i |θ k )=∫P1(o i ,x i |θ k )dx i During the training process, θ k Each Gaussian component in is randomly initialized, and the maximum expectation algorithm EM algorithm is used to calculate the posterior probability of each sample belonging to the Gaussian component. The posterior probability is used to update the parameters of each Gaussian component. Repeat the iteration until convergence to obtain the observation condition o′ at a given real time t in the range movement mode. t The next state variable x′ t ; Similarly, obtain the observation condition o" at a given real time t in the precise adjustment mode t The next state variable x″ t .

7. The method for motion intention recognition and robot arm shared control based on eye movement feature extraction according to claim 4, characterized in that: The method for online recognition and prediction of action intention based on eye movement features and extraction of action intention recognition comprises the following steps: Collect the operator's eye movement characteristics and update the time observation sequence in real time. t ; The conditional distribution probability P1(x t |o t ) and P2(x t |o t ), enter one of the two data sets through the conditional threshold; determine the relationship between the distance between the mechanical gripper position and the intended grasping object and the distance threshold. If the distance is greater than the distance threshold, the eye movement signal is input into the conditional distribution model trained based on the large-scale mobile data set to calculate P1(x t ); if the distance is less than the distance threshold, the eye movement signal is input into the conditional distribution model trained based on the fine-tuning dataset to calculate P2(x t ); After the capture is completed, the relationship between the variance of the eye movement focus in the window and the fixation threshold is continuously calculated; if the variance is greater than the fixation threshold, the eye movement signal is input into P1(x t ); if the variance is less than the fixation threshold, then input P2(x t ); According to the maximum a posteriori probability principle, the predicted value x of the operator's intention at the time t of the robot end moving speed, end posture, etc. is inferred t .

8. An eye movement feature extraction motion intention recognition and robotic arm shared control device, characterized in that: include: Calibration module, used for eye tracker and global camera calibration; Model building and training module, used to build and train action intention recognition model; The action intention recognition model uses eye movement features as data input and action intention as data output, and establishes a relationship model between data input and output; The extraction module is used to perform online recognition and prediction of action intention based on eye movement features and extract action intention recognition; The execution module is used to command the robot arm to perform actions according to the action intention recognition.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for action intention recognition and robotic arm shared control based on eye movement feature extraction as described in any one of claims 2 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for action intention recognition and robot arm shared control based on eye movement feature extraction as described in any one of claims 2 to 7 is implemented.

Images