Robot eye movement control method and system based on speed control

Abstract

The invention discloses a robot eye movement control method and system based on speed control, and the method comprises the steps: obtaining the eye movement data of an operator, and processing the eye movement data; the eye movement data is converted into a speed control instruction of the mechanical arm based on the speed eye movement control normal form; multi-view-angle environment information is obtained through an omni-directional camera matrix arranged at the working end of the mechanical arm, a multi-view-angle view is rendered on a user interface, and view angles are automatically switched according to the falling point of a fixation point; and the speed control instruction is received, kinematics transformation and environment state estimation are carried out, joint motion information of the mechanical arm is obtained, and the mechanical arm is controlled to execute a task. On the basis that direct position mapping is carried out on tail end movement of the robot through eye movement data, the movement speed of the robot serves as an interaction object, speed mapping of the robot is carried out through a gazing sight falling point distance control center displacement vector in the interaction process, and the performance is good in the aspects of continuous movement and dynamic adjustment of the robot.

Description

Technical Field

[0001] This invention relates to the field of robot motion control technology, and in particular to a robot eye-tracking control method and system based on speed control. Background Technology

[0002] As robotic systems expand from industrial applications to healthcare, home applications, and hazardous materials handling, developing intuitive control mechanisms is crucial. Traditional control interfaces, such as joysticks, keyboards, and dedicated input devices, impose a significant cognitive burden, require extensive training, and are unfriendly to people with movement disorders. This limitation hinders the widespread adoption of robot-assisted systems, particularly in scenarios requiring moderate levels of automation and hands-free operation.

[0003] Eye-tracking systems exhibit extremely high accuracy in spatial localization with almost no conscious effort, and visual attention naturally precedes action in the sensorimotor hierarchy. This "see before you act" predictive relationship creates opportunities for control interfaces that directly utilize attention mechanisms. Traditional manual input methods separate visual perception and manual control into different modalities, while gaze-based interfaces integrate both functions into a visual modality, requiring the management of the inherent contradiction between observational and intentional eye movements. Robotic manipulation tasks exacerbate this challenge, requiring operators to allocate visual attention between perceiving the robot's dynamic state and executing gaze-based control actions. To address these pain points, this invention proposes a speed-controlled robot eye-tracking control method. This method is similar to a virtual joystick, where the direction and distance of the gaze relative to a reference point determine the continuous movement speed. This method transforms the concept of manual speed control into eye-tracking control, calculating actuator speed through normalized radial distance, and performs well in continuous path following.

[0004] The existing invention patent CN120326628A, entitled "Adaptive Control Method for End-Factory Pose of Robot Grasping Process Based on Eye-Tracking Guidance," discloses an adaptive control method for end-factory pose of a robot based on eye-tracking guidance. Addressing the problem that existing eye-tracking control can only guide the center position of the robotic arm's gripper, making it difficult to dynamically adjust the gripping angle and placement position based on the object's shape and size, resulting in a low success rate in unstructured environments, an improved eye-tracking guidance control strategy is proposed. Under the operator's eye-tracking guidance, the system moves the gripper near the target object, acquires point cloud data using an end-factory depth camera, generates multiple feasible gripping poses based on a grasping posture detection (GPD) algorithm, and selects the pose that best matches the operator's intention by combining eye-tracking attention and a dynamic weighting mechanism. During placement, the center position of the gripper under eye-tracking guidance is dynamically compensated based on the positional relationship between the object's bottom and the gripper's grasping position, achieving precise placement. This method significantly improves the success rate of grasping and placement and the efficiency of human-robot collaboration. However, due to the susceptibility of confusion between observation intention and interaction intention during eye-tracking control, misoperation is prone to occur in continuously manipulated robot movements.

[0005] The existing invention patent CN113778070A, entitled "Control Method and Device for Robots," discloses a robot control method and device, relating to the field of computer technology. One specific embodiment of the method includes: acquiring and displaying the robot's drive control data; collecting the user's eye-tracking data and analyzing the eye-tracking data and drive control data to generate control commands; and sending the control commands to the robot to control it. This embodiment achieves robot control through eye tracking, eliminating the need for limb movements and overcoming the limitations of manual control methods such as mice, keyboards, and joysticks that rely on human hands. It also eliminates input barriers encountered when interacting with machines due to physical limitations or environmental obstacles, making robot control more flexible and convenient. However, this method uses the robot's drive control commands as the interaction object and is only applicable to the specific robot corresponding to the control command; it is not universally applicable to all target robots. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the existing technology. To achieve the above objective, a robot eye-tracking control method and system based on speed control is adopted to solve the problems mentioned in the background technology.

[0007] A robot eye-tracking control method based on velocity control includes the following steps: Step S1: Acquire the operator's eye movement data and process the eye movement data, wherein the processing includes gaze detection, gaze vector estimation and perception-control error elimination; and the speed-based eye movement control paradigm converts the eye movement data into speed control commands for the robotic arm, wherein the speed control commands are related to the displacement vector of the gaze point relative to the center of the control area, and the speed magnitude is proportional to the displacement distance. Step S2: Obtain multi-view environmental information by using an omnidirectional camera matrix deployed at the working end of the robotic arm, render multi-view views on the user interface, and automatically switch the view according to the point of gaze. Step S3: Receive the speed control command, perform kinematic transformation and environmental state estimation to obtain the joint motion information of the robotic arm, and control the robotic arm to perform the task.

[0008] As a further aspect of the present invention: the specific steps of the velocity-based eye-tracking control paradigm in step S1 include: Define a circular control area anchored at the center of each camera view. When the gaze point falls within the control area and exceeds the dead zone radius, calculate the normalized radial distance and apply a nonlinear scaling function to determine the speed of the robotic arm end effector.

[0009] As a further aspect of the present invention: the nonlinear scaling function determines the actuator speed.

[0010] in, and These represent the lower and upper limits of the end effector speed. It is the normalized radial distance of the gaze point movement, and its calculation formula is:

[0011] in, It is the distance from the point of gaze to the center of the analog joystick control area. It is the radius of the inner blind zone within the control area where no movement occurs. It is the boundary value of the control area.

[0012] As a further aspect of the present invention: when acquiring multi-view environmental information, a triple orthogonal view camera arrangement is adopted, including top view, front view and side view, to enhance spatial perception.

[0013] As a further aspect of the present invention, the specific steps for processing the eye movement data include: smoothing the raw gaze data by confidence threshold filtering and exponential moving average to reduce physiological tremors and preserve intentional saccadic movements.

[0014] As a further aspect of the present invention, it also includes detecting active control intent through blinking patterns, wherein when a preset number of blinks are detected within a continuous preset time, the robotic arm gripper control is triggered.

[0015] As a further aspect of the present invention, step S3 specifically includes the following steps: The position of the robotic arm's end effector is monitored in real time by using a directional marker code at the end effector, and the position of the robotic arm at the next moment is predicted based on the current position.

[0016] The second aspect of the technical solution: a control system employing a speed-controlled robot eye-tracking control method as described in any of the above claims, comprising: The human-computer interaction layer is configured to acquire the operator's eye movement data and process the eye movement data, wherein the processing includes gaze detection, gaze vector estimation and perception-control error elimination; the human-computer interaction layer also converts the eye movement data into speed control commands for the robotic arm based on a speed-based eye movement control paradigm, wherein the speed control commands are related to the displacement vector of the gaze point relative to the center of the control area, and the speed magnitude is proportional to the displacement distance; The multi-view visual feedback system is configured to acquire multi-view environmental information through an omnidirectional camera matrix arranged at the working end of the robotic arm, render multi-view views on the user interface, and automatically switch the view according to the point of gaze. The robot execution layer is configured to receive speed control commands from the human-machine interaction layer, perform kinematic transformations and environmental state estimations, obtain joint motion information of the robotic arm, and control the robotic arm to perform tasks.

[0017] As a further aspect of the present invention: the human-computer interaction layer and the robot execution layer adopt an observation-control separation design, wherein the human-computer interaction layer is responsible for eye-tracking data acquisition and user interface rendering, and the robot execution layer is responsible for physical interaction and task execution.

[0018] Compared with the prior art, the present invention has the following technical advantages: The above-described technical solution maps eye-tracking data to robot motion velocity data, thereby controlling the robot's direction and speed. Compared to existing position-based spatial mapping control methods, velocity-based manipulation exhibits superior path tracking and dynamic adjustment capabilities. Furthermore, the multi-view feedback interface, separating the observation perspective from the control interface, alleviates the operational confusion and high cognitive load caused by the ambiguity between eye perception and control functions during robot eye-tracking control. It also addresses the problem of the perception-control duality in eye-tracking control to a certain extent, ensuring interaction without hand manipulation. Attached Figure Description

[0019] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings: Figure 1 This is a schematic diagram of the steps of the eye-tracking control method according to an embodiment of this application; Figure 2 This is a schematic diagram of a factor graph model according to an embodiment of this application; Figure 3 This is a flowchart illustrating the index reduction process of an embodiment disclosed in this application. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Please refer to Figure 1 and Figure 2 In this embodiment of the invention, a robot eye-tracking control method based on speed control includes the following steps: Step S1: Acquire the operator's eye movement data and process the eye movement data, wherein the processing includes gaze detection, gaze vector estimation and perception-control error elimination; and the speed-based eye movement control paradigm converts the eye movement data into speed control commands for the robotic arm, wherein the speed control commands are related to the displacement vector of the gaze point relative to the center of the control area, and the speed magnitude is proportional to the displacement distance. In this embodiment, the specific steps of the velocity-based eye-tracking control paradigm in step S1 include: Define a circular control area anchored at the center of each camera view. When the gaze point falls within the control area and exceeds the dead zone radius, calculate the normalized radial distance and apply a nonlinear scaling function to determine the speed of the robotic arm end effector.

[0022] The velocity-based eye-tracking data control paradigm consists of the following steps: The velocity-based eye-tracking control paradigm performs different coordinate transformations for each viewpoint to map the 2D gaze projection onto the appropriate 3D robot motion vector. The velocity control paradigm implements various interactive metaphors, where the gaze functions as a virtual lever controlling the velocity vector of the end effector, establishing a direct spatial mapping between the gaze point displacement vector and the generated motion commands. The magnitude of the velocity command is proportional to the Euclidean distance from the robot's current position to the gaze point mapped into space, and its direction is aligned with the displacement vector, replicating the characteristics of the displacement-velocity transfer function of traditional manual input devices. The intended direction and amplitude parameters are derived by calculating the angular displacement and radial distance from the center of the manipulation area to the gaze point.

[0023] In speed mode, the system defines a circular control region anchored at the center of each camera's view. When the gaze point falls outside a defined dead zone, a normalized radial distance d ∈ [0, 1] is calculated using a control algorithm, and a nonlinear scaling function is applied to determine the actuator speed. In this embodiment, the nonlinear scaling function determines the actuator speed:

[0024] in, and These represent the lower and upper limits of the end effector speed. It is the normalized radial distance of the gaze point movement. In this embodiment, where and The calculation formula is as follows:

[0025] in, It is the distance from the point of gaze to the center of the analog joystick control area. It is the radius of the inner blind zone within the control area where no movement occurs. This is the boundary value of the control area. An index of 1.5 ensures precise adjustments at low speeds for minute gaze movements, while a larger movement distance allows the robotic arm to move at faster speeds.

[0026] In this embodiment, when acquiring multi-view environmental information, a triple orthogonal view camera arrangement is adopted, including top view, front view and side view, to enhance spatial cognition.

[0027] In this embodiment, the specific steps for processing the eye movement data include: filtering the raw gaze data using a confidence threshold and smoothing it with an exponential moving average to reduce physiological tremors and preserve intentional saccadic movements.

[0028] In this embodiment, the active control intention is detected by blinking pattern, wherein when a preset number of blinks are detected within a preset time period, the gripper control of the robotic arm is triggered.

[0029] In the specific implementation steps, the velocity-based eye-tracking control paradigm can be regarded as a derivative control method of time-position control. Its difference from the position-based eye-tracking control method is that instantaneous gaze generates proportional velocity rather than absolute position commands, in which continuous gaze in a specific direction leads to continuous movement along that vector.

[0030] Taking a robot grasping task as an example, the eye-tracking data processing process is as follows: Step 1: Use an eye tracker to capture eye movement data, and then enhance spatial cognition through a triple orthogonal perspective (top view, forward view, and side view). Step 2: Gaze information processing. The raw gaze data is filtered by confidence threshold and smoothed by exponential moving average to reduce physiological tremors while preserving intentional saccadic movements. Step 3: Grip control is achieved by detecting a pattern of 5 blinks within 1.2 seconds to distinguish between passive observation and active control intent. Step S2: Obtain multi-view environmental information by using an omnidirectional camera matrix deployed at the working end of the robotic arm, render multi-view views on the user interface, and automatically switch the view according to the point of gaze. Step S3: Receive the speed control command, perform kinematic transformation and environmental state estimation to obtain the joint motion information of the robotic arm, and control the robotic arm to perform the task.

[0031] In this embodiment, step S3 specifically includes the following steps: The position of the robotic arm's end effector is monitored in real time by using a directional marker code at the end effector, and the position of the robotic arm at the next moment is predicted based on the current position.

[0032] In the specific implementation steps, eye-tracking data is acquired through the human-machine interface layer; working environment information is acquired through an omnidirectional camera matrix; user interface rendering is performed; eye-tracking signal information is processed; eye-tracking signal information is converted into robotic arm speed control information; in the robot execution layer, the speed control commands are kinematically transformed to obtain robotic arm joint motion information; environmental state estimation is performed; and the task is executed according to the process.

[0033] like Figure 2 As shown, the diagram illustrates a factor graph model. Environmental perception and multi-view visual feedback system To facilitate comprehensive observation of the robotic arm's movement, three cameras are set up at different positions on the robotic arm's working end to observe from three different perspectives. The multi-view view is rendered in real time on the user interface, and the coordinate transformation mapping method specific to the view is automatically switched according to the movement of the gaze point.

[0034] This multi-view control strategy, based on a comprehensive observation environment, allows users to select the most suitable viewing angle for specific moving components by controlling the range of line-of-sight, enabling all-around observation and enhancing the flexibility of observation space switching and the comprehensiveness of environmental perception. Typically, the establishment of multiple views is usually based on X... Based on the top view of the Y-plane motion and the front or left view adjusted based on the z-axis, a comprehensive observation is achieved through the complementary use of three mutually perpendicular planes. Furthermore, multi-view visual feedback provides the system with interpretable and clear gaze input indicators, helping users eliminate ambiguity between perception and control and clarifying the scope of their gaze's function.

[0035] For speed control methods similar to those using vertical rods, obtaining the calculated motion vector is crucial. The point of view can be determined by scanning a fixed code; by manipulating the gaze plane and calculating the offset of the point of view on that plane, the motion vector can be obtained.

[0036] The second aspect of the technical solution: a control system employing a speed-controlled robot eye-tracking control method as described in any of the above claims, comprising: The human-computer interaction layer is configured to acquire the operator's eye movement data and process the eye movement data, wherein the processing includes gaze detection, gaze vector estimation and perception-control error elimination; the human-computer interaction layer also converts the eye movement data into speed control commands for the robotic arm based on a speed-based eye movement control paradigm, wherein the speed control commands are related to the displacement vector of the gaze point relative to the center of the control area, and the speed magnitude is proportional to the displacement distance; The multi-view visual feedback system is configured to acquire multi-view environmental information through an omnidirectional camera matrix arranged at the working end of the robotic arm, render multi-view views on the user interface, and automatically switch the view according to the point of gaze. The robot execution layer is configured to receive speed control commands from the human-machine interaction layer, perform kinematic transformations and environmental state estimations, obtain joint motion information of the robotic arm, and control the robotic arm to perform tasks.

[0037] In this embodiment, the human-computer interaction layer and the robot execution layer adopt an observation-control separation design, wherein the human-computer interaction layer is responsible for eye-tracking data acquisition and user interface rendering, and the robot execution layer is responsible for physical interaction and task execution.

[0038] The eye-tracking data teleoperation operating system architecture is as follows: like Figure 3 As shown, the diagram is a flowchart of the indicator reduction process; Human-Computer Interaction Layer: The speed-based eye-tracking control paradigm and the observation-control separation multi-view visual feedback system together constitute the human-computer interaction layer for main operation. In this layer, eye-tracking data of the operator is collected, and the human-computer interface layer coordinates eye-tracking data acquisition, user interface rendering, and control paradigm implementation. This layer performs signal processing functions, including gaze detection, gaze vector estimation, and elimination of perception-control duality errors.

[0039] Robot Execution Layer: The robot execution layer mainly includes the kinematic transformations of the robot's motion process, environmental state estimation, and task execution. It acts as a bridge between operational intent recognition and physical interaction. In addition to task execution, the robot execution layer can also feed back slave video data to the master human-machine interaction layer. It monitors the position of the robotic arm's end effector in real time using the pointing marker code at the end effector and predicts the next position based on the current position. After completing these tasks, the robot control commands are finally processed by combining the slave environmental information and the master interactive input information, realizing the physical interaction with the working environment.

[0040] The beneficial effects of this invention are: Traditional eye-tracking interaction methods based on direct position mapping cannot distinguish between the perception and control functions of eye movements. This paper proposes and implements a speed-controlled eye-tracking interaction method. This method solves the spatial distinction between perception and control by constructing a multi-view visual feedback system to partition the perception and control gaze points. Simultaneously, it constructs a speed-based interaction paradigm, using a joystick-like approach to build a speed-mapped cognitive model for controlling the direction and speed of the robotic arm's movement. In this speed-based robotic arm control method, instantaneous gaze generates proportional velocity commands rather than absolute position commands, where sustained gaze in a specific direction leads to continuous movement along that vector.

[0041] In the field of robot control, a velocity mapping-based eye-tracking control method is proposed. During the eye-tracking control process, multi-view feedback and partitioning are used to achieve comprehensive observation of the slave end and to distinguish between the two functions of eye-tracking process perception and observation and motion control. At the same time, a velocity control method similar to a vertical bar is used to calculate the offset of the landing point from the control center and convert it into a motion vector for the movement control of the robotic arm.

[0042] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention. The scope of the invention is defined by the appended claims and their equivalents, all of which should be included within the scope of protection of the invention.

Claims

1. A speed control-based robot eye movement control method, characterized by, The method comprises the following steps: Step S1, obtaining eye movement data of an operator and processing the eye movement data, wherein the processing comprises gaze detection, gaze vector estimation and perception-control error elimination; and a speed-based eye movement control paradigm is used to convert the eye movement data into speed control instructions of a mechanical arm, the speed control instructions being related to a displacement vector of a gaze point relative to a center of a control area, and the speed being directly proportional to the displacement distance; Step S2, obtaining multi-view environmental information through a matrix of omni-directional cameras arranged at a working end of the mechanical arm, rendering multi-view views on a user interface, and automatically switching views according to a landing point of the gaze point; Step S3, receiving the speed control instructions, performing kinematic transformation and environmental state estimation, obtaining joint motion information of the mechanical arm, and controlling the mechanical arm to perform a task.

2. The robot eye motion control method based on speed control according to claim 1, wherein, The specific steps of the speed-based eye movement control paradigm in step S1 comprise: A circular control area is defined and anchored at the center of each camera view, when the gaze point falls within the control area and exceeds a dead zone radius, a normalized radial distance is calculated, and a nonlinear scaling function is applied to determine the speed of the mechanical arm end effector.

3. The robot eye motion control method based on speed control according to claim 2, wherein, The nonlinear scaling function is used to determine the speed of the effector: wherein, and are lower and upper limits of the end effector velocity, is a normalized gaze point movement radial distance, which is calculated as: wherein, is the distance from the fixation point to the center of the simulated joystick control zone, is the inner blind zone radius where no movement occurs within the control zone, is the boundary value of the control zone.

4. The robot eye motion control method based on speed control according to claim 1, wherein, When obtaining multi-view environmental information, a triple orthogonal view camera arrangement is used, including top view, front view and side view, to enhance spatial cognition.

5. The robot eye motion control method based on speed control according to claim 1, wherein, The specific steps of processing the eye movement data comprise filtering the original line of sight data through a confidence threshold and smoothing the data through an exponential moving average to reduce physiological tremor and retain intentional saccadic movement.

6. The robot eye motion control method based on speed control according to claim 1, wherein, Active control intention is also detected through blink pattern detection, wherein when a preset number of blinks are detected within a continuous preset time, the mechanical arm gripper control is triggered.

7. The robot eye motion control method based on speed control according to claim 1, wherein, The specific steps in step S3 comprise: The position of the mechanical arm end is monitored in real time through a pointing marker code at the end of the mechanical arm, and the position of the mechanical arm at the next moment is predicted based on the current moment position.

8. A control system employing a velocity control based robot eye movement control method according to any one of claims 1 to 7, characterized in that, It comprises: A human-computer interaction layer configured to obtain eye movement data of an operator and process the eye movement data, wherein the processing comprises gaze detection, gaze vector estimation and perception-control error elimination; the human-computer interaction layer is also configured to convert the eye movement data into speed control instructions of a mechanical arm based on a speed-based eye movement control paradigm, the speed control instructions being related to a displacement vector of a gaze point relative to a center of a control area, and the speed being directly proportional to the displacement distance; A multi-view visual feedback system configured to obtain multi-view environmental information through a matrix of omni-directional cameras arranged at a working end of the mechanical arm, render multi-view views on a user interface, and automatically switch views according to a landing point of the gaze point; A robot execution layer configured to receive speed control instructions from the human-computer interaction layer, perform kinematic transformation and environmental state estimation, obtain joint motion information of the mechanical arm, and control the mechanical arm to perform a task.

9. The control system of claim 1, wherein, An observation-control separation design is used between the human-computer interaction layer and the robot execution layer, wherein the human-computer interaction layer is responsible for eye movement data acquisition and user interface rendering, and the robot execution layer is responsible for physical interaction and task execution.

Images