A demonstrator control method and device, demonstrator control system and storage medium

Abstract

The application discloses a teaching controller control method and device, a teaching controller control system, a storage medium and a computer program product. The method comprises the following steps: collecting an eye movement signal of an operator and a physical operation signal of a teaching controller; performing environment interference suppression processing on the eye movement signal, and outputting a line-of-sight positioning data; identifying a gaze behavior on an interactive interface based on the line-of-sight positioning data, generating a corresponding interactive intention through a preset multi-level gaze confirmation mechanism; matching a corresponding instruction confirmation rule according to an operation risk level of the interactive intention; and outputting a corresponding robot control instruction when the instruction confirmation rule is met. The scheme improves the teaching interaction efficiency of the industrial robot, enhances the environmental adaptability of the teaching controller in a complex and harsh industrial environment, and improves the industrial-level safety and reliability of the teaching operation.

Description

Technical Field

[0001] This invention belongs to the field of industrial robot teach pendant control technology, specifically relating to a teach pendant control method, device, teach pendant control system, storage medium, and computer program product. Background Technology

[0002] The industrial robot teach pendant is the core human-machine interface for operators to manually teach, program, and configure parameters of the robot. Its interaction efficiency, control accuracy, and operational reliability directly determine the production cycle, operational safety level, and adaptability to complex processes of the robot workstation. It is an indispensable key interaction carrier in the industrial robot application system.

[0003] The teach pendant primarily employs a purely physical contact interaction scheme, heavily relying on the operator's hands to input all commands via physical buttons, joysticks, control levers, and touchscreens. Under this interaction paradigm, the operator must repeatedly shift their visual focus and switch body movements between observing the robot's end effector's spatial motion and controlling the teach pendant interface, leading to frequent interruptions in the workflow. Especially in high-intensity teaching scenarios such as precision positioning adjustments and multi-axis collaborative motion, the fragmented interaction flow significantly reduces operational efficiency and prolongs work time. Furthermore, in high-risk work environments (such as close-range collaborative high-speed robot operation or handling high-temperature toxic materials) or scenarios where the operator's hands are occupied by tools or workpieces, physical contact control not only increases the risk of accidental touches on critical controls and subsequent safety accidents but also fails to meet the needs of operators with limited limb movement, creating an inherent and irreconcilable contradiction between interaction efficiency and operational safety.

[0004] To overcome the limitations of physical contact interaction, the industry has successively tried to introduce non-contact or new interaction methods such as voice control, visual gesture recognition, and enhanced touch control. However, the above solutions all suffer from serious environmental incompatibilities when applied in industrial settings: continuous mechanical noise, aerodynamic noise, and random impact noise significantly reduce the accuracy of voice recognition, while also posing problems such as command privacy leakage and mutual interference on-site; vision-based gesture recognition is easily affected by sudden changes in ambient light, welding arc light, operator posture obstruction, and differences in workwear, making it difficult to meet the rigid requirements of industrial control in terms of long-term operational stability and recognition accuracy; high-sensitivity touchscreens experience a significant decrease in touch accuracy and response reliability under conditions of oil stains, coolant splashes, severe equipment vibration, or operators wearing heavy protective gloves. In summary, existing non-contact interaction solutions cannot provide interaction guarantees that combine high reliability, high accuracy, and high environmental robustness in safety-critical industrial control scenarios, making it difficult to balance interaction convenience with industrial scenario adaptability.

[0005] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The purpose of this invention is to provide a teach pendant control method, device, teach pendant control system, storage medium, and computer program product to solve the problem in related solutions that industrial robot teaching interaction is difficult to simultaneously achieve interaction efficiency, environmental adaptability, and industrial-grade safety and reliability in complex and harsh industrial environments. This invention effectively improves the teaching interaction efficiency of industrial robots, enhances the environmental adaptability of the teach pendant in complex and harsh industrial environments, and improves the industrial-grade safety and reliability of teaching operations.

[0007] This invention provides a teach pendant control method, comprising: acquiring the operator's eye movement signals and the physical operation signals of the teach pendant; performing environmental interference suppression processing on the eye movement signals and outputting gaze positioning data; identifying gaze behavior on the interactive interface based on the gaze positioning data, and generating corresponding interactive intentions through a preset multi-level gaze confirmation mechanism; matching corresponding instruction confirmation rules according to the operational risk level of the interactive intentions; and outputting corresponding robot control instructions when the instruction confirmation rules are met.

[0008] In some implementations, the eye movement signal is subjected to environmental interference suppression processing, including: acquiring the posture motion data of the teach pendant; and dynamically compensating the gaze positioning data based on the posture motion data to filter out positioning deviations introduced by vibration and head shaking.

[0009] In some implementations, a preset multi-level gaze confirmation mechanism is used to generate the corresponding interaction intent, including: using the continuous dwell time of the gaze point on the corresponding element of the interactive interface as the basis for recognizing the gaze behavior, and obtaining the cumulative dwell time of the gaze point on the element; when the cumulative dwell time reaches a first preset dwell time threshold, marking the element as a candidate target and outputting a visual cue; when the cumulative dwell time reaches a second preset dwell time threshold, generating an interaction intent corresponding to the element; the second preset dwell time threshold is greater than the first preset dwell time threshold.

[0010] In some implementations, a corresponding instruction confirmation rule is matched based on the operational risk level of the interaction intention, including: if the operational risk level of the interaction intention is low risk, then it is determined that the instruction confirmation rule is met; if the operational risk level of the interaction intention is high risk, then the interaction intention also needs to be confirmed through physical operation, and after the confirmation is successful, it is determined that the instruction confirmation rule is met.

[0011] In some implementations, the method further includes: real-time monitoring of the effectiveness and operating status of eye movement signals; when preset degradation conditions are met, closing the eye movement control path and retaining only the physical operation control path; the preset degradation conditions include any one of the following: continuous failure of eye movement signals, continuous deviation of the line of sight from the effective area of ​​the interactive interface, or receipt of a physical emergency stop input signal.

[0012] In some implementations, the method further includes: setting a direction control activation area in the edge region of the interactive interface, mapping the movement direction according to the position of the gaze in the activation area, mapping the movement speed according to the duration of the gaze in the activation area, and generating continuous motion control commands for the robot; and / or, extracting the operator's physiological state characteristics from the eye movement signals, and when the operator is detected to be in a fatigue state, adaptively extending the duration threshold of multi-level gaze confirmation and reducing the upper limit of the robot's movement speed. In conjunction with the above method, another aspect of the present invention provides a teach pendant control device, comprising: a data acquisition unit configured to acquire eye movement signals of an operator and physical operation signals of the teach pendant; a processing unit configured to perform environmental interference suppression processing on the eye movement signals and output gaze positioning data; the processing unit is further configured to identify gaze behavior on the interactive interface based on the gaze positioning data and generate a corresponding interactive intent through a preset multi-level gaze confirmation mechanism; a control unit configured to match a corresponding instruction confirmation rule according to the operational risk level of the interactive intent; the control unit is further configured to output a corresponding robot control instruction when the instruction confirmation rule is satisfied.

[0013] In some implementations, the processing unit performs environmental interference suppression processing on the eye movement signal, including: acquiring the posture motion data of the teach pendant; and dynamically compensating the gaze positioning data based on the posture motion data to filter out positioning deviations introduced by vibration and head shaking.

[0014] In some implementations, the processing unit generates a corresponding interaction intent through a preset multi-level gaze confirmation mechanism, including: using the continuous dwell time of the gaze point on the corresponding element of the interactive interface as the identification basis for the gaze behavior, and obtaining the cumulative dwell time of the gaze point on the element; when the cumulative dwell time reaches a first preset dwell time threshold, marking the element as a candidate target and outputting a visual cue; when the cumulative dwell time reaches a second preset dwell time threshold, generating an interaction intent corresponding to the element; the second preset dwell time threshold is greater than the first preset dwell time threshold.

[0015] In some implementations, the control unit matches a corresponding instruction confirmation rule based on the operational risk level of the interaction intention, including: if the operational risk level of the interaction intention is low risk, then it is determined that the instruction confirmation rule is met; if the operational risk level of the interaction intention is high risk, then the interaction intention also needs to be confirmed through physical operation, and after the confirmation is successful, it is determined that the instruction confirmation rule is met.

[0016] In some implementations, the control unit is further configured to: monitor the validity and operating status of eye movement signals in real time, and shut down the eye movement control path and retain only the physical operation control path when preset degradation conditions are met; the preset degradation conditions include any one of the following: continuous failure of eye movement signals, continuous deviation of the line of sight from the effective area of ​​the interactive interface, or receipt of a physical emergency stop input signal.

[0017] In some implementations, the control unit is further configured to: set a direction control activation area in the edge region of the interactive interface, map the movement direction according to the position of the gaze in the activation area, map the movement speed according to the duration of the gaze in the activation area, and generate continuous motion control commands for the robot; and / or, extract the operator's physiological state characteristics from the eye movement signals, and when the operator is detected to be in a fatigue state, adaptively extend the duration threshold of multi-level gaze confirmation and reduce the upper limit of the robot's movement speed. In conjunction with the above-described device, the present invention further provides a teach pendant control system, comprising: the teach pendant control device described above.

[0018] In conjunction with the above method, the present invention further provides a storage medium comprising a stored program, wherein, when the program is executed, the device on which the storage medium is located executes the teach pendant control method described above.

[0019] In conjunction with the above method, the present invention further provides a computer program product comprising a computer program that, when processed and executed, implements the steps of the above-described teach pendant control method.

[0020] The present invention collects the operator's eye movement signals and the physical operation signals of the teach pendant, performs environmental interference suppression processing on the eye movement signals to output gaze positioning data, identifies the gaze behavior of the interactive interface based on the gaze positioning data, generates corresponding interaction intentions through a multi-level gaze confirmation mechanism, and matches instruction confirmation rules according to the operation risk level of the interaction intention. Once the rules are met, the corresponding robot control instructions are output. This effectively improves the teaching interaction efficiency of industrial robots, enhances the environmental adaptability of the teach pendant in complex and harsh industrial environments, and improves the industrial-grade safety and reliability of teaching operations.

[0021] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention.

[0022] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating an embodiment of the teach pendant control method of the present invention; Figure 2 This is a schematic diagram of a structure of an embodiment of the teach pendant control device of the present invention; Figure 3 This is a schematic diagram of the system architecture and data flow; Figure 4 A flowchart illustrating another embodiment of the teach pendant control method; Figure 5 This is a diagram illustrating secure arbitration and state transition.

[0024] Referring to the accompanying drawings, the reference numerals in the embodiments of the present invention are as follows: 101-Acquisition unit; 102-Processing unit; 103-Control unit. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0026] In recent years, eye-tracking technology has made significant progress in consumer electronics, medical assistance, and neuroscience, demonstrating its application potential in directly expressing operational intentions through gaze and reducing the burden on manual operation, providing a new technological path for non-contact interaction in teaching pendants. However, when directly applying eye-tracking technology to industrial-grade teaching control, several unresolved technical obstacles remain, making it unsuitable for the safety and reliability requirements of industrial scenarios: First, rapid changes in light and dark, welding arc light, structural vibrations, and strong electromagnetic interference in industrial environments can easily cause instantaneous loss or drastic changes in eye movement signals. Traditional tracking algorithms based on pupil-corneal reflexes are unstable under these conditions, easily leading to gaze coordinate drift and positioning failure. Second, human eye behaviors such as saccades, gaze, and blinking are continuous, non-actively triggered physiological signals, and existing technologies struggle to unambiguously convert them into deterministic and reliable signals. The industrial safety-grade discrete control commands are prone to false triggering due to unconscious eye movements; thirdly, non-contact eye movement control lacks a dynamic safety coordination mechanism with existing physical control systems, and the switching between modes is often abrupt, unable to dynamically arbitrate control authority and downgrade safety based on task risk level, operator status, and signal quality, posing safety risks during control handover; fourthly, traditional gaze duration confirmation mechanisms cannot distinguish between conscious staring and unconscious daze, and prolonged unconscious staring when the operator is fatigued or distracted can cause false positives, and current technology has not yet established active inhibition measures for unconscious operation at the physiological level.

[0027] Therefore, this invention provides a teach pendant control method that integrates a multi-source sensing hardware architecture combining near-infrared eye tracking, IMU posture compensation, and traditional physical controls. Combined with a graded gaze confirmation mechanism, a context-aware dynamic safety arbitration strategy, and a method for suppressing unconscious operations through micro-saccade physiological recognition, this method solves multiple technical challenges, including signal drift and distortion in harsh industrial environments, ambiguous mapping between physiological signals and control commands, lack of a safety coordination and degradation mechanism between eye and hand control, and the inability of traditional duration thresholds to distinguish between intentional and unconscious gazes, which can easily lead to false triggering. This achieves a unified effect of non-contact, efficient interaction and industrial-grade, highly reliable safety control, comprehensively improving the environmental adaptability, operational efficiency, and operational safety level of the teach pendant.

[0028] The system architecture of this invention is as follows: Figure 3As shown, the system is divided into four core layers based on the bottom-up, layer-by-layer abstraction of data flow: data acquisition layer, processing and fusion layer, arbitration and execution layer, and feedback and interface layer, each with its own independent environmental perception unit. The data acquisition layer consists of an eye-tracking module, an inertial measurement unit (IMU), and traditional physical controls, which collect three types of signals: operator's eye movements, teach pendant posture, and manual operation. The environmental perception unit simultaneously collects on-site lighting, vibration, interference, and other operating condition data. All multi-source signals are sent to the multi-source information fusion and preprocessing center of the processing and fusion layer for timestamp synchronization and signal preprocessing. The operator's interaction intent is then parsed by the eye-tracking-manual hybrid enhanced interaction model. The intent signal is transmitted to the security arbitration and context awareness engine of the arbitration and execution layer, where it is divided into three execution paths based on instruction attributes: eye-tracking instructions, physical button instructions, and hybrid enhanced instructions. Finally, all instructions are fed into the teach pendant main control unit of the feedback and interface layer. One path outputs to the robot control layer to drive the robot to perform the corresponding action, while the other path provides feedback on the interaction status to the operator through the teach pendant display screen, forming a complete human-machine closed-loop interactive data flow.

[0029] According to embodiments of the present invention, a teach pendant control method is provided, such as... Figure 1 The flowchart of an embodiment of the method of the present invention is shown. The teach pendant control method may include steps S110 to S150.

[0030] In step S110, the operator's eye movement signals and the physical operation signals of the teaching pendant are collected.

[0031] Eye movement signals refer to continuous data collected by the teach pendant that reflects the operator's eye movement state, including information such as gaze direction, fixation position, eye movement trajectory, and blinking status. Physical operation signals refer to operation command signals input by the operator through physical control components such as buttons, joysticks, and knobs on the teach pendant.

[0032] In step S120, the eye movement signal is subjected to environmental interference suppression processing, and the gaze positioning data is output.

[0033] Environmental interference suppression processing addresses the drift, distortion, and frame loss issues caused by factors such as vibration, strong light, and electromagnetic interference in industrial environments. It involves correcting and filtering the raw eye-tracking data. The gaze localization data, after interference correction, is the stable gaze point coordinate data mapped to the coordinate system of the teach pendant's interactive interface.

[0034] Industrial environments are often characterized by harsh conditions such as equipment vibration, welding arc light, and strong electromagnetic interference. Raw eye movement signals can suffer from coordinate drift, trajectory jumps, and brief loss of data, leading to positioning errors and false triggers when used directly for interaction. This is a core obstacle hindering the successful implementation of eye-tracking interaction in industrial settings. Therefore, interference suppression processing of the raw signals is essential to output usable gaze positioning results.

[0035] Specifically, the system first filters out abnormal jump values ​​in the original eye movement signal, and then combines data such as the teacher pendant's posture and the intensity of environmental interference to correct the deviation of the gaze coordinates, and finally outputs stable and accurate interface gaze point coordinates.

[0036] In some implementations, step S120 involves performing environmental interference suppression processing on the eye movement signal, including: acquiring the posture motion data of the teach pendant; and dynamically compensating the gaze positioning data based on the posture motion data to filter out positioning deviations introduced by vibration and head shaking.

[0037] Attitude motion data reflects the spatial attitude and motion state of the teach pendant, including information such as yaw, pitch, roll angles, angular velocity, and acceleration. Dynamic compensation synchronously corrects the line-of-sight positioning coordinates based on real-time attitude changes to offset positioning offsets caused by external motion interference.

[0038] Specifically, a motion estimation model of the teach pendant relative to the operator's head is established. Based on real-time posture changes, the gaze positioning data is dynamically corrected in reverse, filtering out positional deviations introduced by vibration and swaying from the original gaze coordinates. The final output is a stable and accurate coordinate of the interface gaze point. The entire compensation process is performed synchronously with eye-tracking signal acquisition to ensure the corrected gaze data is real-time and does not affect the interaction response speed.

[0039] The continuous mechanical vibrations generated by production equipment in industrial settings, coupled with natural hand movements and head posture changes during operator use of the teach pendant, all contribute to shifts in the relative position between the teach pendant and the operator's eyes. Conventional eye-tracking coordinate calculations assume the teach pendant is stationary. When the teach pendant vibrates or shakes, the calculated gaze coordinates drift and jump irregularly, failing to accurately align with interactive elements on the interface, and in severe cases, even preventing normal eye-tracking interaction. Therefore, by fusing and correcting posture and eye-tracking data, interference from equipment vibrations and operator hand movements in industrial settings can be effectively eliminated, significantly improving the accuracy and stability of gaze positioning. This ensures reliable and usable eye-tracking interaction even in industrial conditions with mechanical vibrations, enhancing the teach pendant's environmental adaptability at the signal level and reducing operational errors and false triggers caused by positioning drift.

[0040] In some embodiments, an intelligent filtering algorithm can be superimposed on attitude compensation to further suppress signal noise caused by multi-source mixed interference; or the parameters of the compensation model can be adaptively adjusted according to the frequency characteristics of the field vibration to adapt to different types of vibration conditions.

[0041] In step S130, based on the gaze positioning data, the gaze behavior on the interactive interface is identified, and the corresponding interactive intent is generated through a preset multi-level gaze confirmation mechanism.

[0042] Fixation behavior is the act of an operator's gaze lingering on a fixed area of ​​the interactive interface. A multi-level fixation confirmation mechanism uses progressively layered judgment conditions to distinguish between unconscious eye movements and conscious operational intentions. Interaction intention refers to the operational goal and direction that the operator intends to perform, as interpreted by the system; it is an intermediate state in the transformation of physiological eye movement signals into control commands.

[0043] Most eye saccades and brief pauses are unconscious physiological behaviors. Directly interpreting every gaze as an operational command would result in numerous false triggers, failing to meet the reliability requirements of industrial control. Therefore, a multi-level, progressive confirmation mechanism is needed to gradually filter out the operator's active and stable gaze behaviors before converting them into explicit operational intentions.

[0044] Specifically, when an operator browses the list of teaching programs, their gaze will quickly sweep across multiple program items. These brief glances will not be judged as valid intentions. When the operator stares at the target program item and maintains a stable gaze, the system will generate an interactive intention to select the program item after multiple levels of judgment.

[0045] In some implementations, step S130 involves generating a corresponding interaction intent through a preset multi-level gaze confirmation mechanism. This includes: using the continuous dwell time of the gaze point on a corresponding element of the interactive interface as the basis for recognizing the gaze behavior, and obtaining the cumulative dwell time of the gaze point on the element; when the cumulative dwell time reaches a first preset duration threshold, marking the element as a candidate target and outputting a visual cue; the candidate target is an interface interactive element that the system initially identifies as the operator's gaze object, representing only that the system has captured a potential operation intent, but not triggering an actual operation. The visual cue is a visual effect that the interface provides feedback on the recognition status to the operator, used to clearly inform the operator that the current target has been captured by the system. When the cumulative dwell time reaches a second preset duration threshold, an interaction intent corresponding to the element is generated; the second preset duration threshold is greater than the first preset duration threshold.

[0046] Specifically, the system tracks the position of the gaze point in real time. When the gaze point falls within the valid area of ​​a certain interface element, it begins to accumulate the dwell time corresponding to that element. When the accumulated dwell time reaches a first preset threshold (e.g., 500ms), the system marks the element as a candidate target and outputs a corresponding visual cue on the interface, clearly conveying to the operator that "the target has been recognized by the system." At this stage, only preliminary intent recognition is completed, and no operational intent is generated. If the operator keeps their gaze stable and the dwell time continues to accumulate and reaches a second preset threshold (e.g., 800ms), the system determines that the gaze is an active operation by the operator and generates an interaction intent corresponding to that element. If the gaze point moves out of the valid area of ​​the element before both thresholds are reached, the system will immediately reset the accumulated dwell time of the element to zero, remove the candidate target mark and visual cue, and not generate any interaction intent.

[0047] The human eye frequently pauses briefly while browsing an interface or during natural eye movements. These pauses are unconscious physiological behaviors. If the intention to operate is determined solely by a single duration threshold, either the threshold is too short, leading to numerous false triggers, or the threshold is too long, slowing down the interaction response speed, failing to balance interaction efficiency and operational reliability. Furthermore, without feedback from intermediate states, the user cannot know whether the system has recognized their gaze, resulting in a sense of loss of control over the interaction. Therefore, a two-tiered, progressive duration determination mechanism effectively filters out false triggers caused by unconscious squinting and brief pauses, while providing clear feedback on the interaction status through primary visual cues. Without significantly increasing operation time, this greatly improves the accuracy of intent recognition and operational controllability in eye-tracking interaction, making the logic of eye-tracking interaction more aligned with human cognitive habits.

[0048] In step S140, the corresponding instruction confirmation rule is matched according to the operation risk level of the interaction intent.

[0049] Operational risk level is a safety classification of taught operations based on the severity of the safety consequences of accidental operation. Command confirmation rules are the validity criteria for operations at different risk levels, used to verify whether the operational intent can be converted into a formal control command.

[0050] The safety consequences of different teaching operations vary significantly: errors in menu browsing and parameter viewing do not pose a safety risk, while accidental triggering of operations such as robot startup, permanent parameter writing, and emergency stop recovery could cause equipment damage or even personal injury. Applying strict confirmation rules to all operations would drastically reduce efficiency; conversely, using lenient confirmation rules would compromise the safety of high-risk operations. Therefore, a risk-based matching rule model is adopted to achieve a dynamic balance between efficiency and safety.

[0051] Specifically, the system pre-classifies all teaching operations into risk levels, generates interactive intents, first matches the corresponding risk level, and then calls the instruction validity determination rules corresponding to that level.

[0052] In some implementations, step S140 involves matching the corresponding instruction confirmation rule based on the operational risk level of the interaction intention, including: if the operational risk level of the interaction intention is low risk, then it is determined that the instruction confirmation rule is met; if the operational risk level of the interaction intention is high risk, then the interaction intention also needs to be confirmed through physical operation, and after confirmation, it is determined that the instruction confirmation rule is met.

[0053] Industrial robot teaching scenarios involve numerous operations with varying safety levels. If all operations employ strict multi-confirmation rules, it significantly increases the number of steps, reduces interaction efficiency, and negates the efficiency gains from non-contact eye-tracking interaction. Conversely, if all operations use lenient confirmation rules, the safety of high-risk operations cannot be guaranteed, and accidental triggering could lead to serious production accidents. Therefore, it is necessary to match different levels of instruction confirmation rules based on the risk level of the operation corresponding to the interaction intent, achieving a dynamic balance between efficiency and safety.

[0054] Specifically, all teaching operations are pre-classified with corresponding risk levels. After generating interaction intentions, their respective risk levels are matched. If the interaction intention corresponds to a low-risk operation, the instruction confirmation rule is satisfied after completing multi-level gaze confirmation, and the operation directly proceeds to the instruction execution stage. This preserves the advantages of convenient and efficient eye-tracking interaction while reducing unnecessary steps. If the interaction intention corresponds to a high-risk operation, gaze confirmation alone is insufficient to determine if the rule is satisfied; the operator must also perform the corresponding physical operation confirmation. Only when both gaze confirmation and physical confirmation are completed simultaneously, and the timing of the two confirmation actions conforms to the valid judgment window, is the instruction confirmation rule satisfied. Through this secondary verification of physical operations, false intentions caused by unconscious eye movements and signal interference can be eliminated at the interaction channel level, fundamentally preventing the accidental triggering of high-risk operations.

[0055] For example, operations such as browsing the program directory, switching parameter display pages, and viewing point coordinates are considered low-risk operations. After the operator completes multi-level confirmation through gaze, the system directly determines that the rules are met and executes the corresponding interface operation without any additional manual action, resulting in smooth and efficient operation. When the operator selects operations such as program start, parameter permanent saving, or emergency stop reset, the system identifies them as high-risk operations. Simply completing gaze confirmation will not trigger execution; a confirmation prompt will pop up on the interface, waiting for the operator to press the dedicated physical confirmation button on the teach pendant. Only after the operator presses the confirmation button will the system determine that the confirmation rules are met and execute the corresponding high-risk operation.

[0056] In step S150, when the instruction confirmation rule is met, the corresponding robot control instruction is output.

[0057] Only interaction intentions that fully comply with the verification rules are considered genuine and proactive operation commands from the operator. Outputting commands without proper verification will result in incorrectly judged, invalid intentions being applied to the robot, creating safety hazards. This step is the final verification stage in converting interaction intentions into device actions, ensuring the validity and security of all issued commands.

[0058] Specifically, the system verifies whether the current interaction intent fully meets all the confirmation conditions of the corresponding level. After the verification is passed, the interaction intent is converted into standard robot control instructions and sent to the robot control terminal for execution.

[0059] This invention, through a dual-path fusion design of eye-tracking signals and physical operation signals, combined with a complete process of environmental interference suppression, multi-level gaze confirmation, and risk classification verification, not only breaks through the limitations of traditional purely manual teaching pendants that are hand-eye bound, have low operating efficiency, and cannot be operated in scenarios where both hands are restricted, but also solves the problems of weak anti-interference ability, high risk of false triggering, and insufficient safety and reliability of single non-contact interaction in industrial environments. It can simultaneously take into account interaction efficiency, environmental adaptability, and industrial-grade operational safety in complex industrial sites.

[0060] In some implementations, the method further includes: real-time monitoring of the effectiveness and operating status of eye movement signals, and when preset degradation conditions are met, closing the eye movement control path and retaining only the physical operation control path; the preset degradation conditions include any one of the following: continuous failure of eye movement signals, continuous deviation of the line of sight from the effective area of ​​the interactive interface, or receipt of a physical emergency stop input signal.

[0061] The eye-tracking control path is the transmission and execution channel that translates the interactive intent interpreted from eye movements into control commands and issues them. The physical operation control path is the channel through which commands are input and executed via physical manipulation components; it is the system's fundamental safety and redundancy control path. Eye movement signal validity is a state indicator characterizing whether the quality of the eye movement signal meets the requirements of interactive control. A stable signal and accurate positioning are considered valid; continuous frame drops, severe drift, or tracking loss are considered invalid. Operating status encompasses system operating condition information including the current environmental interference intensity, operator gaze state, and operational input status.

[0062] Specifically, the system synchronously tracks the tracking status of eye-tracking signals, the position range of the gaze, and the input status of physical emergency stop. When any one of the three preset degradation conditions is met, the system automatically performs a safety degradation operation: shutting down the eye-tracking control path, blocking all control commands generated by eye-tracking signals, while fully retaining all functions of the physical operation control path, ensuring that the operator can still complete basic operations and safety management through physical components such as buttons, joysticks, and emergency stop buttons.

[0063] The normal operation of non-contact eye-tracking interaction relies on stable eye-tracking signals and continuous eye focus from the operator. However, industrial environments are complex and may experience abnormal scenarios such as strong light interference causing eye-tracking loss, operator shifting of gaze due to equipment, and sudden emergencies. If the eye-tracking control path remains active under abnormal conditions, signal distortion and unintentional gaze deviation may lead to misoperation, or even create safety hazards such as blurred control. Furthermore, if the system completely loses control after an anomaly occurs, the operator will be unable to perform safety operations such as shutdown and reset, leading to more serious consequences. Therefore, through real-time monitoring and automatic switching, a safety redundancy protection system for abnormal eye-tracking interaction scenarios has been constructed. This system can automatically and safely transfer control in the event of eye-tracking signal failure, operator gaze deviation, or emergency situations. This avoids the safety risks caused by accidental eye-tracking triggering under abnormal conditions, while maintaining complete physical operation paths to ensure basic operation and emergency control capabilities. This improves the operational reliability and safety fault tolerance of the hybrid interactive teaching pendant at the system level.

[0064] In some implementations, the method further includes: setting a direction control activation area in the edge region of the interactive interface, mapping the motion direction according to the position where the gaze stays in the activation area, mapping the motion speed according to the duration of the gaze in the activation area, and generating continuous motion control commands for the robot.

[0065] The direction control activation area is a pre-set triggerable region at the edge of the interactive interface, corresponding to a specific movement direction of the robot. Entering this area will trigger motion control in the corresponding direction. Continuous motion control commands are instructions that control the robot's end effector to move continuously along a specified direction, unlike discrete commands that involve single-point jumps.

[0066] For the design of continuous motion control for robots, traditional debugging methods rely on the operator continuously moving physical joysticks to achieve continuous movement. This requires both hands to maintain an operating posture, and the operator must repeatedly switch their attention between observing the robot's end effector and operating the joysticks, resulting in low operational efficiency and hand fatigue. If the movement direction is controlled directly by the viewer's gaze across the entire screen, it is easy to accidentally trigger movement commands while naturally scanning the interface content, causing safety hazards. Therefore, setting a direction control activation area at the edge of the interactive interface not only conforms to the operator's intuitive operation of adjusting direction through edge controls, but also avoids accidental triggering of motion control while browsing the content in the center of the interface, thus balancing operational convenience and operational safety.

[0067] Specifically, the system predefines activation areas corresponding to different movement directions around the perimeter of the interactive interface. When the user's gaze enters and remains in an activation area in a certain direction, the system maps the direction corresponding to that area to the robot's movement direction along the corresponding coordinate axis. Simultaneously, the system maps the movement speed based on the duration of the gaze within the activation area; the longer the gaze remains, the higher the movement speed, until a preset safe speed limit is reached. When the operator's gaze moves out of the activation area or returns to the center of the screen, the robot's movement immediately decelerates and stops, with motion control responding in real-time to the gaze state. The coordinate system used for motion mapping can automatically switch according to the current operation settings, and the interface synchronously displays the currently active coordinate system type to avoid operational confusion.

[0068] In some implementations, the method further includes: extracting the operator's physiological state features from eye movement signals, adaptively extending the duration threshold of multi-level gaze confirmation when operator fatigue is detected, and reducing the upper limit of robot movement speed.

[0069] Fatigue is a physiological state in which operators experience decreased attention, slowed reaction time, and reduced operational accuracy due to prolonged work. Industrial teaching operations are typically long-duration, making operators prone to fatigue and distraction. In this state, operators' reaction speed decreases, unconscious fixation increases, and fixed interaction parameters lead to a higher probability of accidental triggering. A mismatch between the operational response rhythm and the operator's state poses a safety hazard. Therefore, real-time monitoring of the operator's physiological state based on eye-tracking signals and dynamic adjustment of interaction parameters can adapt to the operational safety needs of different mental states without increasing the operator's workload.

[0070] Specifically, the system continuously extracts multiple physiological characteristics of the operator from eye movement signals and assesses the operator's fatigue level online. When the system detects that the operator's physiological characteristics meet the preset fatigue state judgment criteria, it automatically performs parameter adjustments: proportionally extending the duration threshold of multi-level gaze confirmation to reduce the probability of unintentional accidental triggering; and simultaneously reducing the upper limit of the robot's continuous movement speed to slow down the robot's movement rhythm and allow the operator more time for emergency response. The entire adjustment process is completed automatically without manual intervention from the operator, and a status prompt is simultaneously displayed on the interface to inform the operator that they have entered fatigue adaptation mode.

[0071] For example, the system continuously acquires the operator's eye movement trajectory at a sampling frequency of no less than 120Hz, performs high-speed filtering on the original gaze coordinate sequence, and extracts micro-saccade events. Micro-saccades are defined as spontaneous saccades with an amplitude less than 1 degree of visual angle, a duration of 10-35ms, and a peak velocity of 20-100° / s. The system calculates the amplitude, peak velocity, and frequency of each micro-saccade in real time. The operator is considered to be in an unconscious gaze state when the following conditions are simultaneously met: micro-saccade amplitude consistently below 0.5 degrees, micro-saccade frequency significantly increased (>3 times / second) or significantly decreased (<0.5 times / second), and the current gaze point is held for more than 300ms without normal saccades. When the system determines that the current gaze is unconscious, it performs the following operations: timer freeze, visual cue, and tactile reminder. Timer freeze pauses the gaze confirmation timer's accumulation, preventing unconscious gaze from reaching the confirmation threshold. Visual cue changes the highlight effect of the candidate target to a pulsed flash (distinct from the smooth halo of normal confirmation), prompting the operator to regain focus. Tactile cues are activated by a short vibration (e.g., lasting 50ms) emitted by the teach pendant's built-in vibration motor, waking the operator without disturbing others. When the system detects that the operator has regained conscious gaze (microsaccades return to normal, and active saccades or blinking occur), the system automatically releases the inhibition and resumes normal gaze timing. If the unconscious state persists for more than 5 seconds and the tactile cues are ineffective, the system proactively triggers a state machine downgrade, switching eye control permissions to manual mode and issuing an audible and visual alarm.

[0072] Figure 4This is a flowchart illustrating another embodiment of the teach pendant control method. After the system starts running, it first collects the operator's eye movement data and the teach pendant's IMU attitude data in real time. These two types of data are then fed into a multi-source fusion stage to complete attitude compensation and gaze coordinate calculation, outputting a stable gaze localization result. Subsequently, the system determines whether there is a valid gaze behavior. If no valid gaze is detected, it returns to the multi-source fusion stage to continuously update the gaze coordinates. If a valid gaze behavior is detected, it further matches the UI elements corresponding to the gaze point. After element matching is completed, the system begins to accumulate gaze duration and determines whether it reaches the first-level threshold T1. If not, it continues to identify the corresponding UI elements and update the timer. If the accumulated duration reaches the threshold T1, it provides the operator with basic visual feedback such as highlighting, indicating that the current target has been captured by the system. Based on this, the system continues to determine whether the continuous gaze duration reaches the second-level threshold T2, or whether a confirmation action of a specific blink pattern is detected. If the timer does not meet the threshold or the gaze deviates from the target, it returns to the multi-source fusion and gaze coordinate calculation stage for re-detection. If the second-level confirmation condition is met, the corresponding interface operation command is triggered. After the command is triggered, a safety verification process is initiated. The system determines whether the command is a critical command such as an emergency stop or program start. If it is a non-critical command, it is executed directly. If it is a critical command, it enters a physical button secondary confirmation process, waiting for the operator to complete the confirmation through the physical control. If no physical confirmation signal is received, the command is canceled. If a valid physical confirmation is received, the corresponding command is executed. Regardless of whether the command is executed or canceled, the system will provide feedback on the execution result to the operator and return to the multi-source fusion and line-of-sight coordinate calculation stage to start the next round of interactive detection loop.

[0073] Figure 5This diagram illustrates the safety arbitration and state transition process. Upon system startup, it first enters an initialization calibration mode to calibrate eye tracking and system parameters. After calibration, it enters the standard hybrid enhancement mode, which is the core of normal operation. This mode contains two dynamically switchable sub-states. The default is the eye-tracking-dominant sub-state. When fine-tuning operations such as coordinate adjustments are detected, it switches to the manual auxiliary sub-state. After the fine-tuning operation is completed, it automatically switches back to the eye-tracking-dominant sub-state, thus adapting to different precision operation scenarios. During operation, when the environmental monitoring unit detects moderate or transient interference such as instantaneous strong light, the system switches from the standard hybrid enhancement mode to the high-interference alarm mode. In this mode, interaction reliability is improved through three measures: increasing the eye-tracking confirmation threshold, displaying interference alarms on the interface, and requesting physical confirmation more frequently. After the interference disappears, the system automatically switches back to the standard hybrid enhancement mode. When eye-tracking signals are continuously distorted or lost in standard hybrid enhancement mode, or when manual emergency stop is triggered, or when severe and continuous interference or continuous distortion and loss of eye-tracking signals is detected in high interference alarm mode, the system will switch to safety redundancy mode. In this mode, eye-tracking control functions are temporarily disabled or restricted, and only basic control functions such as physical buttons and joysticks are retained to ensure the most basic safe operation. The system can switch back to standard hybrid enhancement mode only after the subsequent interference is eliminated and the operator actively confirms the restoration.

[0074] The technical solution of this embodiment acquires the operator's eye movement signals and the physical operation signals of the teach pendant. Environmental anti-interference processing is performed on the eye movement signals to output stable gaze positioning data. Based on the gaze positioning data, the eye-gazing behavior of the interactive interface is identified. A multi-level gaze confirmation mechanism generates corresponding interaction intentions. Then, instruction confirmation rules are matched according to the operational risk level of the interaction intentions. Once the rules are met, the corresponding robot control instructions are output. This effectively improves the teaching efficiency of industrial robots, enhances their environmental adaptability under complex and harsh industrial conditions, and ensures industrial-grade safety and reliability of teaching operations. According to an embodiment of the present invention, a teach pendant control device corresponding to the teach pendant control method is also provided. See also Figure 2 The schematic diagram of an embodiment of the device of the present invention shown indicates that the teach pendant control device may include: a data acquisition unit 101, a processing unit 102, and a control unit 103.

[0075] The acquisition unit 101 is configured to acquire the operator's eye movement signals and the physical operation signals of the teach pendant.

[0076] Eye movement signals refer to continuous data collected by the teach pendant that reflects the operator's eye movement state, including information such as gaze direction, fixation position, eye movement trajectory, and blinking status. Physical operation signals refer to operation command signals input by the operator through physical control components such as buttons, joysticks, and knobs on the teach pendant.

[0077] The processing unit 102 is configured to perform environmental interference suppression processing on the eye movement signal and output gaze positioning data.

[0078] Environmental interference suppression processing addresses the drift, distortion, and frame loss issues caused by factors such as vibration, strong light, and electromagnetic interference in industrial environments. It involves correcting and filtering the raw eye-tracking data. The gaze localization data, after interference correction, is the stable gaze point coordinate data mapped to the coordinate system of the teach pendant's interactive interface.

[0079] Industrial environments are often characterized by harsh conditions such as equipment vibration, welding arc light, and strong electromagnetic interference. Raw eye movement signals can suffer from coordinate drift, trajectory jumps, and brief loss of data, leading to positioning errors and false triggers when used directly for interaction. This is a core obstacle hindering the successful implementation of eye-tracking interaction in industrial settings. Therefore, interference suppression processing of the raw signals is essential to output usable gaze positioning results.

[0080] Specifically, the system first filters out abnormal jump values ​​in the original eye movement signal, and then combines data such as the teacher pendant's posture and the intensity of environmental interference to correct the deviation of the gaze coordinates, and finally outputs stable and accurate interface gaze point coordinates.

[0081] In some embodiments, the processing unit 102 performs environmental interference suppression processing on the eye movement signal, including: acquiring the posture motion data of the teach pendant; and dynamically compensating the gaze positioning data based on the posture motion data to filter out positioning deviations introduced by vibration and head shaking.

[0082] Attitude motion data reflects the spatial attitude and motion state of the teach pendant, including information such as yaw, pitch, roll angles, angular velocity, and acceleration. Dynamic compensation synchronously corrects the line-of-sight positioning coordinates based on real-time attitude changes to offset positioning offsets caused by external motion interference.

[0083] Specifically, a motion estimation model of the teach pendant relative to the operator's head is established. Based on real-time posture changes, the gaze positioning data is dynamically corrected in reverse, filtering out positional deviations introduced by vibration and swaying from the original gaze coordinates. The final output is a stable and accurate coordinate of the interface gaze point. The entire compensation process is performed synchronously with eye-tracking signal acquisition to ensure the corrected gaze data is real-time and does not affect the interaction response speed.

[0084] The continuous mechanical vibrations generated by production equipment in industrial settings, coupled with natural hand movements and head posture changes during operator use of the teach pendant, all contribute to shifts in the relative position between the teach pendant and the operator's eyes. Conventional eye-tracking coordinate calculations assume the teach pendant is stationary. When the teach pendant vibrates or shakes, the calculated gaze coordinates drift and jump irregularly, failing to accurately align with interactive elements on the interface, and in severe cases, even preventing normal eye-tracking interaction. Therefore, by fusing and correcting posture and eye-tracking data, interference from equipment vibrations and operator hand movements in industrial settings can be effectively eliminated, significantly improving the accuracy and stability of gaze positioning. This ensures reliable and usable eye-tracking interaction even in industrial conditions with mechanical vibrations, enhancing the teach pendant's environmental adaptability at the signal level and reducing operational errors and false triggers caused by positioning drift.

[0085] In some embodiments, an intelligent filtering algorithm can be superimposed on attitude compensation to further suppress signal noise caused by multi-source mixed interference; or the parameters of the compensation model can be adaptively adjusted according to the frequency characteristics of the field vibration to adapt to different types of vibration conditions.

[0086] The processing unit 102 is also configured to identify gaze behavior on the interactive interface based on the gaze positioning data, and generate corresponding interactive intentions through a preset multi-level gaze confirmation mechanism.

[0087] Fixation behavior is the act of an operator's gaze lingering on a fixed area of ​​the interactive interface. A multi-level fixation confirmation mechanism uses progressively layered judgment conditions to distinguish between unconscious eye movements and conscious operational intentions. Interaction intention refers to the operational goal and direction that the operator intends to perform, as interpreted by the system; it is an intermediate state in the transformation of physiological eye movement signals into control commands.

[0088] Most eye saccades and brief pauses are unconscious physiological behaviors. Directly interpreting every gaze as an operational command would result in numerous false triggers, failing to meet the reliability requirements of industrial control. Therefore, a multi-level, progressive confirmation mechanism is needed to gradually filter out the operator's active and stable gaze behaviors before converting them into explicit operational intentions.

[0089] Specifically, when an operator browses the list of teaching programs, their gaze will quickly sweep across multiple program items. These brief glances will not be judged as valid intentions. When the operator stares at the target program item and maintains a stable gaze, the system will generate an interactive intention to select the program item after multiple levels of judgment.

[0090] In some implementations, the processing unit 102 generates corresponding interaction intents through a preset multi-level gaze confirmation mechanism, including: using the continuous dwell time of the gaze point on a corresponding element of the interactive interface as the identification basis for the gaze behavior, and obtaining the cumulative dwell time of the gaze point on the element; when the cumulative dwell time reaches a first preset duration threshold, marking the element as a candidate target and outputting a visual cue; the candidate target is an interface interactive element that the system initially identifies as the operator's gaze object, which only represents that the system has captured a potential operation intent and does not trigger an actual operation. The visual cue is a visual effect that the interface provides feedback on the recognition status to the operator, used to clearly inform the operator that the current target has been captured by the system. When the cumulative dwell time reaches a second preset duration threshold, an interaction intent corresponding to the element is generated; the second preset duration threshold is greater than the first preset duration threshold.

[0091] Specifically, the system tracks the position of the gaze point in real time. When the gaze point falls within the valid area of ​​a certain interface element, it begins to accumulate the dwell time corresponding to that element. When the accumulated dwell time reaches a first preset threshold (e.g., 500ms), the system marks the element as a candidate target and outputs a corresponding visual cue on the interface, clearly conveying to the operator that "the target has been recognized by the system." At this stage, only preliminary intent recognition is completed, and no operational intent is generated. If the operator keeps their gaze stable and the dwell time continues to accumulate and reaches a second preset threshold (e.g., 800ms), the system determines that the gaze is an active operation by the operator and generates an interaction intent corresponding to that element. If the gaze point moves out of the valid area of ​​the element before both thresholds are reached, the system will immediately reset the accumulated dwell time of the element to zero, remove the candidate target mark and visual cue, and not generate any interaction intent.

[0092] The human eye frequently pauses briefly while browsing an interface or during natural eye movements. These pauses are unconscious physiological behaviors. If the intention to operate is determined solely by a single duration threshold, either the threshold is too short, leading to numerous false triggers, or the threshold is too long, slowing down the interaction response speed, failing to balance interaction efficiency and operational reliability. Furthermore, without feedback from intermediate states, the user cannot know whether the system has recognized their gaze, resulting in a sense of loss of control over the interaction. Therefore, a two-tiered, progressive duration determination mechanism effectively filters out false triggers caused by unconscious squinting and brief pauses, while providing clear feedback on the interaction status through primary visual cues. Without significantly increasing operation time, this greatly improves the accuracy of intent recognition and operational controllability in eye-tracking interaction, making the logic of eye-tracking interaction more aligned with human cognitive habits.

[0093] The control unit 103 is configured to match the corresponding instruction confirmation rule according to the operation risk level of the interaction intent.

[0094] Operational risk level is a safety classification of taught operations based on the severity of the safety consequences of accidental operation. Command confirmation rules are the validity criteria for operations at different risk levels, used to verify whether the operational intent can be converted into a formal control command.

[0095] The safety consequences of different teaching operations vary significantly: errors in menu browsing and parameter viewing do not pose a safety risk, while accidental triggering of operations such as robot startup, permanent parameter writing, and emergency stop recovery could cause equipment damage or even personal injury. Applying strict confirmation rules to all operations would drastically reduce efficiency; conversely, using lenient confirmation rules would compromise the safety of high-risk operations. Therefore, a risk-based matching rule model is adopted to achieve a dynamic balance between efficiency and safety.

[0096] Specifically, the system pre-classifies all teaching operations into risk levels, generates interactive intents, first matches the corresponding risk level, and then calls the instruction validity determination rules corresponding to that level.

[0097] In some implementations, the control unit 103 matches the corresponding instruction confirmation rule according to the operation risk level of the interaction intention, including: if the operation risk level of the interaction intention is low risk, then it is determined that the instruction confirmation rule is met; if the operation risk level of the interaction intention is high risk, then the interaction intention needs to be confirmed through physical operation, and after the confirmation is passed, it is determined that the instruction confirmation rule is met.

[0098] Industrial robot teaching scenarios involve numerous operations with varying safety levels. If all operations employ strict multi-confirmation rules, it significantly increases the number of steps, reduces interaction efficiency, and negates the efficiency gains from non-contact eye-tracking interaction. Conversely, if all operations use lenient confirmation rules, the safety of high-risk operations cannot be guaranteed, and accidental triggering could lead to serious production accidents. Therefore, it is necessary to match different levels of instruction confirmation rules based on the risk level of the operation corresponding to the interaction intent, achieving a dynamic balance between efficiency and safety.

[0099] Specifically, all teaching operations are pre-classified with corresponding risk levels. After generating interaction intentions, their respective risk levels are matched. If the interaction intention corresponds to a low-risk operation, the instruction confirmation rule is satisfied after completing multi-level gaze confirmation, and the operation directly proceeds to the instruction execution stage. This preserves the advantages of convenient and efficient eye-tracking interaction while reducing unnecessary steps. If the interaction intention corresponds to a high-risk operation, gaze confirmation alone is insufficient to determine if the rule is satisfied; the operator must also perform the corresponding physical operation confirmation. Only when both gaze confirmation and physical confirmation are completed simultaneously, and the timing of the two confirmation actions conforms to the valid judgment window, is the instruction confirmation rule satisfied. Through this secondary verification of physical operations, false intentions caused by unconscious eye movements and signal interference can be eliminated at the interaction channel level, fundamentally preventing the accidental triggering of high-risk operations.

[0100] The control unit 103 is also configured to output the corresponding robot control command when the command confirmation rule is met.

[0101] Only interaction intentions that fully comply with the verification rules are considered genuine and proactive operation commands from the operator. Outputting commands without proper verification will result in incorrectly judged, invalid intentions being applied to the robot, creating safety hazards. This step is the final verification stage in converting interaction intentions into device actions, ensuring the validity and security of all issued commands.

[0102] Specifically, the system verifies whether the current interaction intent fully meets all the confirmation conditions of the corresponding level. After the verification is passed, the interaction intent is converted into standard robot control instructions and sent to the robot control terminal for execution.

[0103] This invention, through a dual-path fusion design of eye-tracking signals and physical operation signals, combined with a complete process of environmental interference suppression, multi-level gaze confirmation, and risk classification verification, not only breaks through the limitations of traditional purely manual teaching pendants that are hand-eye bound, have low operating efficiency, and cannot be operated in scenarios where both hands are restricted, but also solves the problems of weak anti-interference ability, high risk of false triggering, and insufficient safety and reliability of single non-contact interaction in industrial environments. It can simultaneously take into account interaction efficiency, environmental adaptability, and industrial-grade operational safety in complex industrial sites.

[0104] In some implementations, the control unit 103 is further configured to: monitor the effectiveness and operating status of eye movement signals in real time, and shut down the eye movement control path and retain only the physical operation control path when preset degradation conditions are met; the preset degradation conditions include any one of the following: continuous failure of eye movement signals, continuous deviation of the line of sight from the effective area of ​​the interactive interface, or receipt of a physical emergency stop input signal.

[0105] The eye-tracking control path is the transmission and execution channel that translates the interactive intent interpreted from eye movements into control commands and issues them. The physical operation control path is the channel through which commands are input and executed via physical manipulation components; it is the system's fundamental safety and redundancy control path. Eye movement signal validity is a state indicator characterizing whether the quality of the eye movement signal meets the requirements of interactive control. A stable signal and accurate positioning are considered valid; continuous frame drops, severe drift, or tracking loss are considered invalid. Operating status encompasses system operating condition information including the current environmental interference intensity, operator gaze state, and operational input status.

[0106] Specifically, the system synchronously tracks the tracking status of eye-tracking signals, the position range of the gaze, and the input status of physical emergency stop. When any one of the three preset degradation conditions is met, the system automatically performs a safety degradation operation: shutting down the eye-tracking control path, blocking all control commands generated by eye-tracking signals, while fully retaining all functions of the physical operation control path, ensuring that the operator can still complete basic operations and safety management through physical components such as buttons, joysticks, and emergency stop buttons.

[0107] The normal operation of non-contact eye-tracking interaction relies on stable eye-tracking signals and continuous eye focus from the operator. However, industrial environments are complex and may experience abnormal scenarios such as strong light interference causing eye-tracking loss, operator shifting of gaze due to equipment, and sudden emergencies. If the eye-tracking control path remains active under abnormal conditions, signal distortion and unintentional gaze deviation may lead to misoperation, or even create safety hazards such as blurred control. Furthermore, if the system completely loses control after an anomaly occurs, the operator will be unable to perform safety operations such as shutdown and reset, leading to more serious consequences. Therefore, through real-time monitoring and automatic switching, a safety redundancy protection system for abnormal eye-tracking interaction scenarios has been constructed. This system can automatically and safely transfer control in the event of eye-tracking signal failure, operator gaze deviation, or emergency situations. This avoids the safety risks caused by accidental eye-tracking triggering under abnormal conditions, while maintaining complete physical operation paths to ensure basic operation and emergency control capabilities. This improves the operational reliability and safety fault tolerance of the hybrid interactive teaching pendant at the system level.

[0108] In some implementations, the control unit 103 is further configured to: set a direction control activation area in the edge area of ​​the interactive interface, map the movement direction according to the position of the gaze in the activation area, map the movement speed according to the duration of the gaze in the activation area, and generate continuous motion control commands for the robot.

[0109] The direction control activation area is a pre-set triggerable region at the edge of the interactive interface, corresponding to a specific movement direction of the robot. Entering this area will trigger motion control in the corresponding direction. Continuous motion control commands are instructions that control the robot's end effector to move continuously along a specified direction, unlike discrete commands that involve single-point jumps.

[0110] For the design of continuous motion control for robots, traditional debugging methods rely on the operator continuously moving physical joysticks to achieve continuous movement. This requires both hands to maintain an operating posture, and the operator must repeatedly switch their attention between observing the robot's end effector and operating the joysticks, resulting in low operational efficiency and hand fatigue. If the movement direction is controlled directly by the viewer's gaze across the entire screen, it is easy to accidentally trigger movement commands while naturally scanning the interface content, causing safety hazards. Therefore, setting a direction control activation area at the edge of the interactive interface not only conforms to the operator's intuitive operation of adjusting direction through edge controls, but also avoids accidental triggering of motion control while browsing the content in the center of the interface, thus balancing operational convenience and operational safety.

[0111] Specifically, the system predefines activation areas corresponding to different movement directions around the perimeter of the interactive interface. When the user's gaze enters and remains in an activation area in a certain direction, the system maps the direction corresponding to that area to the robot's movement direction along the corresponding coordinate axis. Simultaneously, the system maps the movement speed based on the duration of the gaze within the activation area; the longer the gaze remains, the higher the movement speed, until a preset safe speed limit is reached. When the operator's gaze moves out of the activation area or returns to the center of the screen, the robot's movement immediately decelerates and stops, with motion control responding in real-time to the gaze state. The coordinate system used for motion mapping can automatically switch according to the current operation settings, and the interface synchronously displays the currently active coordinate system type to avoid operational confusion.

[0112] In some implementations, the control unit 103 is also configured to: extract the operator's physiological state characteristics from eye movement signals, adaptively extend the duration threshold of multi-level gaze confirmation, and reduce the upper limit of robot movement speed when the operator is detected to be in a state of fatigue.

[0113] Fatigue is a physiological state in which operators experience decreased attention, slowed reaction time, and reduced operational accuracy due to prolonged work. Industrial teaching operations are typically long-duration, making operators prone to fatigue and distraction. In this state, operators' reaction speed decreases, unconscious fixation increases, and fixed interaction parameters lead to a higher probability of accidental triggering. A mismatch between the operational response rhythm and the operator's state poses a safety hazard. Therefore, real-time monitoring of the operator's physiological state based on eye-tracking signals and dynamic adjustment of interaction parameters can adapt to the operational safety needs of different mental states without increasing the operator's workload.

[0114] Specifically, the system continuously extracts multiple physiological characteristics of the operator from eye movement signals and assesses the operator's fatigue level online. When the system detects that the operator's physiological characteristics meet the preset fatigue state judgment criteria, it automatically performs parameter adjustments: proportionally extending the duration threshold of multi-level gaze confirmation to reduce the probability of unintentional accidental triggering; and simultaneously reducing the upper limit of the robot's continuous movement speed to slow down the robot's movement rhythm and allow the operator more time for emergency response. The entire adjustment process is completed automatically without manual intervention from the operator, and a status prompt is simultaneously displayed on the interface to inform the operator that they have entered fatigue adaptation mode.

[0115] Since the processing and functions implemented by the device in this embodiment are basically the same as the embodiments, principles and examples of the aforementioned methods, any details not covered in the description of this embodiment can be found in the relevant descriptions in the aforementioned embodiments, and will not be repeated here.

[0116] The technical solution of this invention employs a dual-pathway interaction architecture combining eye movement and physical operation. First, environmental interference is suppressed on the collected eye movement signals to obtain stable gaze positioning data. Then, a multi-level gaze confirmation mechanism analyzes the interaction intent from the gaze behavior, and the corresponding instruction activation rules are matched based on the operational risk level. After successful verification, robot control commands are issued. This effectively improves the interaction efficiency of industrial robot teaching, enhances the teach pendant's adaptability to harsh industrial environments, and improves the safety and reliability of teaching operations. According to embodiments of the present invention, a teach pendant control system corresponding to a teach pendant control device is also provided. This teach pendant control system may include the teach pendant control device described above.

[0117] Since the processing and functions implemented by the teach pendant control system in this embodiment are basically the same as those in the embodiments, principles and examples of the aforementioned device, any details not covered in this embodiment can be found in the relevant descriptions in the aforementioned embodiments, and will not be repeated here.

[0118] The technical solution of this invention improves the teaching and interaction efficiency of industrial robots, enhances their environmental adaptability in complex and harsh industrial environments, and ensures industrial-grade safety and reliability of teaching operations. According to an embodiment of the present invention, a storage medium corresponding to the teach pendant control method is also provided, the storage medium including a stored program, wherein the program controls the device where the storage medium is located to execute the teach pendant control method described above when it is executed.

[0119] Since the processing and functions implemented by the storage medium in this embodiment are basically the same as the embodiments, principles and examples of the aforementioned methods, any details not covered in this embodiment can be found in the relevant descriptions in the aforementioned embodiments, and will not be repeated here.

[0120] The technical solution of this invention collects two types of signals: eye movement and physical operation. After suppressing environmental interference, it outputs stable gaze positioning data. Relying on a multi-level gaze confirmation mechanism, it identifies the interaction intent and matches corresponding instruction confirmation conditions according to the operational risk level. When the conditions are met, it outputs robot control commands. This effectively improves the teaching interaction efficiency of industrial robots, enhances the adaptability of the teach pendant in complex and harsh industrial environments, and strengthens the industrial-grade safety and reliability of teaching operations. According to an embodiment of the present invention, a computer program product corresponding to the teach pendant control method is also provided, the computer program product comprising a computer program that, when processed and executed, implements the steps of the above-described teach pendant control method.

[0121] Since the processing and functions implemented by the computer program product in this embodiment are basically corresponding to the embodiments, principles and examples of the aforementioned methods, any details not covered in the description of this embodiment can be found in the relevant descriptions in the aforementioned embodiments, and will not be repeated here.

[0122] The technical solution of this invention collects operator eye movement signals and teach pendant physical operation signals. Environmental interference suppression processing is applied to the eye movement signals to obtain gaze positioning data. Based on this data, gaze behavior on the interactive interface is identified. An interactive intent is generated through a preset multi-level gaze confirmation mechanism. Instruction confirmation rules are matched according to the operational risk level of the interactive intent. When the rules are met, corresponding robot control instructions are output. This effectively improves the interactive efficiency of industrial robot teaching, enhances the environmental adaptability of the teach pendant in complex and harsh industrial environments, and improves the industrial-grade safety and reliability of teaching operations.

[0123] In summary, it is readily understood by those skilled in the art that, without conflict, the aforementioned advantageous methods can be freely combined and superimposed.

[0124] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A teach pendant control method, characterized in that, The method includes: Collect the operator's eye movement signals and the physical operation signals of the teaching pendant; The eye movement signals are processed to suppress environmental interference, and the gaze positioning data is output. Based on the gaze positioning data, the system identifies gaze behavior on the interactive interface and generates corresponding interactive intentions through a preset multi-level gaze confirmation mechanism. Match the corresponding instruction confirmation rule based on the operational risk level of the interaction intent; When the instruction confirmation rule is met, the corresponding robot control instruction is output.

2. The teach pendant control method according to claim 1, characterized in that, Environmental interference suppression processing is applied to the eye movement signals, including: Acquire the attitude and motion data of the teach pendant; The gaze positioning data is dynamically compensated based on the posture motion data to filter out positioning deviations caused by vibration and head shaking.

3. The teach pendant control method according to claim 1, characterized in that, The corresponding interaction intent is generated through a pre-defined multi-level gaze confirmation mechanism, including: The continuous dwell time of the gaze point on the corresponding element of the interactive interface is used as the basis for identifying the gaze behavior, and the cumulative dwell time of the gaze point on the element is obtained; When the cumulative dwell time reaches a first preset time threshold, the element is marked as a candidate target and a visual cue is output. When the cumulative dwell time reaches the second preset time threshold, an interactive intent corresponding to the element is generated; the second preset time threshold is greater than the first preset time threshold.

4. The teach pendant control method according to claim 1 or 3, characterized in that, Based on the operational risk level of the stated interaction intent, corresponding instruction confirmation rules are matched, including: If the operational risk level of the interaction intent is low risk, then the instruction confirmation rule is satisfied. If the operational risk level of the interaction intent is high risk, then the interaction intent must be confirmed through physical operation. Once confirmed, the instruction confirmation rule is deemed to be met.

5. The teach pendant control method according to any one of claims 1 to 3, characterized in that, The method further includes: The system monitors the effectiveness and status of eye movement signals in real time. When preset degradation conditions are met, the eye movement control path is closed, and only the physical operation control path is retained. The preset degradation conditions include any one of the following: continuous failure of eye movement signals, continuous deviation of the line of sight from the effective area of ​​the interactive interface, or receipt of a physical emergency stop input signal.

6. The teach pendant control method according to any one of claims 1 to 3, characterized in that, The method further includes: A directional control activation area is set at the edge of the interactive interface. The movement direction is mapped according to the position where the gaze stays in the activation area, and the movement speed is mapped according to the duration of the gaze in the activation area, thereby generating continuous motion control commands for the robot. And / or, The operator's physiological state features are extracted from eye movement signals. When the operator is detected to be fatigued, the duration threshold of multi-level gaze confirmation is adaptively extended and the upper limit of robot movement speed is reduced.

7. A teach pendant control device, characterized in that, include: The acquisition unit is configured to acquire the operator's eye movement signals and the physical operation signals of the teach pendant; The processing unit is configured to perform environmental interference suppression processing on the eye movement signal and output gaze positioning data; The processing unit is also configured to identify gaze behavior on the interactive interface based on the gaze positioning data, and generate corresponding interactive intentions through a preset multi-level gaze confirmation mechanism. The control unit is configured to match the corresponding instruction confirmation rule based on the operational risk level of the interaction intent; The control unit is also configured to output the corresponding robot control command when the command confirmation rule is met.

8. A teach pendant control system, characterized in that, include: The teach pendant control device as described in claim 7.

9. A storage medium, characterized in that, The storage medium includes a stored program, wherein, when the program is executed, it controls the device containing the storage medium to perform the teach pendant control method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the teach pendant control method as described in any one of claims 1 to 6.

Images