Human-multi-robot team interaction-oriented eye-brain fusion input method and system
Through the eye-brain fusion input method, combined with line-of-view tracing and FRP brain-computer interface, the natural and efficient problems of multi-robot control in dynamic environments are solved, and the accurate control of multi-robots and virtual visual interactive interface in dynamic environments is realized, which improves the Human-multi-robot team interaction capabilities.
Patent Information
- Application Number
- PCT/CN2024/079285
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-31
- Filing Date
- 2024-02-29
- Publication Date
- 2024-12-05
AI Technical Summary
In dynamic, unstructured, non-open environment operation scenarios, traditional human-computer interaction methods are difficult to achieve natural and efficient multi-robot control, especially when the operator needs to keep his hands occupied and move silently, lacking accuracy. and the single-modal interaction method cannot meet the control needs of multiple degrees of freedom.
Using the eye-brain fusion input method, the head-mounted augmented reality device combines line-of-sight tracing and FRP brain-computer interface to realize control command input without explicit external visual stimulation, fuses line-of-sight and EEG signals to generate control commands, and uses the augmented reality module. Establish a virtual visual interactive interface to enhance the interaction capabilities of human-multi-robot teams.
Provides additional accurate input channels, reduces the operator's cognitive load, enhances the multi-robot control capability in dynamic environments, realizes natural senseless control command selection and virtual visual interactive interface, and enhances people-multi-multi- The operational capabilities of the robot team.
Smart Images

Figure CN2024079285_05122024_PF_FP_ABST
Abstract
Description
Eye-brain fusion input method and system for human-multi-robot team interaction Technical Field
[0001] The present invention belongs to the field of human-machine fusion interaction and robot control, and specifically relates to an eye-brain fusion input method and system for human-multi-robot team interaction. Background Art
[0002] In dynamic, unstructured, non-open environment operations, firefighting and search and rescue scenarios, it is currently difficult for unmanned swarms to complete operations relying solely on autonomous intelligence. Operators are required to work closely with multiple robots to achieve human-multi-robot team interaction. However, traditional human-computer interaction methods such as keyboards and mice are difficult to complete combat missions. There is an urgent need to develop natural and efficient human-computer interaction technology.
[0003] In scenarios where operators must keep their hands occupied and move silently, there is a lack of natural and accurate multi-robot control input methods. Brain-computer interface technology is a typical hands-free input method. Currently, SSVEP brain-computer interfaces are commonly used to input multiple commands. However, this requires continuous flickering visual stimulation, which is extremely unnatural. This not only affects the ability to observe potential hazards but also easily causes visual fatigue. Therefore, there is an urgent need to research methods that can accurately detect the operator's visual attention without explicit external visual stimulation, thereby enabling accurate selection of control commands in a natural and seamless manner. Fixation-related potential (FRP) is an EEG component that reflects the human brain's processing of cognitive behaviors of interest induced by gaze events. The FRP time-domain waveforms induced by objects of interest and non-interest are clearly distinct, effectively capturing information related to the object of interest. Therefore, using FRP-based brain-computer interfaces can accurately detect the operator's visual attention without explicit external visual stimulation, thereby enabling accurate selection of control commands in a natural and seamless manner.
[0004] Due to the high flexibility of multi-robot systems, current single-modal interaction methods are often unable to control multiple degrees of freedom. Therefore, there is an urgent need to study multi-modal interaction methods to achieve more flexible, natural, and accurate control of multi-robots. On the other hand, operators need to detect the operating status of multiple robots so that they can control them in real time when necessary. Currently, operators mainly rely on visual feedback on the interaction interface to control multiple robots. However, in dynamic, unstructured, and non-open environments, such as combat and firefighting and search and rescue, this can easily lead to problems such as poor performance of multiple robots. Therefore, there is an urgent need to study human-multi-robot control methods to enhance the operational capabilities of human-multi-robot teams in dynamic, unstructured, and non-open environments.
[0005] Summary of the Invention
[0006] To solve the above problems, the present invention discloses an eye-brain fusion input method and system for human-multi-robot team interaction. The operator wears an augmented reality device and realizes control input of multiple robots through an eye-brain hybrid interface, which enhances the operator's ability to control multiple robots when both hands are occupied; at the same time, an augmented reality module is used to establish a virtual visualization interaction interface, which enhances the interaction ability between human-multi-robot teams in dynamic, unstructured, and non-open environments.
[0007] To achieve the above object, the technical solution of the present invention is as follows:
[0008] Eye-brain fusion input method and system for human-multi-robot team interaction, including an interaction interface module, a communication module, and a multi-robot control module;
[0009] The interactive interface module includes a gaze tracking module, an FRP brain-computer interface module, a fusion module and an augmented reality module; the gaze tracking module includes an eye movement signal acquisition module and an eye movement signal processing module, the eye movement signal acquisition module detects the operator's effective gaze events on the interactive interface through the head-mounted augmented reality device and transmits them to the eye movement signal processing module, the eye movement signal processing module identifies the gaze position of the effective gaze event, obtains the identification result and sends it to the fusion module; the FRP brain-computer interface module includes a control instruction menu presentation module, an electroencephalogram signal acquisition module and an electroencephalogram signal processing module, the control instruction menu presentation module induces the operator's cerebral cortex to generate FRP signals by designing a free search paradigm on the interactive interface, the electroencephalogram signal acquisition module detects the FRP signals generated by the operator and transmits them to the electroencephalogram signal processing module Processing module, the EEG signal processing module identifies the FRP signal to extract the effective single-trial FRP signal. The FRP-based brain-computer interface does not require explicit external visual stimulation and can accurately detect the operator's visual attention, obtain the recognition result and send it to the fusion module; the fusion module fuses the output results of the gaze tracking module and the FRP brain-computer interface module, obtains the target selection instruction according to the fusion result, maps it into a control instruction and sends it to the multi-robot control module, and realizes the accurate selection of the control instruction in a natural and unconscious way; the multi-robot control module receives the control instruction sent by the fusion module to control the multiple robots; the augmented reality module displays the control instruction menu presentation interface in real time through the head-mounted augmented reality device, establishes a virtual visualization interactive interface, and can enhance the display of the real working environment;
[0010] The communication module is used to realize data interaction between the interactive interface module and the multi-robot control module;
[0011] The multi-robot control module is used to generate multi-robot control instructions and receive control instructions sent by the fusion module to control the multi-robots.
[0012] Furthermore, the gaze tracking module detects valid gaze events generated by the operator looking at the interactive interface through the head-mounted augmented reality device. Specifically, gaze events are extracted based on a threshold algorithm. The present invention uses three thresholds for gaze event detection: eye velocity, eye acceleration, and eye amplitude. Eye movement signals with eye movement thresholds below these three thresholds are considered gaze events. Gaze events without blinking and exceeding a certain time threshold are then selected as valid gaze events, and the gaze positions of valid gaze events are obtained.
[0013] Furthermore, the FRP brain-computer interface module uses an augmented reality device to present a control command menu interface, providing various multi-robot control modes. Specifically, a free search paradigm is designed within the interactive interface, using an electroencephalogram (EEG) device to induce FRP signals in the operator's cerebral cortex. These FRP signals are then recognized to extract valid single-trial FRP signals. This FRP-based brain-computer interface, without the need for explicit external visual stimulation, accurately detects the operator's visual attention, generates recognition results, and transmits them to the fusion module.
[0014] Furthermore, the fusion module fuses the output results of the FRP brain-computer interface module and the gaze tracking module, obtains the target selection instruction based on the fusion result, maps it into a control instruction and sends it to the multi-robot control module, thereby achieving accurate selection of the control instruction in a natural and seamless manner.
[0015] Furthermore, the augmented reality module displays a control instruction menu presentation interface in real time through a head-mounted augmented reality device, establishes a virtual visualization interactive interface, and creates an augmented reality working environment that integrates the real world and virtual reality.
[0016] The eye-brain fusion input method for human-multi-robot team interaction includes the following steps:
[0017] Step 1: Construct an interactive interface module with an integrated structure; the interactive interface module can integrate eye tracking, FRP brain-computer interface, fusion module and augmented reality module, provide an operation scene under augmented reality, and realize control input of multiple robots. The eye tracking module can track eye movements and gaze movements, realize eye tracking control, and detect the operator's effective gaze events on the interactive interface based on the eye tracking module; the FRP brain-computer interface module can collect FRP signals generated by the cerebral cortex. The FRP-based brain-computer interface does not require explicit external visual stimulation and can accurately detect the operator's visual attention to obtain an effective single-trial FRP signal; the fusion module can fuse the output results of the FRP brain-computer interface module and the eye tracking module, obtain the target selection instruction based on the fusion result, map it into a control instruction and send it to the multi-robot control module, and realize the accurate selection of the control instruction in a natural and unconscious way; the augmented reality module can establish a virtual visualization interactive interface and create an augmented reality operation environment that integrates the real world and virtual reality;
[0018] Step 2: Use a head-mounted augmented reality device to display a control command menu in real time, establish a virtual visualization interactive interface, and enhance the display of the real working environment;
[0019] Step 3: The operator's valid gaze events on the interactive interface are detected by the head-mounted augmented reality device and transmitted to the eye movement signal processing module. The eye movement signal processing module identifies the gaze position of the valid gaze event. The FRP-based brain-computer interface does not require explicit external visual stimulation and can accurately detect the operator's visual attention. The recognition results are obtained and sent to the fusion module.
[0020] Step 4: The FRP brain-computer interface module includes a control command menu presentation module, an EEG signal acquisition module, and an EEG signal processing module. The control command menu presentation module induces the operator's cerebral cortex to generate FRP signals by designing a free search paradigm on the interactive interface. The EEG signal acquisition module detects the FRP signals generated by the operator and transmits them to the EEG signal processing module. The EEG signal processing module recognizes the FRP signals to extract valid single-trial FRP signals, obtains the recognition results, and sends them to the fusion module.
[0021] Step 5: The output results of the gaze tracking module and the FRP brain-computer interface module are fused, and the target selection instructions are obtained based on the fusion results. These instructions are mapped into control instructions and sent to the multi-robot control module, achieving accurate selection of control instructions in a natural and seamless manner.
[0022] Step 6: The interactive interface module realizes data interaction with the multi-robot control module through the communication module;
[0023] Step 7: The multi-robot control module is used to generate multi-robot control instructions and receive control instructions sent by the fusion module to control the multi-robots.
[0024] The beneficial effects of the present invention are:
[0025] (1) The present invention adopts a control command input method using an eye-brain hybrid interface, using EEG and line of sight signals as accurate and powerful control input in scenarios where the operator needs to keep both hands occupied at all times and move silently. This provides the operator with an additional command input channel and enhances the operator's ability to interact with multiple robots.
[0026] (2) The present invention adopts a brain-computer interface based on FRP signals as one of the control command input methods, which can accurately detect the operator's visual attention without explicit external visual stimulation, providing the operator with accurate selection of control commands in a natural and unconscious manner, thereby reducing the operator's cognitive load.
[0027] (3) The present invention uses a head-mounted augmented reality device to display a control instruction menu presentation interface in real time, establishes a virtual visualization interactive interface, and enhances the display of the real working environment.
[0028] (4) The present invention uses an augmented reality module to establish a virtual visualization interactive interface, which enhances the interaction capability between human-multi-robot teams in dynamic, unstructured, and non-open environments. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] FIG1 is a diagram showing the overall system architecture of the present invention.
[0030] Figure 2 is a flow chart of the eye-brain fusion input method for human-multi-robot team interaction. DETAILED DESCRIPTION
[0031] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.
[0032] This invention designs an eye-brain fusion input method and system for human-multi-robot team interaction. This method combines eye tracking, FRP brain-computer interface, and augmented reality as the input interface for multi-robot control commands, while also enabling human control of multiple robots within a work environment. As shown in Figure 1, the system includes an interaction interface module, a communication module, and a multi-robot control module. As shown in Figure 2, the interaction interface module includes an eye tracking module, an FRP brain-computer interface module, a fusion module, and an augmented reality module. The communication module is used to enable data exchange between the interaction interface module and the multi-robot control module. The multi-robot control module is used to generate multi-robot control commands to control multiple robots.
[0033] The eye tracking module includes an eye movement signal acquisition module and an eye movement signal processing module. The eye movement signal acquisition module detects the operator's valid gaze events on the interactive interface through the head-mounted augmented reality device and transmits them to the eye movement signal processing module. The eye movement signal processing module identifies the gaze position of the valid gaze event, obtains the recognition result and sends it to the fusion module.
[0034] The FRP brain-computer interface module includes a control instruction menu presentation module, an EEG signal acquisition module and an EEG signal processing module. The control instruction menu presentation module induces the operator's cerebral cortex to generate FRP signals by designing a free search paradigm on the interactive interface. The EEG signal acquisition module detects the FRP signals generated by the operator and transmits them to the EEG signal processing module. The EEG signal processing module recognizes the FRP signals to extract effective single-trial FRP signals. The FRP-based brain-computer interface does not require explicit external visual stimulation, can accurately detect the user's visual attention, obtain recognition results and send them to the fusion module.
[0035] The fusion module fuses the output results of the gaze tracking module and the FRP brain-computer interface module, obtains the target selection instruction based on the fusion result, maps it into a control instruction and sends it to the multi-robot control module, thereby achieving accurate selection of the control instruction in a natural and seamless manner.
[0036] The multi-robot control module receives the control instructions sent by the fusion module to control the multi-robots.
[0037] The augmented reality module uses a head-mounted augmented reality device to display a control instruction menu presentation interface in real time, establishes a virtual visualization interactive interface, and can enhance the display of the real working environment.
[0038] The working process and principle of the present invention are as follows:
[0039] The operator sends control instructions through the eye tracking module and the FRP brain-computer interface module.
[0040] First, the present invention uses a head-mounted augmented reality device to display a control instruction menu presentation interface in real time, thereby establishing a virtual visualization interactive interface.
[0041] The gaze tracking module detects valid gaze events generated by the operator looking at the interactive interface through the head-mounted augmented reality device. Specifically, gaze events are extracted based on a threshold algorithm. The present invention uses three thresholds for gaze event detection: eye velocity, eye acceleration, and eye amplitude. Eye movement signals below these thresholds are considered gaze events. Gaze events without blinking and exceeding a certain time threshold are then selected as valid gaze events, and the gaze positions of valid gaze events are obtained.
[0042] The FRP brain-computer interface module uses an augmented reality device to present a control menu interface, providing various multi-robot control modes. Specifically, a free search paradigm is designed within the interactive interface, and an electroencephalogram (EEG) device is used to induce FRP signals in the operator's cerebral cortex. These FRP signals are then recognized to extract valid single-trial FRP signals.
[0043] The fusion module fuses the output results of the gaze tracking module and the FRP brain-computer interface module, obtains the target selection instruction based on the fusion result, maps it into a control instruction and sends it to the multi-robot control module. The specific implementation method is as follows:
[0044] Among them, R represents the fusion result, R eeg represents the recognition result of the FRP brain-computer interface module, R eye represents the recognition result of the gaze tracking module, and α represents the set threshold.
[0045] It should be noted that the above content merely illustrates the technical idea of the present invention and cannot be used to limit the scope of protection of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications all fall within the scope of protection of the claims of the present invention.
Claims
1. Eye-brain fusion input method for human-multi-robot team interaction, Features: The eye tracking module, FRP brain-computer interface module, fusion module and augmented reality module are integrated through the interactive interface module as the control command input interface of multiple robots; at the same time, the augmented reality device is worn on the head, and the eye-brain fusion method is used to realize the control of multiple robots.
2. The eye-brain fusion input method for human-multi-robot team interaction according to claim 1, Features: The specific steps include: Step 1: Construct an interactive interface module with an integrated structure; the interactive interface module integrates a sight tracking module, an FRP brain-computer interface module, a fusion module and an augmented reality module, provides an operation scene under augmented reality, and enables operators to control multiple robots at close range; Step 2: Use a head-mounted augmented reality device to display a control command menu presentation interface in real time, establish a virtual visualization interactive interface, and be able to enhance the display of the real working environment; Step 3: Detect the operator's effective gaze events on the interactive interface through the head-mounted augmented reality device and transmit them to the eye movement signal processing module, which is then sent to the fusion module after recognition; Step 4: The FRP brain-computer interface module designs a free search paradigm on the interactive interface to induce the operator's cerebral cortex to generate FRP signals. After identifying the FRP signals, the effective single-trial FRP signals are extracted. The FRP-based brain-computer interface does not require explicit external visual stimulation and can accurately detect the operator's visual attention. The recognition results are sent to the fusion module. Step 5: Fuse the output results of the gaze tracking module and the FRP brain-computer interface module, obtain the target selection instruction based on the fusion result, map it into a control instruction and send it to the multi-robot control module, so as to achieve accurate selection of the control instruction in a natural and unconscious way; Step 6: The interactive interface module realizes data interaction with the multi-robot control module through the communication module; Step 7: The multi-robot control module is used to generate multi-robot control instructions and receive control instructions sent by the fusion module to control the multi-robots.
3. The eye-brain fusion input method for human-multi-robot team interaction according to claim 1, Features: The eye tracking module described in step (1) can track eye movements and gaze movements, realize eye tracking control, and detect the operator's effective gaze events on the interactive interface based on the eye tracking module; the FRP brain-computer interface module can collect FRP signals generated by the cerebral cortex. The FRP-based brain-computer interface does not require explicit external visual stimulation and can accurately detect the operator's visual attention to obtain an effective single-trial FRP signal; the fusion module can fuse the output results of the FRP brain-computer interface module and the eye tracking module, obtain the target selection instruction based on the fusion result, map it into a control instruction and send it to the multi-robot control module, so as to realize the accurate selection of the control instruction in a natural and imperceptible way; the augmented reality module can establish a virtual visualization interactive interface and create an augmented reality working environment that integrates the real world and virtual reality.
4. Eye-brain fusion input system for human-multi-robot team interaction, Features: It includes an interactive interface module, a communication module and a multi-robot control module; The interactive interface module includes a gaze tracking module, an FRP brain-computer interface module, a fusion module and an augmented reality module; the gaze tracking module includes an eye movement signal acquisition module and an eye movement signal processing module. The eye movement signal acquisition module detects the operator's effective gaze events on the interactive interface through a head-mounted augmented reality device and transmits them to the eye movement signal processing module. The eye movement signal processing module identifies the gaze position of the effective gaze event, obtains the recognition result and sends it to the fusion module; the FRP brain-computer interface module includes a control instruction menu presentation module, an electroencephalogram signal acquisition module and an electroencephalogram signal processing module. The control instruction menu presentation module induces the operator's cerebral cortex to generate FRP signals by designing a free search paradigm on the interactive interface. The electroencephalogram signal acquisition module detects the FRP signals generated by the operator and transmits them to the electroencephalogram signal processing module. The electroencephalogram signal processing module identifies the FRP signals to extract effective single-trial FRP signals. The FRP-based brain-computer interface does not require explicit external visual stimulation and can accurately detect the operator's visual attention, obtain the recognition result and send it to the fusion module; The fusion module fuses the output results of the sight tracking module and the FRP brain-computer interface module, obtains the target selection instruction according to the fusion result, maps it into a control instruction and sends it to the multi-robot control module, so as to realize the accurate selection of the control instruction in a natural and senseless way; the multi-robot control module receives the control instruction sent by the fusion module to control the multi-robots; the augmented reality module displays the control instruction menu presentation interface in real time through the head-mounted augmented reality device, establishes a virtual visualization interactive interface, and can enhance the display of the real working environment; The communication module is used to realize data interaction between the interactive interface module and the multi-robot control module; The multi-robot control module is used to generate multi-robot control instructions and receive control instructions sent by the fusion module to control the multi-robots.
5. According to claim 4, the eye-brain fusion input system for human-multi-robot team interaction, It is characterized in that The eye tracking module detects valid gaze events generated by the operator's gaze at the interactive interface through a head-mounted augmented reality device; specifically, gaze events are extracted according to a threshold algorithm, and three thresholds, namely, eye movement velocity, eye movement acceleration, and eye movement amplitude, are used for gaze event detection. Eye movement signals whose eye movement thresholds are all less than these three thresholds are regarded as gaze events, and then gaze events without blinking during the process and exceeding a certain time threshold are screened out as valid gaze events, and the gaze positions of the valid gaze events are obtained.
Citation Information
Patent Citations
Control method based on synchronous acquisition and fusion of multimode man-machine sensing information
CN109634407A
Wearable augmented reality robot control system combining eye movement and brain-computer interface
CN110442232A
Multi-mobile robot cooperative control system based on multi-modal interaction interface
CN111890389A
Eye and brain fusion input method and system for human-multi-robot team interaction
CN116627257A
Fixation-locked measurement of brain responses to stimuli
US20100100001A1
Cited By
Hand motion ability recovery system based on brain-computer interface and virtual reality
CN122261404A
Method and device for feedback to determine state of human-machine interaction based on virtual-real integration, and electronic device
RU2851042C1
