Brain-computer interface

Abstract

A computer brain interface system including: a brain monitoring system for monitoring a user's brain for the detection of a displayed steady-state visually evoked potential structure (SSVEP) by the user from the users visual input; a visual display device for the display of visual information to a user; including a potential command action associated with the SSVEP pattern; and a command execution device for performing the corresponding command action upon detection of the SSVEP pattern by the brain monitoring system.

Description

BRAIN-COMPUTER INTERFACERELATED APPLICATION

[0001] The present disclosure claims benefit of priority to Australian Provisional Patent Application Number: 2023902984 filed 15 September 2023, entitled: BRAIN-COMPUTER INTERFACE, the contents of which are incorporated herein by reference. In jurisdictions where incorporation by reference is not permitted, the applicant reserves the right to add any or the whole of the contents of the Australian Provisional Patent Application Number: 2023902984 as an Appendix hereto, forming part of the specification.FIELD OF THE INVENTION

[0002] The present disclosure relates to steady state visually evoked potentials for use with a braincomputer interface.BACKGROUND OF THE INVENTION

[0003] Any discussion of the background art throughout the specification should in no way be considered as an admission that such art is widely known or forms part of common general knowledge in the field.

[0004] Currently, the main method of interacting with devices and machines is through the use of physical interaction with one or more control devices, typically using hands. For example, a user may use a keyboard and mouse to interact with a computer, a touch screen to interact with a mobile phone or tablet, while another user may use a gamepad to interact with a gaming console. However, such hand based interactions may not be possible for all users, such as for individuals with certain disabilities.

[0005] Brain-Computer Interfaces (BCIs) offer a hands-free solution that allows a user to interact with devices and machines using their brain signals. There are several BCI paradigms that can be used to determine an intention of a user. A brain-computer interface system typically consists of four main components: brain signal acquisition, feature extraction / leaming, intention classification, and an interactive system. Specific machine learning algorithms may be employed to learn the features and extract user intentions based on the chosen brain -computer interface paradigm.

[0006] The primary challenge with BCI systems today is how to select and collect significant features from recorded brain signals. At the moment, the most commonly used paradigms for BCI systems include Event-Related Potential (ERP)), Motor Imagery, slow cortical potentials and visual evoked potentials. Among these features, Steady State Visual Evoked Potentials (SSVEP) are considered to be distinguishable and reliable features for BCI systems.

[0007] The idea of combining SSVEP -based BCI and a robotic system has been an ongoing research topic. Different SSVEP -based BCI systems have been designed to control various devices. For example, the system created by C.-T. Lin, et al., IEEE Transactions on Cognitive and Developmental Systems 11, 375 (2018) uses SSVEP signals to control a robotic arm that helps the paralysed feed themselves. Users focus on flickers of different frequencies displayed on a computer monitor and then issue the corresponding command depending on the brain signal detected. Beyond using computer monitors, SSVEP -based BCI systems have also been used with mobile devices such as smartphones and tablets to increase portability (Y.-T. Wang, Y. Wang, C.-K. Cheng, T.-P. Jung, 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (IEEE, 2013), pp. 5271— 5274). In fact, compared the accuracy of flickering frequencies on mobile devices against that of a computer monitor and found that the tablets performed almost identically to the computer monitors. Over time, more sophisticated and futuristic technology has been developed with the potential to improve the performance of SSVEP BCI systems. Predominantly, the concepts of virtual (VR) and augmented reality (AR) have been evaluated as tools that could be integrated into SSVEP -based BCI systems (see S. Kim, S. Lee, H. Kang, S. Kim, M. Ahn, Sensors 21, 5765 (2021)). For instance, the study by Zhang et al. S. Zhang, X. Gao, X. Chen, Frontiers in Human Neuroscience 16, 908050 (2022). presented a SSVEP -BCI for robot navigation in a maze using AR, which proved the system’s effectiveness in navigating complex tasks and highlighted the potential of AR in enhancing the applicability of BCI systems. Yet another example from B. Fang, et al., IEEE Transactions on Cognitive and Developmental Systems (2022).) integrated an SSVEP-BCI system with AR for more dynamic human-robot interaction. Their system automatically updated the mapping relationship between visual targets and objects, improving flexibility and effectiveness in complex scenarios.

[0008] Steady-State Visually Evoked Potential (SSVEP)enables a brain-computer interface system to achieve a higher information transfer rate, compared to other brain-computer interface approaches.

[0009] SSVEPs are signals that are natural responses by a user to visual stimuli at frequencies ranging from 3.5 Hz to 75 Hz. In general, the visual stimuli are flickering lights and, in response to seeing these lights, the brain generates electrical activity at a similar frequency in the occipital area. A user is presented with a number of flickers, each flickering at a different frequency. A user focuses on one of the flickers, displayed on a computer monitor, and the brain-computer interface determines which of the flickers is being focused on based on the frequency of the flicker. The brain-computer interface is able to issue a corresponding command based on the flicker frequency, depending on the brain signal detected. SSVEP stimulus can be recorded using electroencephalogram (EEG) data using a dry sensor-based EEG device. Beyond using computer monitors, SSVEP -based brain- computer interface systems have also been used with mobile devices such as smartphones and tablets to increase portability.

[0010] Most existing SSVEP -based BCI systems have been tested under laboratory conditions where users are typically immobilised and must remain in close proximity to the computers and devices to ensure the system’ s stability. For the purposes of developing a portable BCI system, it is crucial that the designed experiment allows the participants to move around while recording the EEG data in an uninterrupted fashion.

[0011] While BCIs have been transformative, traditional BCI methods, particularly those utilising SSVEP, are predominantly constrained to indoor environments and utilise 2D displays. These limitations separate the user from actual robotic operations and are not suited for dynamic and varied environments typically essential in telerobotics. Current SSVEP systems often lack the flexibility and portability required for dynamic and varied environments, typically essential in telerobotics. These systems, usually dependent on a personal computer screen for visual stimuli, struggle with portability and flexibility. Alternatives such as AR glasses have been explored, which could potentially address these limitations by allowing stimuli to be displayed on wearable devices, though they face challenges in maintaining accuracy and minimising interference. Additionally, traditional SSVEP systems frequently result in user fatigue and limited engagement due to the unnatural interaction paradigm, which only allows simple command executions and lacks support for sophisticated multi-user operations.

[0012] Among various BCI modalities, SSVEP has been widely recognised for its high information transfer rates and relatively simple implementation. Nonetheless, these implementations often present the flickers in a 2D head-locked (0 DOF) display, thus limiting the interactive capacity and spatial integration.

[0013] Current BCI systems for robot control often lack adequate feedback mechanisms, a limitation that can be manageable when the controlled entity is within the direct line of sight. However, in telerobotic applications where operators do not have visual contact with the robots, this lack of feedback leads to decreased control accuracy and reduced situational awareness. Research by (B. Ahkami, F. Ghassemi, Brain Topography 34, 467 (2021) underscores this issue, investigating error-related potentials in BCIs that occur due to inaccuracies and unintended events. Their findings on the impact of different feedback types on BCI robustness highlight the general inadequacy of feedback mechanisms, especially in error recognition and handling.

[0014] Advanced methods such as Filter Bank Canonical Correlation Analysis (FBCCA) (X. Chen, X. Huang, Y. Wang, X. Gao, IEEE Transactions on Neural Systems and Rehabilitation Engineering 28, 3140 (2020)) have been used in correlation analysis between EEG signals and SSVEP stimuli, improving accuracy while reducing the duration of visual stimuli. Additionally, multi -template algorithms have become increasingly popular for boosting the performance of SSVEP-based BCIs (see L. Chen, et al., Journal of Neural Engineering 18, 066005 (2021), P. Ma, et al., Journal of Neuroscience Methods 371, 109502 (2022). These studies exemplify the integration of AR and SSVEP in BCIs, particularly in systems linked with robotic devices. However, these approaches predominantly use wet sensor-based EEG systems, which pose significant challenges for practical applications due to their complex setup and restricted mobility. Moreover, the design often involves displaying SSVEP stimuli via the HoloLens in a manner akin to a traditional computer screen, which could severely limit the user’s awareness of their surroundings during real-world operations.

[0015] Chen and others have developed a robotic arm assistance system using AR-SSVEP, where users can output target coordinates to the robot through a BCI in an AR environment to assist it in target grabbing (L. Chen, et al., Journal of Neural Engineering 18, 066005 (2021).). However, their AR-SSVEP interface is fixed and not adjustable by the user. Additionally, the object picking in their system is limited to a coordinate chessboard, meaning the robot can only work within a limited space, resulting in low degrees of freedom. Fang and others combined computer vision with AR-SSVEP to develop a robotic grasping system (B. Fang, et al., IEEE Transactions on Cognitive and Developmental Systems, 15, (2023)). However, their system requires that after completing object recognition, the image of the object must be sent to the SSVEP interface for the user to choose before controlling the robot to complete the grasping. Their SSVEP interface is also fixed, has poor operability, and does not allow users to intuitively select objects. Zhang and others developed a system for a robot to navigate a maze by combining AR andSSVEP (Zhang Shangen, Xiaorong Gao, and Xiaogang Chen, Frontiers in Human Neuroscience 16, 908050 (2022)). However, their interface within AR is also fixed in spatial distribution, requiring users to watch the flickering stimuli and then shift their attention to the robot task to receive feedback, which is not intuitive enough.

[0016] It is desired to address or ameliorate one or more disadvantages or limitations associated with the prior art, or to at least provide a useful alternative.SUMMARY OF THE INVENTION

[0017] It is an object of the invention, in its preferred form to provide an improved brain computer interface using steady state visually evoked potentials.

[0018] In accordance with one aspect of the present invention, there is provided a computer brain interface system including: a brain monitoring system for monitoring a user’s brain for the detection of a displayed steady-state visually evoked potential structure (SSVEP) by the user from the users visual input; a visual display device for the display of visual information to a user; including a potential command action associated with the SSVEP pattern; and a command execution device for performing the corresponding command action upon detection of the SSVEP pattern by the brain monitoring system.

[0019] In some embodiments, the SSVEP is displayed in a perspective manner to the user and said visual display device includes augmented reality glasses. In some embodiments, the SSVEP includes a temporally changing grid structure. In some embodiments, the command action includes activities associated with the control of movable autonomous device such as aerial or land based drones.

[0020] In some embodiments, multiple users are provided with corresponding brain monitoring systems and visual display devices with corresponding command actions for the multi user control of an environment. In some embodiments, the display device simultaneously displays multiple separate perspectives of an environment.

[0021] Disclosed is a method of displaying a plurality of flicker regions in an extended reality display, each of the plurality of flicker regions being configured to provide a steady state visually evoked potential for a brain-computer interface, the method comprising: receiving a three dimensional map for at least a portion of an environment visible in the extended reality display, the three dimensional map includingposition information about two or more objects in the scene; selecting a location for each of the plurality of flicker regions in the scene, each of the locations being associated with one of the two or more objects; determining appearance information for each of the plurality of flicker regions at the selected location, the appearance information including a shape of the flicker region based on a perspective associated with the associated object of the two or more objects; and displaying the plurality of flicker regions at the location with the determined appearance, each of the plurality of flicker regions having a different display frequency to allow the brain-computer interface to determine which of the plurality of flicker regions is being viewed by a user of the brain-computer interface.

[0022] In one embodiment, the appearance information of each of the plurality of flicker regions includes a size. In one embodiment, the appearance information of each of the plurality of flicker regions includes a colour. In one embodiment, the size of flicker region is determined based on a size of the associated object. In one embodiment, the colour of the flicker region is selected based on a colour of the associated object. In one embodiment, the size of each of the plurality of flicker regions mimics depth cues, at the location, from the three dimensional map. In one embodiment, determining the appearance information of at least one of the plurality of flicker regions includes determining the shape of the flicker region so the flicker region appears to be positioned on a real world surface in the scene. In one embodiment, at least one of the plurality of flicker regions rotates based on a location of the user.

[0023] In one embodiment, the method further comprises" determining that the user of the braincomputer interface has selected one of the plurality of flicker regions by focusing on the selected flicker region; and applying an action by the brain-computer interface associated with the selected flicker region.

[0024] Also disclosed is a method of displaying a plurality of flicker regions, the method comprising: positioning the plurality of flicker regions in an extended reality environment visible to a user, at least two of the plurality of flicker regions being located at different depths from the user; determining appearance information for each of the plurality of flicker regions based on a location of the user relative to each of the plurality of flicker regions; displaying each of the flicker regions in the augmented reality environment; and determining a selection of one of the plurality of flicker regions using steady state visually evoked potentials detected by a sensor pad on the user.

[0025] In one embodiment, the appearance information is a shape of each of the flicker regions. In one embodiment, positioning the plurality of flicker regions includes positioning each of the plurality of flickerregions on a linked object in the augmented reality environment. In one embodiment, the appearance information is further determined according to a shape of the linked object. In one embodiment, the appearance information of each of the plurality of flicker regions includes a size.

[0026] In one embodiment, the size of flicker region is determined based on a size of the associated object. In one embodiment, the appearance information of each of the plurality of flicker regions includes a colour. In one embodiment, the shape of each of the plurality of flicker regions mimics depth cues, at the location, from the three dimensional map. In one embodiment, determining the appearance information of at least one of the plurality of flicker regions includes determining the shape of the flicker region so the flicker region appears to be positioned on a real world surface in the scene. In one embodiment, the method further comprises: applying an action by the brain- computer interface associated with the selected flicker region.

[0027] As disclosed is a system for display a plurality of flicker regions, the system comprising a processing system configured to: position the plurality of flicker regions in an extended reality environment visible to a user, at least two of the plurality of flicker regions being located at different depths from the user; determine appearance information for each of the plurality of flicker regions based on a location of the user relative to each of the plurality of flicker regions; display each of the flicker regions in the augmented reality environment; and determine a selection of one of the plurality of flicker regions using steady state visually evoked potentials detected by a sensor pad on the user.

[0028] In accordance with a further aspect of the present invention, there is provided a method of controlling multiple spatially movable robot devices via multiple users, the method including: providing each user with a virtual or augmented reality visual device and brain scanning device for monitoring the SSVEP responses of each user to a corresponding virtual or augmented reality; displaying a series of flicker controls for controlling the robot devices in the simulated virtual or augmented reality; and decoding the SSVEP responses of the users to the flicker controls activate actions by the robot devices; and providing a control system so that multiple users can simultaneously control multiple robot devices to accomplish a task.

[0029] In some embodiments the method includes displaying to a user a visual perspective of a robot device. In some embodiments the visual perspective is displayed in a sub window of the virtual oraugmented reality experienced by the user. The multiple users can operate in a master / slave relationship with one user maintaining overall control of the robots.

[0030] In accordance with a further aspect of the present invention, there is provided system for allowing multiple users to control multiple movable robot devices to accomplish a task, the system including: for each user, a virtual or augmented reality display unit for wearing by the user and a corresponding brain monitoring system for monitoring and decoding the SSVEP responses to the visual information displayed to the user; a series of spatially movable robot devices; and a virtual or augmented reality environment, including flicker control activations for the multiple users, the flicker control activations having corresponding robot control activations, upon detection of activation by the users by the brain monitoring system.

[0031] In some embodiments, the movable robot devices include visual environment monitoring devices, with the visual feed being displayed to the user via the augmented reality display. In some embodiments, the visual feed is displayed in a sub window of the virtual or augmented reality experienced by the user.BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Embodiments of the invention will now be described, by way of example only, with reference to the accompanying drawings in which:

[0033] Fig. 1 illustrates a functional block diagram of an example processing system that can be utilised to embody or give effect to a particular embodiment;

[0034] Fig. 2 illustrates an example network infrastructure that can be utilised to embody or give effect to a particular embodiment;

[0035] Fig. 3A and Fig. 3B illustrates an example of a sensor pad that can be used as part of a braincomputer interface;

[0036] Fig. 4 illustrates an example brain-computer interface headband according to one embodiment;

[0037] Fig. 5 illustrates a hardware schematic of the sensor pad of Fig. 3;

[0038] Fig. 6 illustrates flicker region properties according to one embodiment;

[0039] Fig. 7 illustrates a BCI control configuration process according to one embodiment;

[0040] Fig. 8 illustrates a process of BCI control in augmented reality according to one embodiment;

[0041] Fig. 9A illustrates an example of a navigation selection environment using perspective -based steady state visually evoked potentials according to one embodiment;

[0042] Fig. 9B illustrates an example of an urban environment using perspective -based steady state visually evoked potentials according to one embodiment;

[0043] Fig. 10A illustrates a house environment using perspective-based steady state visually evoked potentials according to one embodiment;

[0044] Fig. 10B illustrates a car environment using perspective -based steady state visually evoked potentials according to one embodiment;

[0045] Fig. 11 illustrates a top down view of perspective based flicker regions according to one embodiment;

[0046] Fig. 12 illustrates an AR headset initialisation process as used in the BCI control configuration process of Fig. 7;

[0047] Fig. 13 illustrates an external depth camera initialisation process as used in the BCI control configuration process of Fig. 7;

[0048] Fig. 14 illustrates a manual flicker location selection process as used in the BCI control configuration process of Fig. 7;

[0049] Fig. 15 illustrates a flicker location based on the object detection process as used in the BCI control configuration process of Fig. 7;

[0050] Fig. 16 illustrates a schematic diagram of a brain -computer interface system according to one embodiment;

[0051] Fig. 17 illustrates a schematic overview of one form of BCI-based Hands-free Telerobotic Communication Framework;

[0052] Fig. 18illustrates an example photograph of a single controller use case of the AR SSVEP- based control system;

[0053] Fig. 19 illustrates an example single user point of view;

[0054] Fig. 20 to Fig. 25 illustrate multi-view angle images showcasing the system use case in both a multi-user and multi-robot scenario;

[0055] Fig. 26 and Fig. 27 illustrate a window and window operation mode, including window-in- Window Mode with minimised view Fig. 26 and enlarged view Fig. 27;

[0056] Fig. 28 illustrates one form of placement of SSVEP flickers;

[0057] Fig. 29 illustrates an image showcasing both the pSSVEP from the user’s POV and the auto placement feature visualisation;

[0058] Fig. 30 illustrates a system flowchart for the multi-user control system;

[0059] Fig. 31 illustrates an overview chart of the multi-user system architecture; and

[0060] Fig. 32 to Fig. 34 illustrate an image containing all of the major components used in the system.

[0061] Detailed Description of the Embodiments

[0062] In some embodiments there is provided a hands-free telerobotic communication framework for a telerobotic system. The developed communication framework leverages an augmented brain -robot interface (aBRI). It integrates augmented reality (AR) and portable brain- computer interfaces (BCIs) for dynamic, hands-free control of robots across indoor and outdoor environments. Utilising novelperspective-based steady-state visually evoked potentials (pSSVEP), the system merges real-world visual stimuli with virtual flickers via an AR head-mounted display, facilitating intuitively sending commands to robots through brain signals. The developed communication framework also supports multiple operators in managing multiple robots through a dynamic role transition mechanism between observer and controller roles, adapting to the mission’s requirements as they evolve. The system further develops an approach based on the reinforcement learning algorithm to enhance the classification accuracy for an individual operator. This approach adaptively adjusted the input EEG signal based on the operator’s EEG signal quality before the classification algorithm, thereby improving user interaction and system responsiveness. The embodiments demonstrate the feasibility of integrating AR with BCI for human -robot communication.

[0063] Initial Embodiments

[0064] The following modes, given by way of example only, are described in order to provide a more precise understanding of one or more initial embodiments. In the figures, like reference numerals are used to identify like parts throughout the figures.

[0065] The steady state visually evoked potentials paradigm requires a visual stimulus, known as flicker, to elicit a strong brain response in the occipital area. In a conventional setup, the visual stimulus is displayed on a 2D screen, and the user is required to focus their attention on the screen. Unfortunately, there are two major drawbacks to this setup. The first is that the user has to constantly shift their gaze between the 2D monitor displaying the steady state visually evoked potentials stimulus, and the surrounding environment. For example, the user may need to monitor the system's response or check if the brain-computer interface is executing the correct task associated with the flicker. The second drawback is the dense display of flickers on the 2D monitor can be overwhelming for the user, especially when there are a high number of flickers. This can make it difficult for the user to focus on a specific flicker due to the interference from neighbouring flickers, resulting in signal noise.

[0066] The disclosed brain-computer interface uses flickers, displayed in a flicker region, aligned with real objects in the user's environment using Augmented Reality (AR). The disclosed brain -computer interface merges a scene visible to the user with flicker regions, which are placeable SSVEP flickers. The brain-computer interface allows a user to interact with the flicker regions within the scene to provide control of a user interface.

[0067] Disclosed is a method of displaying flicker regions in an extended reality display. Each of the flicker regions is configured to provide a steady state visually evoked potential for a brain-computer interface. The method includes receiving a three dimensional map for at least a portion of an environment visible to the user in the extended reality display where the use may use a headset, or other systems, to view the extended reality. The three dimensional map includes position information about two or more objects in the scene. A location, in the scene, for each of the of flicker regions is selected with each of the locations being associated with one of the two or more objects. Next, appearance information for each of the flicker regions at the selected location is determined. The appearance information includes a shape of the flicker region based on a perspective associated with the associated object of the two or more objects. Next, the plurality of flicker regions are displayed at the location using the determined appearance. Each of the flicker regions have a different display frequency to allow the brain-computer interface to determine which of the plurality of flicker regions is being viewed by a user of the brain -computer interface.

[0068] Also disclosed is a method of displaying flicker regions. The flicker regions are positioned in an extended reality environment visible to a user. At least two of the plurality of flicker regions are located at different depths from the user. Next, appearance information for each of the plurality of flicker regions is determined based on a location of the user relative to each of the plurality of flicker regions. Each of the flicker regions in the augmented reality environment is displayed and a selection of one of the plurality of flicker regions is determined using steady state visually evoked potentials detected by a sensor pad on the user.

[0069] A particular embodiment of the brain-computer interface, or at least one or more components thereof, can be realised using a processing system, an example of which is shown in Fig. 1. In particular, the processing system 100 generally includes at least one processor 102, or processing unit or plurality of processors, memory 104, at least one input device 106 and at least one output device 108, coupled together via a bus or group of buses 110. In certain embodiments, input device 106 and output device 108 could be the same device. An interface 112 can also be provided for coupling the processing system 100 to one or more peripheral devices, for example interface 112 could be a PCI card or PC card. At least one storage device 114 which houses at least one database 116 can also be provided. The memory 104 can be any form of memory device, for example, volatile or non-volatile memory, solid state storage devices, magnetic devices, etc. The processor 102 could include more than one distinct processing device, for example to handle different functions within the processing system 100.

[0070] Input device 106 receives input data 118 and can include, for example, a keyboard, a pointer device such as a pen-like device or a mouse, audio receiving device for voice controlled activation such as a microphone, data receiver or antenna such as a modem or wireless data adaptor, data acquisition card, etc. Input data 118 could come from different sources, for example keyboard instructions in conjunction with data received via a network. Output device 108 produces or generates output data 120 and can include, for example, a display device or monitor in which case output data 120 is visual, a printer in which case output data 120 is printed, a port for example a USB port, a peripheral component adaptor, a data transmitter or antenna such as a modem or wireless network adaptor, etc. Output data 120 could be distinct and derived from different output devices, for example a visual display on a monitor in conjunction with data transmitted to a network. A user could view data output, or an interpretation of the data output, on, for example, a monitor or using a printer. The storage device 114 can be any form of data or information storage means, for example, volatile or non-volatile memory, solid state storage devices, magnetic devices, etc.

[0071] In use, the processing system 100 is adapted to allow data or information to be stored in and / or retrieved from, via wired or wireless communication means, the at least one database 116. The interface 112 may allow wired and / or wireless communication between the processing unit 102 and peripheral components that may serve a specialised purpose. The processor 102 receives instructions as input data 118 via input device 106 and can display processed results or other output to a user by utilising output device 108. More than one input device 106 and / or output device 108 can be provided. It should be appreciated that the processing system 100 may be any form of terminal, server, specialised hardware, or the like.

[0072] The processing system 100 may be a part of a networked communications system 200, as shown in Fig. 2. Processing system 100 could connect to network 202, for example the Internet or a WAN. Input data 118 and output data 120 could be communicated to other devices via network 202. Other terminals, for example, thin client 204, further processing systems 206 and 208, notebook computer 210, mainframe computer 212, PDA 214, pen- based computer or tablet 216, server 218, etc., can be connected to network 202. A large variety of other types of terminals or configurations could be utilised. The transfer of information and / or data over network 202 can be achieved using wired communications means 220 or wireless communications means 222. Server 218 can facilitate the transfer of data between network 202 and one or more databases 224. Server 218 and one or more databases 224 provide an example of an information source.

[0073] Other networks may communicate with network 202. For example, telecommunications network 230 could facilitate the transfer of data between network 202 and mobile, cellular telephone or smartphone 232 or a PDA-type device 234, by utilising wireless communication means 236 and receiving / transmitting station 238. Satellite communications network 240 could communicate with satellite signal receiver 242 which receives data signals from satellite 244 which in turn is in remote communication with satellite signal transmitter 246. Terminals, for example further processing system 248, notebook computer 250 or satellite telephone 252, can thereby communicate with network 202. A local network 260, which for example may be a private network, LAN, etc., may also be connected to network 202. For example, network 202 could be connected with Ethernet 262 which connects terminals 264, server 266 which controls the transfer of data to and / or from database 268, and printer 270. Various other types of networks could be utilised.

[0074] The processing system 100 is adapted to communicate with other terminals, for example further processing systems 206, 208, by sending and receiving data, 118, 120, to and from the network 202, thereby facilitating possible communication with other components of the networked communications system 200.

[0075] Thus, for example, the networks 202, 230, 240 may form part of, or be connected to, the Internet, in which case, the terminals 206, 212, 218, for example, may be web servers, Internet terminals or the like. The networks 202, 230, 240, 260 may be or form part of other communication networks, such as LAN, WAN, Ethernet, token ring, FDDI ring, star, etc., networks, or mobile telephone networks, such as GSM, CDMA, 4G, 5G etc., networks, and may be wholly or partially wired, including for example optical fibre, or wireless networks, depending on a particular implementation.

[0076] Fig. 3A and Fig. 3B show one form of an example sensor pad 300 that can be used to measure brain activity of a user allowing a brain-computer interface to measure steady state visually evoked potentials. The sensor pad 300 is a portable dry sensor that may be placed on the back of the head of a user to measure activity, typically using a head mount, such as a headband. In one example, the sensor pad 300 may be part of a head mounted display. The sensor pad 300 has a back plate 310 on which are mounted eight EEG sensors 320. Each of the EEG sensors 320 has a number of electrodes 330 that may aid in providing a suitable connection to the head of the user. The sensor pad 300 can be connected to a computer such as the processing system 100, and communicate over a network 202 to send collected information for processing by other computers. In another embodiment, the EEG sensors 320 compriseother types of EEG sensors, such as dry EEG sensors, non-contact EEG sensors, wireless EEG sensors, The electrodes 330 of the EEG sensors may also be made from different conducting and semi-conducting materials, such as gold pin-based electrodes, multiple spikes-based electrodes, solid-gel based electrodes, etc.

[0077] Fig. 4 shows an exemplary BCI (Brain-Computer Interface) headband 400. The BCI headband 400 has a headband 410 on which is mounted a sensor pad 420, that may be the sensor pad 300 described above, as well as a processing pack 430. The processing pack may be used to store a processor, such as a small form factor computer, for receiving signals from the sensor pad 420, along with a transmitter for sending signals to a remotely located service, using a wireless communication protocol such as WiFi. The processing pack 430 may also contain a power source, such as a battery to power the processor, communications system and the sensor pad 420.

[0078] A sensor pad hardware schematic 500 will now be described in relation to Fig. 5. The hardware schematic shows an embodiment of the processing components used to operate a sensor pad, such as the sensor pad 300. The components may all form part of the sensor pad or may be split over two or more locations, such as the sensor pad 420 and the processing pack 430 described above. As shown, the sensor pad hardware schematic 500 has components located on an outside of the sensor pad, sensor pad externals 502 and component located on an inside of the sensor pad, sensor pad internals 504. By incorporating the components into the sensor pad, along with a battery, the pad may operate without any connecting cables.

[0079] The sensor pad externals 502 includes electrodes 510, such as the EEG sensors 320 described above in relation to the sensor pad 300. Leads from the electrodes 510 connect to a fdter 520 that reduces, or removes, noise and any other undesirable artefacts from the electrode signals. The fdtered output from the fdter 520 passes to an amplifier 530 which increases the filtered signal level from the electrodes 510 to a level suitable as input to either a processor 540 or a Bluetooth module 550. In some embodiments the input levels for the processor 540 and the Bluetooth module 550 are the same. The amplified signal from the electrodes 510 may be sent directly to the Bluetooth module 550 for transmission to a remotely located server where the signal is processed. Alternatively, the amplified signal from the electrodes 510 is passed to the processor 540 for processing, before the processed output is sent to the Bluetooth module 550 from the processor 540. The amplified signal from the electrodes 510 is typically converted to a digital signal before being sent to the Bluetooth module 550. The processor 540 may be a digital signal processor that provides analog to digital conversion. While the Bluetooth module 550 is described as using Bluetoothcommunication protocol, other types of communication protocols could be used instead, such as near field communications, WiFi, or even wired communications. The amplifier 530 and the processor 540 may be in the form of a digital signal processing chip or circuit board and in some embodiments the amplifier 530 and the processor 540 are combined.

[0080] Output from the sensor pad is either filtered signals from the electrodes 510, or processed data from the processor 540 and is used as input to determine steady state visually evoked potentials that a user of the sensor pad is focusing on or looking at.

[0081] Fig. 6 shows properties of flicker region 600 that may be shown to a user of a brain-computer interface. A plurality of flicker regions are shown to a user of the brain- computer interface to allow the user to select one of the flicker regions. Each of the flicker regions is linked to specific user input and may be considered to operate in a similar manner to a button in a traditional user interface, where selection of the flicker region triggers an associated action. Once the user of the brain-computer interface selects one of the flicker regions, by focusing on the selected flicker region, an action associated with the selected flicker region is applied by the brain-computer interface. Examples of how flicker regions may be used will be described below.

[0082] Each flicker region 600 can have three properties that can be varied. These properties include a position 610, a size and shape 620 and a frequency 630 of the flicker. Additional properties may also be used. For example, the flicker region 600 can have a colour. Details of the position 610 will be described in more detail below. The size and shape 620 of the flicker region can vary to allow the flicker region to integrate into the environment visible to the user, while the frequency 630 of the flicker is used to determine which flicker region the user is focusing on. The frequency describes how fast the flicker region changes intensity. The intensity change generally uses an on / off change with an equal on and off duration.

[0083] Fig. 7 shows a BCI control configuration process 700 which may be performed by a user with a headset connected to a computer such as the processing system 100, communicating over a network 202, generating an augmented reality environment for display to the user. The BCI control configuration process 700 uses a depth camera to generate a three dimensional map of a scene captured by the camera. The scene is part of the environment that the user is able to view. Flicker regions are placed in the augmented reality environment, which allows the user to select one of the flicker regions by looking at the flicker region. The BCI control configuration process 700 describes how flicker region appearanceinformation of positioned, shape and size, is set up in the augmented reality. Once the flicker regions are set up in the BCI control configuration process 700 the flicker region appearance information can be used during operation of the brain-computer interface, as will be explained below.

[0084] The BCI control configuration process 700 starts with a system initialisation 710 where the headset is configured for operation. The nature of the system initialisation 710 varies depending on the headset hardware and the brain-computer interface. Typically, the system initialisation 710 includes powering up devices such as a display, camera and sensor pad associated with the headset, establishing communication between the systems, such as the headset and any external devices, and running the devices.

[0085] After the system initialisation 710 is complete, the brain-computer interface is ready for a parameter setup 720 where operating parameters of the brain-computer interface are configured. The parameters may include the number of flicker regions, the associated action for each of the flicker regions, i.e. what action the flicker region triggers when the user selects the flicker region, a colour for each of the flicker regions, and a flicker frequency. Additionally, the parameter setup 720 can add or modify an IP address, stored by the headset, for an external device that the headset communicates with.

[0086] There are two ways to perform the parameter setup 720, either manually or by loading predefined values. Manual parameter setup may be done by providing a user interface that allows setting or selection of values for the parameters. Alternatively, pre -defined values can be loaded from a default configuration or from values used previously with the brain-computer interface.

[0087] Next, in a 3D map building step 730, a three dimensional (3D) map of part of the environment is received after being generated, or loaded. The scope of the mapped environment typically includes what is visible to the user when using the headset which is at least a portion of an environment visible in the extended reality display. The mapped environment may also extend beyond what the user can see. This may occur when the system used to generate the three dimensional map and the system used to display the augmented reality do not have a matching scope of view of the scene. In some implementations, the three dimensional map is only for a portion of the scene visible to the user of the headset. The three dimensional map may be generated and stored in the form of a point cloud, a depth map or low polygon mesh spatial mapping. The three dimensional map includes position information for two or more objectsin the scene. Hardware specific techniques for generating the three dimensional map will be described in more detail in relation to Fig. 12 and Fig. 13.

[0088] At a set flicker locations 740, the BCI control configuration process 700 determines where to place the flicker regions in the scene, for viewing by the user in the augmented reality. A location for each of the flicker regions in the scene is selected with, each of the locations being associated with one of the two or more objects. The set flicker locations 740 may be achieved by manually selecting the locations to display the flicker regions or by using an object detection process to locate predetermined objects in the scene and selecting the flicker region based on a classification of the object. An example of such a system would be to locate a coffee machine in the scene and to place flicker location above the coffee machine. Examples of how the flicker locations are set will be described in more detail in Fig. 14 and Fig. 15.

[0089] At a set flicker size 750, appearance information for each of the flicker regions at the selected location are determined. The set flicker size 750 may set a shape of each of the flicker regions, and / or the size as part of determining the appearance information. The shape of the flicker regions may be based on a perspective associated with the associated object of the two or more objects. The size of each flicker region may be based on a distance from the user, in order to provide depth cues. That is, flicker regions located further from the user may be smaller than flicker regions closer to the user. If the user moves towards the flicker regions the size will increase in proportion to the increase of size of the associated object. The shape of the flicker regions may also be set to provide perspective information to the user. In one embodiment, the shape of the flicker region is determined so that the flicker region appears to be positioned on a real world surface in the scene. For example, the flicker region may be transformed to overlap a table surface, a television screen or other flat surface. In one example, the flicker region may be transformed to appear to be positioned on a curved surface, such as a drinking glass or lamp shade.

[0090] In addition, the three dimensional map built at the 3D map building 730 allows each flicker to stay in fixed position in the environment, even when the user shifts to a different location. For example, if the flicker region is placed in front of the user, the flicker is not visible to the user when the user turns around to look in an opposite direction to the flicker.

[0091] At display flicker regions 760 the flicker regions are displayed as part of the augmented reality display to the user. Each of the flicker regions are displayed at the locations as determined at set flicker locations 740 and with an appearance information as determined at set flicker size 750. At display flickerregions 760 the user may then review each of the flicker regions and accept the configuration or return to set flicker locations 740 and reconfigure them. The plurality of flicker regions may be displayed at the location using the determined appearance information. As mentioned above, each of the plurality of flicker regions have a different display frequency to allow the brain-computer interface to determine which of the plurality of flicker regions is being viewed by a user of the brain-computer interface. Once the flicker regions are displayed the brain-computer interface can determine that the user has selected one of the plurality of flicker regions by focusing on the selected flicker region. An action can be applied, by the brain-computer interface, associated with the selected flicker region.

[0092] A BCI control process 800 will now be described in relation to Fig. 8. The BCI control process 800 may use the sensor pad 300 with the BCI headband 400 described above. One or more parts of the BCI control process 800 may be practiced on a computer such as the processing system 100 communicating over a network 202. The BCI control process 800 loads flicker regions into an augmented reality for selection by a user using steady state visually evoked potentials. The location and appearance information for each of the flicker regions may be configured using the BCI control configuration process 700 described above.

[0093] At load environment data 810 information from the BCI control configuration process 700 may be loaded or generated in near real time. The information may include flicker region locations and appearance information along with any three dimensional maps. At capture image frame 820 a depth image of the scene visible to the user is captured using a headset camera or a camera external to the headset. If an external camera is used, the 3D information may be transformed for use from the location of the user.

[0094] At place flicker regions 830 the flicker region locations, received at load environment data 810, are mapped to the information from the capture image frame 820. This allows to the flicker regions to be placed, or positioned, in the augmented reality environment visible to the user. The flicker regions may be located at different depths from the user.

[0095] At determine flicker region properties 840 the appearance information for the flicker regions is adjusted based on the location of each of the flicker regions relative to the user so that the appearance information for each of the flicker regions is based on a location of the user relative to each of the flicker regions. The location of the flicker regions may be dynamically adjusted based on aposition of the objects.For example, when two objects are placed close together, when viewed by the user, the BCI control process 800 determines a space between the closely placed flickers. If the distance is enough to ensure the flickers do not overlap, or influence / interfere with each other, the flicker regions will be superimposed on the objects. If the distance is no sufficient to prevent overlap or interference, the flicker regions will be relocate to a pre-set location within a field of view of the user or have the separating distance adjust to maintain a specified distance between the flicker regions, as viewed by the user. In another example, the sized of the flicker regions is adjusted based on a distance from the user, with the size of the flicker region being varied to mimics depth cues from the three dimensional map according to a location of the user. The size of the flicker region may also be determined based on a size of the associated object, with the size being proportional to the associated object. A large associated object will have a larger flicker region than a smaller associated object.

[0096] The shape of the flicker regions may also be adjusted so that the flicker regions match a perspective of the associated object. In one embodiment, a colour of the flicker region may be selected based on a colour of the associated object. For example, the colour of the flicker region may be selected to match a colour of the associated object. In another example, the colour of the flicker regions may be selected to contrast with the associated object. The determine flicker region properties 840 may also change an orientation of the flicker regions so that the flicker regions face the user. This allows the flicker regions to rotate based on a location of the user. Rotating flicker regions may be used for flicker regions that are placed on surfaces of objects in the three dimensional when the surface of the object is no longer visible to the user. When the surface is not visible, the flicker region may detach from the surface and be rotated to face the user.

[0097] At display flicker regions 850 the flicker regions are incorporated into the augmented reality and displayed to the user in the augmented reality environment. The user may then focus on one of the flicker regions to select the flicker.

[0098] Determining which of the flicker regions the user is focused on occurs at detect steady state visually evoked potentials 860, which is described in more detail below. At the detect steady state visually evoked potentials 860 a selection of one of the flicker regions, using steady state visually evoked potentials detected by a sensor pad on the user, is determined. At steady state visually evoked potential match 870 a test is made to determine if the brain-computer interface has detected a frequency that matches the flicker frequency of one of the flicker regions. If the match is correct, then the BCI control process 800 moves toa trigger associated action 880 where the action associated with the flicker region is performed. If the match is not correct, then the BCI control process 800 returns to capture image frame 820 and repeats.

[0099] The BCI control process 800 may continue to loop to provide the user to interact with the user interface presented to the brain-computer interface. The BCI control process 800 loops to the capture image frame 820 if the user is moving around. If the user is stationary, then the BCI control process 800 may loop to the display flicker regions 850 as updating the captured frame at capture image frame 820, and processing information from the captured frame, may not be required.

[0100] Examples of how flicker regions may be used in a brain-computer interface will now be described in relation to Fig. 9A and Fig. 9B. Fig. 9A shows a navigation selection environment 900 where a user of a brain-computer interface system can select a destination for a mobile robot. The navigation selection environment 900 shows a robot 910 with a first destination 920, a second destination 921 and a third destination 922. To select the destination, the user can look at a corresponding flicker regions, being a first destination flicker 930, a second destination flicker 931 or a third destination flicker 932 to select the first destination 920, the second destination 921 or the third destination 922 respectively. Each of the three destination flickers is a flicker region with a location in the navigation selection environment 900 at, or close to, the respective destination. The user can look at one of the destination flickers to select a destination for the robot 910. Each of the three flicker regions has an associated frequency of the flicker for the flicker region. In one example the flicker frequencies may be 20, 30 or 40 Hz. Typically, the frequencies are between 3.5 to 75Hz. In the example of the navigation selection environment 900, the user looks at the second destination flicker 931 to select the second destination 921 for the robot 910. A route 940 to the second destination 921 is determined, avoiding the obstacles 945.

[0101] Fig. 9B shows an urban environment 950 where a user 960 can see five different shops. The user may select one of the shops to display more information about the shop, such as opening hours, products for sale or even to find the location of competitors. The urban environment 950 has roads 955 with shop A 970, shop B 971, shop C 972, shop D 973 and shop E 974 visible to the user 960. Each shop has an associated flicker displayed in a flicker region, shop A flicker 980, shop B flicker 981, shop C flicker 982, shop D flicker 983 and shop E flicker 984. The user 960 can look at one of the flickers to select one of shops A to E. Each of the flickers has a different frequency so that a sensor pad, such as the sensor pad 300, worn by the user 960 can determine which flicker the user 960 is viewing.

[0102] Additional examples of environments configured for use with the brain-computer interface will be described in relation to Fig. 10A and Fig. 10B. Figure 10A shows an example of a house environment 1000 where a user can use a brain -computer interface to control items typically found in a home. The house environment 1000 shows a user 1010, who may have limited mobility such as being in a wheelchair user. A number of items are located in the house environment 1000, such as door 1 1020, a sofa 1021, a television 1022, a light 1023 and door 2 1024. Visible to the user 1010, through a head mounted display which includes a sensor pad, are flickers, displayed in flicker regions, associated with each of the described items. Shown are a door 1 flicker 1030, a sofa flicker 1031, a television flicker 1032, alight flicker 1033 and a door 2 flicker 1034. The flicker frequency for each of the flickers is unique and allows the braincomputer interface to determine which flicker the user 1010 is intentionally focusing on. When the user 1010 looks at one of the flickers, the brain -computer interface can determine which item is selected and execute an associated action. For example, selecting either door 1 1020 may toggle a lock on door 1 from locked to unlocked, or unlocked to locked. Selecting the sofa flicker 1031 may toggle a light associated with an area around the sofa. In a similar manner, selecting the light 1023 may turn the light on or off. Selecting the television 1022 may turn the TV on or off. Alternatively, if the user 1010 selects the television flicker 1032 the flickers shown in house environment 1000 may be removed and a sub-menu, associated with the selected flicker, is shown to allow more detailed control of the television 1022, such a channel up, channel down, volume up and volume down.

[0103] A car environment 1050 will now be described in relation to Fig. 10B. The car environment 1050 allows a user to control different car systems. Shown in the car environment 1050 are climate controls for decrease temperature 1060, increase temperature 1061, as well as sound system controls decrease volume 1062 and increase volume 1063. Each of the controls has an associated flicker region that display a flicker, with the climate controls having a decrease temperature flicker 1070 and increase temperature flicker 1071 while the sound system controls have a decrease volume flicker 1072 and an increase volume flicker 1073. Such flickers allow an operator of a vehicle to operate the controls. Alternatively, the flickers may be operated by passenger who can see a virtual control panel shown on a back of the driver's seat and allows the passenger to control the car systems.

[0104] Fig. 11 shows a top down view of flicker regions 1100 with a different depths for flicker regions, with respect to a user 1110. Each of the flicker regions show a flicker that can be measured as a perspective-based steady state visually evoked potential, such as the flicker regions used in Figs 9A, 9B, 10A and 10B. A field of view 1120 of the user 1110 shows that flicker region 1 1130, flicker region 21140, flicker region 3 1150, flicker region 4 1160 and flicker region 5 1170 are located with different depths and angled so that each of the flicker regions forms part of the environment. Inside each of the flicker regions are flickers.

[0105] An augmented reality (AR) headset initialisation process 1200 will now be described using the pseudocode of Fig. 12. The AR headset initialisation process 1200 may be used with a headset, such as a HoloLens by Microsoft Corporation and is one example of how to implement the 3D map building 730 of the BCI control configuration process 700. The AR headset initialisation process 1200 scans an environment to build a three dimensional map.

[0106] At line 03 a loop starts that executes until the HoloLens stops scanning the environment, typically when powered off. At line 04 a captured frame of a scene of the environment is retrieved from the sensor. At line 05 the frame is processed to get depth information of pixels in the captured frame. At line 07 a loop starts to process each pixel in the frame. If the current pixel is part of the environment, line 08, then the depth information of the pixel is added to a point cloud. The for loop of lines 07, 08 and 09 may be used to reject some pixels of the captured frame that are not considered to be part of the scene that is to be used, such as when the operating area of the augmented reality is set within a predetermined boundary.

[0107] At line I l a check is made to determine if the point cloud from line 09 has enough points to allow a 3D mesh to be generated. An example of a suitable test may be if the number of pixels in the point cloud exceeds a predetermined threshold, such as half the pixels in the captured frame. Another example may be based on coverage of pixels in the captured frame, where the test is based on how much of the captured frame is covered and the pixel density of the coverage. If there are enough pixels in the point cloud a three dimensional mesh is constructed at line 12, using known techniques, and refined and optimised at line 13.

[0108] At line 15 the AR headset initialisation process 1200 checks to see if the refined and optimized three dimensional mesh has sufficient information for later use in the BCI control configuration process 700, described above. Typically, the three dimensional mesh will be tested to determine if the resolution is sufficient. For example, if the depth information of the three dimensional mesh indicates that the surfaces are too far away, i.e. the depth exceeds a threshold. If the three dimensional mesh has sufficient detail, tested at line 17, then the three dimensional mesh is saved as the three dimensional map at line 18and scanning of the current frame terminates at line 19. At line 21 the current three dimensional map is displayed. The outcome of this process is used for the next stages of the BCI control configuration process 700.

[0109] An external depth camera initialisation process 1300 will now be described in relation to the pseudocode of Fig. 13. The external depth camera initialisation process 1300 is an alternative implementation of the 3D map building 730 of the BCI control configuration process 700. The external depth camera initialisation process 1300 is used when there is an external depth camera, such a RealSense depth camera made by Intel Corporation. The external depth camera is considered to be external as it is separate to the headset. This differs to the arrangement, described above, for the AR headset initialisation process 1200 where the camera is part of the headset. The external depth camera initialisation process 1300 differs to the AR headset initialisation process 1200 as the information from the external depth camera must be translated so that the depth information can be used by the headset worn by the user of the brain-computer interface.

[0110] The external depth camera initialisation process 1300 starts by initialising the camera at line 01. At line 03 a loop starts that will process new frames from the external depth camera until the process terminates. A frame from the external camera is captured at line 04 and the depth information is extracted from the frame at line 05 to as a first step in making a depth map. A loop starts at line 07 to process each pixel of the depth map. At line 08 the depth of the current pixel is converted to a three dimensional point in the external depth camera co-ordinate system. A transform is applied, at line 09, to map the three dimensional point from the depth camera co-ordinate system to a world coordinate system that allows the three dimensional point to be used by the augmented reality system of the headset.[0011 1] A check is made at line 11 to see if the location of the three dimensional point, now in the world coordinate system, is part of the environment used by the augmented reality system. If the three dimensional point is within the scene the three dimensional point is added to the point cloud at line 12.

[0112] The remainder of the external depth camera initialisation process 1300 is the same as the AR headset initialisation process 1200, from line 11 onwards, described above.

[0113] A manual flicker location selection process 1400 will now be described in relation to Fig. 14. The manual flicker location selection process 1400 is an example of how the flicker region locations canbe set manually at the set flicker locations 740 of the BCI control configuration process 700. The manual flicker location selection process 1400 is shown as pseudocode that maybe used to place a flicker region in the scene using hand gestures and a HoloLens 2 by Microsoft Corporation. The HoloLens 2 is used due to the hand tracking functionality included in the system.

[0114] The manual flicker location selection process 1400 starts with an initialisation of the hand tracking system at line 01 before a loop starts at line 02. The loop ensures the rest of the manual flicker location selection process 1400 runs continuously while the HoloLens is active or until the manual flicker location selection process 1400 is manually stopped by the user. At line 03 hand gestures and hand movements are tracked by the system.

[0115] At line 04, a test is made to determine if a hand gesture is detected. If a hand gesture is detected then the nature of the hand gesture is determined, with different hand gestures triggering different actions. The hand gestures include pinch, grab, thumb-up, a fist or a wave. At line 05 a test is made to determine if the gesture is either a pinch or a grab. If so, at line 06, a direction is determined from the pointing vector of the hand, using ray casting. At line 07 a test is made to see if the pointing vector intersects with a surface in the 3D map. If the pointing vector does intersect, at line 08, an object is placed on the surface. Line 09 determines is the gestures is a grab gestures and, if the gesture is a grab gesture, the object on the surface can be moved as during the grab gesture, at line 10. Line 11 tests to see if the gestures is a pinch, and if the gestures is a pinch at line 12, the object is released at the current location. When the object is released the object is remains at the current location.

[0116] At line 13 a test is made to determine if the gesture is a thumb up. If the gestures is a thumb up the object, if one has been placed during an earlier loop, may be rotated by the user rotating the thumb at line 14. The user may use the thumb direction to select an axis of rotation and then rotated the object about the selected axis. At line 15 a test is made to determine if the gesture is a fist. If the gesture is a fist, the object may be resized at line 16. To resize the user may move the fist along an axis towards or away from object. Alternatively, the user may resize using a distance between fingers on the alternative hand to the fist. The last gesture to check is a wave gesture, which is tested by the if statement at line 17. The wave gesture is used to delete, or alternatively to hide, the object at line 18.

[0117] At line 19 the state of the object is updated on the augmented reality display for viewing by the user. The manual flicker location selection process 1400 then loops again to determine if the user has updated their gesture.

[0118] A flicker location selection process 1500 will now be described in relation to the pseudocode of Fig. 15. The flicker location selection process 1500 is an example of how the flicker region locations can be set at the set flicker locations 740 of the BCI control configuration process 700. The flicker location selection process 1500 uses an external RealSense depth camera made by Intel Corporation to place the flicker regions in the scene, automatically, without the locations being set by a user of the system.

[0119] The flicker location selection process 1500 stars at line 01 where the depth camera is initialised. At line 02 a loop starts. The loop will process frames from the depth camera and continue to operate while until exited by the user or the brain -computer interface is powered off.

[0120] The first line of the loop, line 03 gets a current frame from the depth camera, Next the frame data is processed for object detection at line 04. The object detection may be a known object detection process, such as fast R-CNN or You Only Look Once (YOLO). The object detection process provides a location and orientation of the objects detected in the frame. At line 08, if any of the detected objects are within a suitable range, that is, the objects are within the frame of the camera and are within a predetermined distance of the user. At line 09 an object is placed at the location of the detected object. The object is a location at which a flicker region may be placed. At line 10 the orientation of the flicker object, and the flicker region, is adjusted to match the determined orientation of the object.

[0121] Once the flicker object has been placed in the scene, the augmented reality display is updated to show a flicker region at each of the flicker object location, using the flicker object orientation. The flicker location selection process 1500 then loops to process a new frame from the depth camera.

[0122] In one embodiment, the flicker location selection process 1500 also determines a classification of the object detected in the frame. The object classification can be used to select a suitable flicker to associate with the object, using the classification. In one example, the flickers may have object information associated with them that is matched with the object classification. In the house environment 1000, a flicker may include data that the flicker is relevant to a door, a television, or a light. If the object is determined to be a light, then the flicker used at a flicker region linked to the light object. If more thanone flicker region could be linked to an object, for example door 1 flicker 1030 or door 2 flicker 1034, the user may be provided with an option to select which of the two flickers apply to the flicker region on the door object.

[0123] Steady state visually evoked potentials matching process

[0124] One method of matching the flicker frequency of a flicker displayed in a flicker region in focus by the user with one of the frequencies of the flicker regions is to use Canonical Correlation Analysis (CCA). CCA is a multi-variable statistical method that finds linear correlations between two data sets. CCA can help decrease the influence of the channel location and the time window length on the classification process in a steady state visually evoked potentials based brain -computer interface. CCA enables finding an underlying correlation between two sets of variables. For two sets of variables X and Y, CCA consists of finding vectors w and v that maximize the correlation between wTX and v'Y. The inputs are both the reference signal (i.e., the frequencies of each of the flicker regions) plus a multi- channel EEG signal from the sensor pad. The output of the CCA model classifies which frequency has the highest correlated with the EEG signal, which, in turn, dictates which flicker the participants were focused on. A maximum correlation coefficient is determined by:

[0125] where p indicates the maximum correlation coefficient, X refers to the set of multi-channelEEG data signals, and Y is a reference signals set that have the same length as X:

[0126] Here, H is the number of harmonics, the classifier’s output is recognized as:

[0128] CCA allows recognition of the flicker region frequency that the user is intentionally focusing on and allow selection of the flicker region by the user. Once the selected flicker region is identified the brain -computer interface can perform any actions associated with the flicker region.

[0129] Further Variations

[0130] While the above system is described in relation to an augmented reality system, the system may also be implemented in an extended reality system. Extended reality includes augmented reality, virtual reality, and mixed reality systems.

[0131] The navigation selection environment 900, urban environment 950, house environment 1000 and car environment 1050 show flicker regions in an augmented reality with varying depth from the user. In one embodiment the appearance information for each of the flicker regions is varied based on a distance from the user, with closer flicker regions being larger than flicker regions further away. In one embodiment, the flicker regions can have a shape that matches the associated objects. That is, the outline of the flicker region is the same as the associated object. Alternatively, the outline of the flicker region may be similar to the outline of the associated object. For example, in the urban environment 950 the door 1 flicker 1030 and door 2 flicker 1034 may have a door shape, the sofa flicker 1031 has a sofa shape, the television flicker 1032 has a television shape and the light 1023 has a light shape. Alternatively, or in addition, the flicker regions may have a perspective that matches the object. In the example of the house environment 1000, the door 1 flicker 1030 and sofa flicker 1031 may have a perspective that matches the door 1 1020 and the door 2 1024 respectively.

[0132] The brain-computer interface described above uses a headset, such as HoloLens, with a sensor pad. Alternative augmented reality or extended reality display systems may be used. In one example, a mobile phone may be placed over the eyes of a user and an augmented reality or extended reality displayed to the user.

[0133] Some aspects described in relation to the BCI control configuration process 700 and the BCI control process 800 may be used interchanged. For example, appearance information configuration described in relation to BCI control configuration process 700 may also be used in the BCI control process 800 and appearance information configuration described in the BCI control process 800 may be used in the BCI control configuration process 700.

[0134] Reference is now made to Fig. 16, which shows a schematic diagram of a brain- computer interface system 1600 according to one embodiment. The brain-computer interface 1600 comprises an EEG system 1610 sending signals to an Al system 1620 engaging with one or more interfaces 1630 for controlling the interactive system 1640. The EEG system comprises one or more sensors 1612 for reading EEG signals of a user. The EEG signals are amplified by an amplifier 1614 and forwarded to the Al system 1620 for processing. The Al system 1620 comprises a signal processing circuit 1622 and a classifier 1624 to determine one or more events signified by the EEG signals received. Next, the events and data generated and processed by the Al System 1620 are sent to the interfaces 1630. The interfaces 1630 can be a monitor 1632 or an extended reality device, such as virtual or augmented reality, device 1634. The user may confirm, add, cancel, alter, or control the events through the interfaces 1630. The interfaces 1630 send the events to the interactive system 1640. The interactive system 1640 may comprise a single agent 1642, multiple agents 1644, or a swarm 1646. The single agent 1642 comprises a single device or robot. The multiple agents 1644 may comprise two or more devices or robots. The swarm 1646 comprises multiple devices or robots exhibiting collective behaviour resulting from local interactions between the robots / devices and between the robots / devices and the environment in which they act.

[0135] Advantages

[0136] The described brain-computer interface using perspective-based steady state visually evoked potentials may eliminate the need for constant shifts in gaze by a user between a flicker stimulus screen and the actions / locations of real-world objects. By incorporating flickers in an augmented reality or extended reality, a user may seamlessly interact with the flicker stimuli while remaining engaged with their surroundings.

[0137] Furthermore, perspective-based steady state visually evoked potentials approach can provide optimal distances between flickers to prevent interference and confusion caused by neighbouring flickers, thereby enhancing the signal quality and reducing noise.

[0138] Further Alternative Embodiments

[0139] The foregoing embodiments provide for a significant advance in human -machine interaction allowing for robots to be controlled by the brain signals of users in real-time. This technology holds immense potential to revolutionise various fields, from manufacturing to disaster rescue.

[0140] Telerobotic systems, which allow for remote control of robots, benefit significantly from the integration of brain-computer interfaces (BCIs). Unlike traditional control methods like keyboards and controllers, the BCIs offer a hands-free, intuitive approach to system operation, enhancing accessibility for all users, including those with physical disabilities. This method reduces physical and cognitive load, as users can control robots directly through brain signals, aligning the operation more closely with natural cognitive processes. Consequently, BCIs not only simplify complex tasks but also improve operational efficiency and user comfort, making them an optimal choice for controlling telerobotic systems.

[0141] The present SSVEP (pSSVEP) utilises 6 DOF world- anchoring flickers that can be viewed from any direction and orientation, even while moving. This innovative use of pSSVEP integrates visual cues directly within a three- dimensional AR environment, enhancing the intuitiveness of the interface by maintaining a connection between the user’s visual focus and the robot’s actions. This advancement not only allows for more dynamic and immersive interaction, as the flickers remain consistent in the environment regardless of the motion of the user, but it also extends the usability of SSVEP to outdoor settings and multi-user scenarios.

[0142] To address challenges of the prior art, the present system of this alternative embodiment implements a window-in-window solution that streams the robot’s first-person view directly onto an augmented reality (AR) display. This setup provides real-time visual feedback, enhancing control precision. Additionally, it includes a representation of the robot’s digital twin, offering a third-person perspective that helps controllers understand the robot’s spatial orientation and position. This dual-view approach not only improves interaction quality but also allows users to navigate around the space, choosing different angles and distances from which to observe or control the robot, thereby enhancing the engagement and effectiveness of the interaction.

[0143] Furthermore, unlike typical BCI systems that are designed for single-user operation, our system introduces capabilities for multi-user collaboration. This allows multiple controllers to jointly operate a single robot or manage multiple robots simultaneously. Such multi-user, multi-robot capabilities significantly expand the potential applications of BCI systems in complex environments, promoting teamwork and enhancing operational flexibility across various telerobotic scenarios.

[0144] Research into dynamic user interfaces in BCIs, highlights that dynamic interfaces, such as the one in our system, may significantly reduce operational errors and decrease time delays. By incorporatingpSSVEP into our AR-BCI system, we enable a more dynamic and flexible interaction model that supports complex interactions within both single-user and multi-user contexts. This approach allows precise control over multiple robotic agents and facilitates collaborative operations in a shared virtual and physical environment, thereby addressing the critical gaps identified in existing BCI applications for telerobotics.

[0145] Fig. 17 presents a schematic example 1700 of a proposed BCI-based hands-free telerobotic communication framework. To operate the system, each individual user 1701 is first equipped with a HoloLens headset 1702, which are fitted with dry EEG electrodes 1703 attached to the back of the head strap and positioned at the occipital area of the head. These EEG electrodes 1703 are connected to a MentaLab wireless amplifier 1704 (available at htps : / / mentalab.conv)-

[0146] The EEG device 173 in Fig. 17, facilitates the transmission of EEG data to a SSVEP decoding system (not shown). To control the robots, users must initially set the 3D location of the pSSVEP flickers 1705 and world anchors 1706 within their surrounding environment using the HoloLens. This setup not only allows users to visualise the flickers' locations but also synchronises the HoloLens AR environment with the robots and other users, ensuring a cohesive operational space. Subsequently, the robots, the HoloLens, and the EEG system are connected via the same local network, enabling users to commence control. This methodical setup provides a robust framework for the precise and interactive control of robotic systems through the AR-BCI system. The communication framework is designed to be compatible with various types of robots, such as drones 1708 and ground vehicles 1709.

[0147] In this embodiment there is introduced an innovative hands-free BCI system integrating SSVEP with augmented reality interfaces, termed perspective SSVEP (pSSVEP). This system blends the immediate surroundings of the user with the dynamically placed SSVEP flickers, allowing users to navigate robots across various settings using the spatial mapping capabilities of the AR device. The pSSVEP also serve as spatial anchors that the robot can recognise and position itself relatively. To accommodate multiple users, the system incorporates multi- user control in AR, enabling two participants to simultaneously interact with and control the robots, enhancing collaborative tasks and shared experiences in real-time environments. To enhance command accuracy from brain signals, a reinforcement learning (RL) algorithm that filters out noise, ensuring that only clear signals are used for command classification.

[0148] Whilst reinforcement learning was used in the embodiment, various other classification algorithms could be used.

[0149] The reliability of the system can be improved with multiple participants using a personalised machine-learning model trained on their EEG data. The primary goal is to establish pSSVEP as an advancement towards a more immersive and portable BCI control system. The demonstrated augmented brain-robot interface (aBRI) enables direct interaction between users and robots via brain signals in a dynamic, non-laboratory environment.

[0150] In practical applications such as manufacturing environments, the hands-free control of our system can prove invaluable. Consider a scenario where a user operating a forklift needs to open a gate without disengaging from vehicle controls. Traditionally, this would require stopping the forklift and manually opening the gate, thereby disrupting workflow and reducing efficiency. With our system, the user can remotely command the opening or closing of factory doors — through simple gaze-directed pSSVEP flickers, all while continuously operating the forklift. This functionality not only streamlines complex multitasking operations but also enhances safety by reducing the need for physical interaction with multiple control systems. By integrating pSSVEP -based controls into the user interface, the system provides a robust solution for complex control environments where hands-free operation is not just a convenience but a critical requirement. This approach not only caters to the needs of able-bodied users but also opens up new possibilities for users with physical limitations, making technology more inclusive and adaptable to a wide range of user scenarios.

[0151] AR Interface Design

[0152] The experimental framework of our augmented reality brain-computer interface system enhances control over unmanned aerial and ground vehicles through an innovative user interfaces. Unlike traditional SSVEP BCI systems that display control flickers on a two- dimensional screen, the present system integrates these visual cues directly within the user's three- dimensional space. This integration bridges the disconnection often experienced between the flickers and the robot’s actions or locations, thereby enhancing intuitiveness. By merging real-world and virtual elements seamlessly, the system supports complex interactions within both single-user and multi-user contexts, enabling precise control over multiple robotic agents and facilitating collaborative operations in a shared virtual and physical environment.

[0153] Single User Mode

[0154] Turning now to Fig. 18, there is illustrated the system in single-user mode, with a user 1804 using the AR system 1805 to control 3 drones e.g. 1801, 1802, 1803, the system utilises perspective steadystate visual evoked potentials (pSSVEP), to create intuitive waypoints for robot navigation. This allows a single operator to control multiple robots simultaneously, as demonstrated in Fig. 18, where one user efficiently manages three drones through the portable AR-BCI setup. This mode underscores the robustness of individual control and enhances the operator's ability to manage tasks effectively.

[0155] As shown in Fig. 19, virtual drones act as digital twins, providing a path preview in AR when controlling robots in direct line-of-sight and accurately representing their actual locations in 3D space for remote operations. The implementation of hands-free control via BCI represents a significant advancement, enhancing the operator's engagement with complex environments while maintaining full situational awareness. Fig. 19 illustrates the Hololens point of view and shows three drones 1902-1904 controlled by BCI interface 1906 - 1908.

[0156] Multi-User Collaboration Mode

[0157] Turning now to Fig. 20, there is illustrates a multi user mode with two users 2001, 2002. In the multi-user setting, the system's flexibility is enhance. It begins with an initial user 2001, designated as the host, who configures the system settings, including the types and velocities of robots. This foundational setup ensures that subsequent users 2002, who initially assume observer roles, can seamlessly transition into controllers based on operational needs. This feature is important in environments where quick adaptation to changing scenarios is critical. Adynamic user interface supports robust interaction among multiple users, allowing seamless transitions between observing and controlling. Such flexibility is essential for tasks that require collective input and collaboration, adapting effortlessly to varying operational roles as required.

[0158] The AR system’s capability of supporting six degrees of freedom (6-DOF) enhances its suitability for mobile teleoperation, enabling users to interact with and view virtual objects from any angle. This feature is critical for the mobile teleoperation of robots, as it allows operators to control robotic units without requiring line-of-sight visibility. Operators can interact with the virtual counterparts of the robots, which accurately display the robots' 3D poses in the augmented reality setting, even when operating fromremote locations. This flexibility allows the system to accommodate various operational environments, whether they replicate the actual layout where the robots are located or are completely different spaces, such as empty rooms. The AR headset's ability to project both the remote environment and the virtual counterpart of the robot enhances the user’s control capabilities, allowing them to navigate and manage the system as if they were directly on-site with the robots.

[0159] The innovative integration of AR and BCI technologies in the system offers significant advantages for the mobile teleoperation of robots. Enabling a new level of interaction that transcends traditional control paradigms, allowing operators to manage robotic systems remotely. The setup facilitates direct visualisation of robots, paths, and waypoints, enhancing control precision. The capability to operate multiple robots, either individually in single-user mode or collaboratively in multi-user settings without direct line-of-sight, not only expands the operational scope but also greatly improves the system’s effectiveness and efficiency. Additionally, the hands-free, intuitive control enabled by SSVEP, coupled with detailed environmental projection via AR, allows operators to execute complex manoeuvres and tasks from any location. This setup positions the system as a powerful tool for advancing remote robotic management.

[0160] Robot Control and Digital Twin

[0161] Within the AR interface, the controller can initiate system functions, such as positioning perspective SSVEP flickers as three-dimensional waypoints to navigate both the robot digital twins and physical unmanned vehicles. This seamless integration of virtual planning and real-world execution not only enhanced operational efficiency but also facilitated safe, simulated testing environments prior to actual deployment.

[0162] The control of real robots is managed through the transmission of a continuously updating array of coordinates and orientation data. This information, which dictates the navigation paths, is derived directly from their virtual counterparts displayed in the AR environment. Leveraging the robust spatial mapping capabilities enabled by the HoloLens, which includes simultaneous localisation and mapping (SLAM) and comprehensive environmental scanning, the system constructs a dynamic 3D occlusion map. This map significantly enhances environmental awareness, enabling connected robots to navigate and avoid obstacles effectively, even in the absence of innate sensing capabilities.

[0163] Fig. 20 to Fig. 24 illustrate one multi user scenario, with Fig. 20 showing two operates users 2001, 2002 controlling a drone swarm.

[0164] Fig. 21 illustrates the corresponding simulated virtual environment 2100 with users 2101, 2102 with drone swarm activation balloons e.g. 2103.

[0165] Fig. 22 illustrates the corresponding world view of operator 1 when viewing the scene via the Hololens showing virtual counterparts and waypoints 2203, 2204.

[0166] Fig. 23 illustrates the corresponding view 2300 of operator 2 when viewing the scene via a Hololens showing virtual counterparts e.g. 2301 and waypoints e.g. 2302.

[0167] Turning now to Fig. 24, there is illustrated the third person view of a simulated drone scouting mission.

[0168] Fig. 25 illustrates a real life view of the corresponding rover navigating to the drones scouted location.

[0169] The system is able to provide many use cases for both multi-user and multi-robot scenarios. Furthermore, the system design accommodates the management of various types and numbers of robots simultaneously. The arrangement shown with respect to Fig. 20 to 25, illustrates a scenario where the system controls three Crazyflie drones executing complex swirling manoeuvres to reach six designated waypoints. These images capture a real-life third-person view, a simulation from the same angle, and two views from HoloLens controllers, all taken at the exact moment, demonstrating the integration of AR and BCI in managing a swarm of drones.

[0170] Fig. 24 and Fig. 25 illustrate a collaborative task involving two users. In this scenario, the first user employs a drone equipped with a camera for aerial scouting, while the second user controls a Leo Rover on the ground, navigating to the location identified by the drone. This demonstration underlines the flexibility of the system, enabling different robots to be used in tandem to complete complex missions like search and rescue operations.

[0171] The figures provide a dynamic demonstration of the system's operational capabilities in a simulated search and rescue scenario. It shows that operator 2 is controlling a drone to a location usingthe BCI system. Subsequently, operator 1 takes control of the Leo Rover, navigating it to the ground location directly beneath the drone. The figures illustrate how a such coordinated control between aerial and ground units could be effectively deployed in search and rescue missions. This demonstration highlights the potential of the system to facilitate complex, multi-agent operations in real-time, showcasing its adaptability to critical real -world applications.

[0172] These tests have successfully demonstrated not only the system's capability to manage diverse robotic assets, such as unmanned aerial vehicles (UAVs) and unmanned ground vehicles (UGVs) but also its adaptability for simultaneous control by multiple users. This capacity to manage diverse robotic assets underscores the sophisticated technological foundation of the AR control system and its potential for broad application in complex operational environments.

[0173] Further Hands-free Features

[0174] Even though the system setup process can be performed using hand-tracking and hand raycasting to manipulate and spawn the virtual objects at the user’s desired position, the system refines the interface for robotic control by fully integrating hands-free functionality alongside optional hand tracking for various interactive tasks. By leveraging SSVEP for intuitive user interactions and combining this with advanced object recognition technologies and precise distance measurements, the system can operate entirely without manual input. This dual-capability design allows for the autonomous setting of waypoints for robotic navigation, which are visualised as pSSVEP flickers within the augmented reality interface. These waypoints, along with other world anchors, are automatically placed, significantly streamlining the setup process, and reducing the need for manual configuration.

[0175] In addition to navigational controls, the system extends its hands-free capabilities to critical operational aspects, including robot selection, user role swapping, and accessing remote robot point-of- view (POV) camera feeds. These functions can be accessible through an SSVEP- enabled side menu, specifically designed to facilitate all decision-making processes without requiring hand interaction. This menu allows users to seamlessly switch between different robots, change operational roles among multiple users, and view live camera feeds from robotic units, all through simple BCI interactions.

[0176] The impact of this hands-free functionality extends significantly, particularly in scenarios where manual control is impractical or impossible. For instance, in situations where users are engagedwith tasks requiring both hands, such as operating machinery or driving, the system still allows for the effective teleoperation of robots or other equipment. This capability is crucial not only for enhancing operational efficiency but also for ensuring accessibility for users with disabilities who might be unable to use traditional control interfaces.

[0177] Window in Window Robot POV in Augmented Reality

[0178] Turning now to Fig. 26 and Fig. 27, the system can further provide a feature termed “Windowin-Window Remote Robot Control”, which enhances the teleoperation of robots through an AR interface. This function leverages the HoloLens 2 to facilitate the wireless reception of live video feeds 2601 from robot cameras directly into the user's AR display. To optimise the user’s field of view and minimise distractions, the video feed initially appears in a minimised format 2601 at the bottom -right comer of the display.

[0179] As shown in Fig. 27, upon the shift of user attention towards this minimised window, the interface automatically enlarges the video feed to the centre of the AR display 2702. Simultaneously, a side menu containing SSVEP flickers 2703 slides in from the left, facilitating intuitive control over the operations of the robot. To ensure the focus remains on the essential controls, background SSVEP flickers and the rendering of unrelated virtual objects e.g. 2704 are temporarily disabled. This adjustment ensures that only the necessary controls — those on the side menu 2703 and related to the robot’s point of view — are active and visible. This dynamic resizing and focus-centric display modulation allows users to transition seamlessly between general observation and focused control. The interface proves particularly valuable in scenarios where the robot moves out of the direct line of sight of the controller or when detailed remote inspection and intervention are required. By integrating brain -computer interface (BCI) technologies with AR, the system facilitates efficient remote operation while enhancing the situational awareness of the operator and control precision. This capability is crucial for maintaining operational continuity and safety, especially in complex or hazardous environments where direct visual monitoring of robotic agents is not feasible.

[0180] Perspective steady state visually evoked potentials (pSSVEPs)

[0181] The experimental setup exploited the advanced spatial mapping capabilities of the HoloLens 2 to enhance the user interface of the perspective SSVEP (pSSVEP) system. For the visual stimulus, aninitial test of six flickers with six frequency values: 7 Hz, 7.5 Hz, 8 Hz, 8.5 Hz, 9 Hz, and 11 Hz was utilised. The spatial mapping feature of HoloLens 2 facilitated a comprehensive scanning of the immediate environment of the user, creating an intricate three-dimensional occlusion map of real-world surfaces and structures. This capability was crucial for integrating virtual content seamlessly into the physical environment, allowing for an immersive augmented reality experience.

[0182] Fig. 28 illustrates one form of placement of SSVEP flickers. In the developed pSSVEP scenario, this environmental mapping enabled the precise placement of SSVEP flickers onto actual surfaces within the immediate surroundings. By leveraging the natural depth cues of these surfaces, the system was able to anchor the flickers firmly within the field of view. This not only helped maintain a consistent location for the flickers as the user moved but also enhanced the naturalistic interaction with the augmented elements. In one embodiment, as illustrated in Fig. 28, the SSVEP flickers e.g. 2802 are presented as the flashing white block on top of each red waypoint sphere e.g. 2803.

[0183] The HoloLens 2 employs a sophisticated ray cast cursor feature. The user is able to manipulate this to interact with the virtual environment. Through the intuitive movements of this raycast cursor, users can dynamically place the SSVEP flickers on any desired location on the mapped surfaces. This feature not only facilitates user-specific customisation of the flicker placement but also supports adaptive experimental conditions tailored to the interaction preferences of each user and spatial orientation.

[0184] Fig. 29 illustrates an example of the auto placement feature visualization.

[0185] Fig. 29 illustrates an image showcasing both the pSSVEP from the user’s POV and the auto placement feature visualisation, illustrating object recognition with camera raycast, SLAM occlusion map and computer view in bounding box frustum.

[0186] The incorporation of spatial mapping and raycast cursor features significantly advances user control over virtual elements within the augmented reality brain-computer interface system. An important enhancement to the system is the implementation of adaptive sizing for the six perspectives SSVEP flickers, which are displayed as white square boxes e.g. 2903. These flickers are programmed to maintain a constant visual size from the perspective of the user, irrespective of the distance from them. This is achieved by calculating the distance between the headset camera pose and all of the flickers, setting minimum and maximum size thresholds, which prevent the flickers from resizing to unreasonabledimensions. Consequently, the SSVEP flickers remain at an optimal viewing size on the AR display, ensuring that all flickers are perceived uniformly by the user. This uniformity is essential for the consistency of the EEG signals evoked by the flickers, as it maintains the ideal setup for signal elicitation and subsequent decoding. Moreover, the stability of the flickers, which remain stationary relative to the environment even as the user navigates through different locations, is crucial for consistent SSVEP signal elicitation, thus enhancing the overall efficacy and user experience of the system.

[0187] Auto Hands-free Placement via Object Recognition

[0188] In addition to user-directed placement of flickers, the positioning of the pSSVEP flickers can also be automated through computer vision technology. This process begins with the use of the RGB-D camera on the HoloLens to capture the live video feed as well as an occlusion map of the surrounding area. The computer vision algorithm then identifies objects within these images and generates 2D bounding boxes around recognised objects.

[0189] To accurately determine the 3D pose of an object from these 2D images, the system utilises the camera frustum — a geometric representation of the field of view of the camera — to project the 2D bounding boxes into the 3D space. This projection is further refined by incorporating data from the SLAM (Simultaneous Localisation and Mapping) sensors onboard the HoloLens, which provide a detailed occlusion map of the environment. This map identifies the 3D structure of the terrain surrounding the targeted object, thereby enhancing object positioning accuracy. Additionally, the system can calculate the horizontal distance between the user and the identified objects by employing a ray cast that extends from the user’s head-gaze direction to the collision point of the environment and the object. This measurement allows for the precise positioning of objects within the surroundings. By integrating these measurements, the system can determine an exact 3D pose for each object. Using this information, a pSSVEP flicker is then automatically placed at the location of the object.

[0190] Turning now to Fig. 29, the white geometries 2902 represent the frustum projections of the field of view for both the RGB-D camera on the HoloLens as well as the bounding box area extending from the camera pose. These projections trace the path from the camera to the nearest collision points as determined by the occlusion map, illustrating the spatial relationship between the camera, the detected object, and the environment.

[0191] In the top left comer of the, the HoloLens camera view in the virtual environment is shown 2903, targeting an apple 2904 on the ground, identified as the object of interest. Above the apple 2904 is the pSSVEP flicker 2906 with a dot on the ground indicating the flicker placement position. In the bottom right comer is the view of the bounding box 2907, which is a 3D representation of the same area that can also be seen in the frustum projection. This automated placement of flickers is important for enabling intuitive and efficient user interaction with the system, as it allows users to control the environment or navigate through tasks without manual setup, thereby improving the interaction experience and accessibility of the augmented reality system. This automated placement is important as it enhances usability and efficiency, particularly enabling hands-free operation, which is important in scenarios where users need to maintain their manual dexterity for other tasks.

[0192] AR System Flowchart

[0193] Turning now to Fig. 30, there is illustrates a system flowchart 3000 of the multi-user control system.

[0194] After initialisation 3001, the first step is user defined selection 3002, offering three operational modes: primary controller 3004, secondary controller 3003, and observer 3005. Each role is tailored to accommodate varying levels of control and observation within the system. Upon system startup during the initialisation and role selection phase, users are prompted to select their operational role.

[0195] The primary controller 3004 takes responsibility 3007 for setting up the local world anchors, which synchronise the scale and orientation of the world coordinate system across both robotic systems and other users, and for positioning the SSVEP flickers. Meanwhile, secondary controllers and observers establish 3008, 3009, local anchors within their immediate environment and await the completion of the primary controller’s setup.

[0196] In terms of operational roles and responsibilities, the primary controller manages the main robotic operations and configures the SSVEP flickers. If a second robot is connected, the secondary controller assumes control of it 3011 and participates in SSVEP trials to direct the robot’s actions. The observer monitors all activities within the virtual space 3012, keeping an eye on the status of SSVEP flickers and the positions of both virtual and real robots. Observers do not interact directly with the robotic controls until a role swap is initiated 3015.

[0197] Following the initial setup, the primary and secondary controllers begin the SSVEP trials to issue commands to the robots 3010. During these trials, the observer may request to swap roles with a controller. This role swapping can be initiated by either party, with each user selecting whom to swap with based on the other’s IP address. This mechanism allows for dynamic changes in control tailored to the system’s operational needs or user preferences.

[0198] In the command execution and feedback loop stage, if an SSVEP trial 3010 yields a valid decoded command, it is sent to the designated robot system for execution 3018. The system then enters a waiting state, monitoring the robot’s task completion 3020. If the robot successfully completes the command 3019, the system advances to the next SSVEP trial. If the task is not completed, the system remains in the waiting state, pending feedback from the robot.

[0199] SSVEP Classification Algorithm

[0200] The system classifies the SSVEP signal with a flexibility algorithm that combines a Complex spectrum feature-based convolutional neural network (CCNN) with an RL-based temporal filter. CCNN uses the complex spectrum feature extracted by the Fast Fourier Transform (FFT) as input and a CNN model as a decoder to get the target frequency. Firstly, raw data is filtered with a Butterworth band -pass filter ranging from 6Hz to 25Hz. Then, a FFT is used to calculate the input complex spectrum feature FCCNN, as per Equation 1 below. The real part and imaginary part of each frequency point are combined to build the final feature:

[0201] Next, the CNN model of the CCNN is used to classify the target frequency. Finally, the output is a one-hot vector that represents the n-th target. The benefit of CCNN in comparison with traditional CCA and TRCA algorithms is that the CCNN leverages advanced deep learning techniques, enabling it to handle complex and high-dimensional data with greater accuracy and robustness.

[0202] Unlike CCA, which is primarily linear, and TRCA, which focuses on maximising the reproducibility of task -related components, CCNN can automatically learn and optimise features directly from EEG data. This capability allows it to discern intricate patterns and relationships within the data than more straightforward linear or semi- linear methods may overlook. Additionally, the convolutional structure of CCNN is particularly advantageous in managing variations in signal characteristics, providing inherent robustness to shifts and distortions in the input space. This makes CCNN exceptionally suitable for EEG signal decoding, where electrode placement and individual differences can significantly impact the performance of traditional methods.

[0203] To enhance the data integrity utilised with CCNN, the embodiments use an adaptive temporal filter employing reinforcement learning system (RL). This methodology provides an adaptive temporal filter which is learned to selectively isolate high-quality data segments while discarding those of lower quality. Each SSVEP data trial is encapsulated as a two-dimensional matrix, delineating channels, and temporal samples. The RL agent commences its exploration from the initial temporal sample, examining a designated window of time samples (state s). Subsequently, the agent executes actions a filtering decision at each step to ascertain whether to retain or exclude segments. Upon completing the exploration within the training environment, the filtered segments of the data trial, alongside a reference signal subjected to the identical filter, are processed via the CCNN algorithm. The efficacy of the agent is assessed based on the accuracy derived from the CCNN, and a corresponding reward rais computed. The agent’s policy is refined through the Proximal Policy Optimisation (PPO) framework, as delineated in for example: J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, arXiv preprint arXiv: 1707.06347 (2017), leveraging the computed reward. Following multiple iterations of RL training, the refined model is preserved for operational deployment.

[0204] System Architecture

[0205] Real-Time EEG Signal Decoding For SSVEP

[0206] Equipped with HoloLens two headsets, each participant has a sensor pad with eight dry EEG electrodes attached at the occipital region to record brain activity effectively. These electrodes, linked to a MentaLab Explore 8 amplifier, transmit the EEG signals in real- time. The Lab Streaming Layer (LSL) protocol wirelessly streamed these data to an SSVEP classification script running on a Jetson Orin Nano single board computer, which was important in decoding the signals to enable direct, real-time interactionwith the AR interface. As part of the process, the streamed EEG signals underwent several pre-processing steps to ensure their usability for classification. Following pre-processing, the SSVEP classification script was designed to initiate the classification algorithm upon receiving a start event signal from the HoloLens. This signal acted as a trigger, prompting the script to read the EEG data precisely while the S SVEP flickers were active. By synchronising the reading of the EEG data with the flickering, the system could more accurately interpret the participant’s brain responses to the visual stimuli. The classification results were then reflected on the AR interface by directly displaying the results to inform and guide user interaction with the AR environment. This timing alignment ensured the classification was based on the most relevant brain activity, enhancing the system’s responsiveness and accuracy.

[0207] Communication Framework

[0208] Turning now to Fig. 31, there is illustrates one form of architecture for a multi-user robotic control system. The architecture integrates two user stations, each consisting of a user interface 3131, 3135, a single- board computer (SBC) 3132, 3136, and EEG system components 3133, 3137, with a central robotic system that both users connect to. Arrows in the figure indicate the direction of event handling and data flow, not merely the communication protocols. The communication framework of the system is structured into three primary components: BCI Components, the Robotic System, and the HoloLens Clients. Each plays a crucial role in ensuring seamless operation and interaction within the system.

[0209] BCI Components: The BCI setup includes dry EEG electrodes with eight channels connected to the MentaLab EEG signal amplifier and wireless transmitter. This setup uses Bluetooth to connect to a single-board computer that handles the data processing. The raw EEG data are transmitted via the Lab Streaming Layer (LSL) to the SSVEP classification script, which interprets the signals and translates them into commands for the robotic system.

[0210] Robotic System: In this configuration, the system controls a swarm of drones. The control terminal, which may be a separate PC or integrated into the single-board computer connected to the BCI components, uses a radio transceiver (nRF24LUl+ from Nordic Semiconductor) to communicate with the drones. This setup operates at data rates of 2Mbps, 1Mbps, and 250Kbps as available. Additionally, this computer establishes connections to the HoloLens units via a Wi-Fi access point. It uses TCP to link to the SSVEP classification script and UDP to facilitate real-time communications with the robots, facilitating rapid command transmission and execution.[0021 1] HoloLens Clients: The HoloLens users are integrated into the communication network, where the first user typically acts as the main host. Subsequent users connect to this host via UDP, allowing for peer-to-peer connections where any client can assume the host role if permissions are granted. The HoloLens devices are linked to the robotic system using UDP to ensure efficient, real-time command execution. Each user also connects to their respective SSVEP classification script via TCP, ensuring reliable data transmission for the processing of EEG signals and subsequent robot control.

[0212] Indoor and Outdoor System Deployment

[0213] Unlike traditional BCI systems, which are predominantly designed for indoor use due to their reliance on stationary equipment and controlled environments, the present embodiment allows for extensive portability and mobility. This enables effective deployment in a diverse range of settings, from structured indoor environments to dynamic outdoor scenarios. Such versatility underscores the system’s capability to adapt to and operate within varied operational contexts, expanding the potential applications of BCI technology beyond conventional boundaries.

[0214] In an example indoor setting, the system facilitated a cooperative task involving two operators: one controlling a UAV and the other a UGV. Initially, the UAV operator directed the vehicle to a predetermined location to assess potential hazards. Following this, the UGV operator manoeuvred towards the location of the UAV. This sequence not only demonstrates the system’s practical utility in surveying and mitigating risks but also enhances safety and operational efficiency by ensuring that ground units are deployed into vetted, secure environments.

[0215] In the outdoor setting example, a cooperative task involving two operators collaboratively controlling a UAV through a turn was conducted. Initially, the first operator controlled the UAV to various predetermined destinations until reaching a turn visible to both operators. At this point, control of the UAV was handed over to the second operator, who then continued to navigate the UAV towards their own position. This example sequence of control demonstrates the system’s capacity for seamless operator transition and coordination in real-time scenarios. This task is crucial as it tests the system’s ability to handle dynamic control switching under real-world conditions, ensuring that the UAV can be managed effectively by multiple operators from different locations. Such functionality is vital for complex operations where tasks may need to be handed off between various teams or in emergency situations whereimmediate response from the nearest available operator is necessary, thereby enhancing operational flexibility and responsiveness.

[0216] SSVEP Classification

[0217] Table 1: Accuracy of different algorithms on various subjects

[0218] Table 1 displays the classification accuracies under different settings. This research compares the proposed CCNN with RL filter with other 7 classification algorithms, namely CCNN, CCA, TRCA, ECCA, ETRCA, SSCOR and ESSCOR.

[0219] Relevant details on a number of the other algorithms include: ECCA -X. Chen, et al., Proceedings of the national academy of sciences 112, E6058 (2015); ETRCA - M. Nakanishi, et al., IEEE Transactions on Biomedical Engineering 65, 104 (2017); ESSCOR (G. K. Kumar, M. R. Reddy, IEEE transactions on neural systems and rehabilitation engineering 27, 2044 (2019).

[0220] From this table, it is evident that the CCNN achieves the highest accuracy compared to all other CCA- based algorithms. The 17% of the accuracy of ETRCA, SSCOR and ESSCOR means that they always predict a single class without understanding the true label. Moreover, with the aid of RL filtering, an enhancement in accuracy for certain subjects is observed. For instance, subject 2 initially exhibited low accuracy due to noisy samples, achieving only 33% accuracy without the temporal filter. However, after filtering out the noisy segments from the entire spectrum, the accuracy markedly increased to 64%. Such improvements render the robotic system more robust. This is important in real-time applications where data quality is influenced by numerous factors, including environmental instability and deviceuncertainties. The selective data filtering method effectively eliminates undesirable noise from the environment, thereby stabilising the system’s output.

[0221] DISCUSSION

[0222] The use of an AR headset for SSVEP stimuli is known in SSVEP-based BCI research. Beyond the use of AR headsets, the classification techniques for SSVEP signals are important in SSVEP-based BCI research.

[0223] The embodiments provide an innovative pSSVEP -based BCI system featuring a unique method for displaying flickers, which enables user interaction with a robotic system from various positions. The design of pSSVEP greatly enhances the practicality of BCI systems for real-world usage over traditional SSVEP methods. An advantage of this approach is its portability, allowing users to move freely within an environment while maintaining access to BCI controls through proximate SSVEP flickers. Unlike conventional SSVEP- based BCIs that require users to remain stationary, the system supports mobility.

[0224] Additionally, the EEG data necessary for system control are processed by a compact, portable, single-board computer, enhancing system mobility. The pSSVEP scenario also offers a more immersive experience, giving users the flexibility to position SSVEP flickers as needed, in contrast to traditional systems that fix flickers at a single spot on a display. However, placing flickers too close to each other can lead to interference, potentially lowering classification accuracy. While the spatial positioning of flickers relative to the user may influence pSSVEP accuracy, flickering duration is the primary determinant of classification accuracy.

[0225] With the help of computer vision, users can freely place the flicker to intuitively complete object selection. In this mode of operation, users can bypass the limitations of the SSVEP interface and focus more on the perception of real -world objects. Furthermore, the embodiments provide for multiperson synchronous control operations in AR-BCI. The embodiments provide for a multi-person remote control AR-SSVEP BCI system. Therefore, in many aspects, the proposed aBRI system is better suited for practical applications than the traditional BCI paradigms previously developed.

[0226] From the perspective of telerobotic control, the use of AR glasses, such as the HoloLens, significantly enhances the control and monitoring of remote robots through the creation of digital twins. These digital twins provide a three-dimensional representation of the robot, capturing its location,orientation, and elevation to be displayed in the field of view of the controller. This 3D visualisation is important for operators to precisely understand and manage the movements of robots, substantially improving operational accuracy compared to traditional 2D displays. Traditional systems often relay robot telemetry on flat screens, leading to a potential disconnect between the digital twin and the physical robot, which can result in misinterpretation of spatial data. The traditional flat metric -semantic maps used in robot telemetry can lead to misinterpretations in complex environments. The hierarchical and more integrated 3D scene representations enhances real-time interaction and spatial understanding, addressing the disconnect between digital twin representations and the actual robot operations.

[0227] Moreover, AR glasses allow for the incorporation of advanced functionalities that extend beyond basic visualisation. Equipped with on-board LiDAR and RGB-D cameras, these devices facilitate hands-free operations such as automatic placement of flickers through object recognition. This capability is important for setting dynamic interaction points in the user's field of view, enhancing the interface's intuitiveness. Additionally, the integration of 6DOF SLAM technology within the HoloLens enables precise tracking of both users and robots in the environment. This tracking supports multi-user collaboration by maintaining accurate positional awareness of each participant and their robotic counterparts, thereby enabling a cohesive and efficient operational workflow.

[0228] Overall, the AR glasses help transform the user interface for telerobotic systems by merging real- world perception with enhanced digital interaction, leading to more intuitive control, better information accuracy, and increased user engagement in complex robotic operations.

[0229] The conventional BCI systems of the prior art were normally validated under stationary conditions, thus the data quality is typically good. Consequently, the performance of the classification is high. However, when interacting with a telerobot system, or in a more natural form of cognitive task, the system's performance might drop due to a large amount of artifact contamination in the EEG data caused by significant physical movement.

[0230] We have validated several methods that have been reported in the literature (Table 1), and their performance was found to be not acceptable for the telerobot system. We have developed a method that addresses several key challenges in SSVEP classification.

[0231] The proposed RL-based temporal filtering technique leverages an RL agent to automatically detect and filter low-quality segments in SSVEP data, significantly improving the signal- to-noise ratio (SNR) and reducing the impact of artifacts and individual differences. This approach also acts as a quality classifier to differentiate between good and bad quality data, ensuring that only high-quality signals are fed into subsequent classification algorithms. By enhancing data quality and improving compatibility with CCNN algorithms, the RL-based temporal filtering method leads to substantial accuracy improvements, resulting in a remarkable improvement in classification accuracy.

[0232] The method of the embodiment stands out from other techniques due to its adaptability and explainability. The independent nature of the RL-based temporal filter allows it to be integrated with CCNN algorithms, enhancing their classification performance. Additionally, the quality classifier ensures that the RL-based filter only processes data when necessary, reducing computational overhead. The visualizable filtering process provides transparency and reliability, enabling domain experts to interpret and validate the results. The feasibility of this approach was demonstrated through a evaluation using a dry sensor-based dataset comprising 8 subjects, where the RL-based filtering method significantly improved accuracy compared with several CCA algorithms. The adaptability, robustness to noise, and reduction in data insufficiency issues underscore the effectiveness and practicality of this method for real- world SSVEP classification applications.

[0233] Further refinements are possible. One key feature is the ability for users to switch control of their individual robots with each other. This facilitates greater collaboration and flexibility, allowing users to exchange roles and responsibilities dynamically during tasks and enhancing teamwork and adaptability in various scenarios. Additionally, each user can have a customisable and auto-select personal profile for SSVEP decoding. This feature can automatically adjust the system’s decoding algorithms to each user’s unique brainwave patterns, improving the accuracy and responsiveness of command execution. By personalising the decoding process, the system can adapt more effectively to individual differences, ensuring smoother and more efficient interactions. These refinements can enhance the overall functionality and user experience of the system, making it more suitable for a wide range of applications.

[0234] A further refinement is an event-free design. By analysing the confidence of predictions from the CCNN algorithm, a dynamic window can be provided that determines whether the participant is focusing on a specific flicker. Such a design can make the robot system more flexible, eliminating the need for a fixed stimulus window size.

[0235] One limitation of the system pertains to the flickering duration of the SSVEP flickers. While initial testing demonstrated the ability to decode brain signals for classification in three seconds or less, the actual effectiveness is influenced by individual differences in reaction and focus times on the flickers. This factor is important because it directly impacts the system’s real-world usability and performance. The variability in how quickly and effectively users can concentrate on the flickers can determine the responsiveness of the system in practical applications. For instance, in scenarios requiring rapid decisionmaking, such as operating machinery or navigating through hazardous environments, any delay or inconsistency in focusing on the flickers could result in delayed or incorrect SSVEP classification. The underlying issue is that the brain may not produce the signals needed for command classification while the user focuses on the flicker. This can lead to a mismatch between the user’s intended command and the system’s response, potentially compromising safety, and operational efficiency. Therefore, to mitigate these issues in practical settings, the flicker duration has been set to a predetermined period, for example, longer than three seconds. In addition, the Al -powered SSVEP classification necessitates data collection and a pretraining model before online inference can occur. This process is time-consuming and introduces variance into the system.

[0236] Of course, additional advances in computational hardware (and software) can be constantly deployed to provide further improvements.

[0237] Example System

[0238] In one example study, eight healthy participants (seven males and one female, aged 20-40 years, mean ± SD: 24.26 ± 3.46; all right-handed) engaged in testing. Each participant possessed normal or corrected-to-normal vision and provided ethical consent. Participants were required to partake in two testing sessions using the Holo Lens, during which their EEG data was recorded using dry sensors. This data was subsequently utilised to train the algorithm for each individual. For dry sensors, the spring -loaded sensor (L.-D. Liao, et al., IEEE Transactions on Instrumentation and Measurement 63, 1545 (2014)) integrated with the MentaLab Explore 8 system (Mentalab GmbH, Munich, Germany) was used in the online scenario testing.

[0239] Turning now to Fig. 32, there is illustrates the basics of the Robotic System 350. In the system, each robotic platform, as illustrated in Fig. 32, was equipped with sensors to enhance its functionality and integration. The Leo Rover 351, was outfitted with the RPLidar A2 M12 SLAM sensor and onboardcameras. These features allow the Rover to autonomously navigate and map complex environments, providing detailed visual feedback into the Window -in-Window function in the AR interface. This makes the Leo Rover particularly effective in tasks requiring robust environmental interaction and high levels of autonomy. The DJI Tello Drone 352 is similarly equipped with an onboard camera, which plays a pivotal role in supporting remote control operations. The onboard camera, similar to the Leo Rover, integrates seamlessly with our window-in-window mode, allowing operators to view the point of view on the drone directly on their display. This functionality is important for performing teleoperation and monitoring from a distance, ensuring operators can make informed decisions in real-time. Additionally, the Crazyflie 2.1 mini drones e.g. 354 are equipped with Lighthouse sensors, incorporating four TS4231 infrared receivers located atop each drone. These sensors interact with HTC Vive base stations are important for operations that require high levels of spatial accuracy and synchronisation among multiple aerial units. The accuracy is important for the drones to accurately follow their digital counterparts in the augmented reality interface, resulting in precise swarm control. Such fidelity in positioning enhances the system’s effectiveness in teleoperation and remote monitoring, ensuring that operators can execute complex manoeuvres and coordinate activities with high precision and reliability.

[0240] Sensors Deployment: Traditionally, EEG-based systems have relied on wet sensors and 2D screens for data acquisition and SSVEP flicker display. While these setups offer high signal quality, they are cumbersome, uncomfortable for continuous use, and lack the mobility required for applications that involve controlling robots on the move, both indoors and outdoors. To address these limitations and enhance user experience and system portability, the example embodiment includes a hardware configuration as described below.

[0241] Fig. 33 and Fig. 34 presents a visual representation ofthe hardware components and their setup. Fig. 33 displays the individual components used in our system, including the HoloLens 2, the MentaLab Explorer 8 EEG signal amplifier, custom-made dry EEG electrodes, and a ground electrode. Fig. 34 illustrates how these components are assembled on a fake human head made of Styrofoam, demonstrating the practical and compact configuration for mobile use.

[0242] The EEG sensor pad is strategically positioned between the back of the HoloLens 2 head-strap and the occipital region of the head, optimising the capture of EEG signals from this brain area known for its prominence in visual processing tasks. Additionally, a ground electrode is affixed at the back of the right ear, serving as an optimal ground reference point for EEG measurements. The MentaLab Explorer 8is directly connected to the sensor pad’s wiring, facilitating the reliable transmission of EEG data. The MentaLab Explorer 8 is a suitable component in this setup, offering a sampling rate adjustable from 250 to 1000 Hz and an input range of ±400 mV. It also features a Bluetooth connection with up to 10 meters range and provides up to 8 hours of battery life, making it suitable for prolonged use in both indoor and outdoor environments. The HoloLens 2 enhances this configuration with its LiDAR and RGB-D cameras, supporting six degrees of freedom and a high-quality display that is effective under various lighting conditions. This AR device is capable of running complex computational tasks, including multi-user functionalities and precise object recognition, and it can stream video from different robotic units while accurately displaying various SSVEP frequencies for effective brain -computer interface applications.

[0243] Classification Algorithms

[0244] The CCNN and Reinforcement Learning filter: To integrate the Complex Spectrum Featurebased Convolutional Neural Network (CCNN) and Reinforcement Learning (RL) into the robotic system, the Fast Fourier Transform (FFT) resolution was adjusted to about 0.25Hz following extensive testing. The resulting complex spectrum feature matrix, Nch x Nfc, where Nch represents the number of channels (8 channels) and N / cdenotes the number of features per channel (164 features), forms the input to the CCNN. The example architecture of the CCNN is provided in Table 2, illustrating each layer’s configuration from input through to output. For the training process of the CCNN, this work uses the Keras toolkit, selects the Adam optimiser, sets the learning rate to KT4, and trains for 250 epochs.

[0245] Table 2: The Detailed Structure of the CCNN Model.

[0246] For the RL filter, the slice window was set to 50 time samples, corresponding to 0.1 seconds under a 500 Hz sampling rate. Consequently, the input to the RL agent’s EEGNet -based model is an 850 matrix, reflecting data from the 8 EEG channels over the specified segment duration.

[0247] The experimental setup and the configuration of the EEGNet were as detailed in Table 3. Adjustments to the kernel size and activation functions were made to optimise EEGNet for our wearable dataset. Additionally, we meticulously defined the Proximal Policy Optimisation (PPO) training parameters: a total of 10,000 learning steps with updates every 128 steps. These specifications were deliberately chosen to ensure a streamlined and effective learning process for the RL agent. This section concludes with a discussion of the experimental results, providing insights into the performance enhancements achieved through our integrated approach.

[0248] Table 3: Network Architecture Summary and Learning Settings of PPO.

[0249] Interpretation

[0250] The figures included herewith show aspects of non-limiting representative embodiments in accordance with the present disclosure, and particular structural elements shown in the figures may not be shown to scale or precisely to scale relative to each other. The depiction of a given element or consideration or use of a particular element number in a particular figure or a reference thereto in corresponding descriptive material can encompass the same, an equivalent, an analogous, categorically analogous, or similar element or element number identified in another figure or descriptive material associated therewith. The presence ofin a figure or text herein is understood to mean "and / or" unless otherwise indicated, i.e., “A / B” is understood to mean “A” or “B” or “A and B”. The recitation of a particular numerical value or value range herein is understood to include or be a recitation of anapproximate numerical value or value range, for instance, within + / - 20%, + / - 15%, + / - 10%, + / - 5%, + / - 2.5%, + / - 2%, + / - 1%, + / - 0.5%, or + / - 0%. The term "essentially all" or "substantially" can indicate a percentage greater than or equal to 50%, 60%, 70%, 80%, or 90%, for instance, 92.5%, 95%, 97.5%, 99%, or 100%.

[0251] Many modifications will be apparent to those skilled in the art without departing from the scope of the present invention.

[0252] The reference in this specification to any prior publication (or information derived from the prior publication), or to any matter which is known, is not, and should not be taken as an acknowledgment or admission or any form of suggestion that the prior publication (or information derived from the prior publication) or known matter forms part of the common general knowledge in the field of endeavour to which this specification relates.

[0253] Throughout this specification and the claims which follow, unless the context requires otherwise, the word "comprise", and variations such as "comprises" or "comprising", will be understood to imply the inclusion of a stated integer or step or group of integers or steps but not the exclusion of any other integer or step or group of integers or steps.

[0254] Unless the context clearly requires otherwise, throughout the description and the claims, the words, “comprise”, “comprising”, and the like are to be construed in an inclusive sense as opposed to an exclusive or exhaustive sense; that is to say, in the sense of “including, but not limited to”.

[0255] Reference throughout this specification to “one embodiment”, “some embodiments” or “an embodiment” means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, appearances of the phrases “in one embodiment”, “in some embodiments” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment, but may. Furthermore, the particular features, structures or characteristics may be combined in any suitable manner, as would be apparent to one of ordinary skill in the art from this disclosure, in one or more embodiments.

[0256] As used herein, unless otherwise specified the use of the ordinal adjectives "first", "second", "third", etc., to describe a common object, merely indicate that different instances of like objects are beingreferred to, and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking, or in any other manner.

[0257] In the claims below and the description herein, any one of the terms comprising, comprised of or which comprises is an open term that means including at least the elements / features that follow, but not excluding others. Thus, the term comprising, when used in the claims, should not be interpreted as being limitative to the means or elements or steps listed thereafter. For example, the scope of the expression a device comprising A and B should not be limited to devices consisting only of elements A and B. Any one of the terms including or which includes or that includes as used herein is also an open term that also means including at least the elements / features that follow the term, but not excluding others. Thus, including is synonymous with and means comprising.

[0258] As used herein, the term “exemplary” is used in the sense of providing examples, as opposed to indicating quality. That is, an “exemplary embodiment” is an embodiment provided as an example, as opposed to necessarily being an embodiment of exemplary quality.

[0259] It should be appreciated that in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of this invention.

[0260] Furthermore, while some embodiments described herein include some but not other features included in other embodiments, combinations of features of different embodiments are meant to be within the scope of the invention, and form different embodiments, as would be understood by those skilled in the art. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0261] Furthermore, some of the embodiments are described herein as a method or combination of elements of a method that can be implemented by a processor of a computer system or by other means of carrying out the function. Thus, a processor with the necessary instructions for carrying out such a method or element of a method forms a means for carrying out the method or element of a method. Furthermore, an element described herein of an apparatus embodiment is an example of a means for carrying out the function performed by the element for the purpose of carrying out the invention.

[0262] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the invention may be practiced without these specific details. In other instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

[0263] Similarly, it is to be noticed that the term coupled, when used in the claims, should not be interpreted as being limited to direct connections only. The terms "coupled" and "connected," along with their derivatives, may be used. It should be understood that these terms are not intended as synonyms for each other. Thus, the scope of the expression a device A coupled to a device B should not be limited to devices or systems wherein an output of device A is directly connected to an input of device B. It means that there exists a path between an output of A and an input of B which may be a path including other devices or means. "Coupled" may mean that two or more elements are either in direct physical or electrical contact, or that two or more elements are not in direct contact with each other but yet still co-operate or interact with each other.

[0264] Thus, while there has been described what are believed to be the preferred embodiments of the invention, those skilled in the art will recognize that other and further modifications may be made thereto without departing from the spirit of the invention, and it is intended to claim all such changes and modifications as falling within the scope of the invention. For example, any formulas given above are merely representative of procedures that may be used. Functionality may be added or deleted from the block diagrams and operations may be interchanged among functional blocks. Steps may be added or deleted to methods described within the scope of the present invention.

Claims

CLAIMS:

1. A computer brain interface system including: a brain monitoring system for monitoring a user’s brain for the detection of a displayed steadystate visually evoked potential structure (SSVEP) by the user from the users visual input; a visual display device for the display of visual information to a user; including a potential command action associated with the SSVEP pattern; and a command execution device for performing the corresponding command action upon detection of the SSVEP pattern by the brain monitoring system.

2. A computer brain interface system as claimed in claim 1 wherein the SSVEP is displayed in a perspective manner to the user and said visual display device includes augmented reality glasses.

3. A computer brain interface system as claimed in claim 1 wherein the SSVEP includes a temporally changing grid structure.

4. A computer brain interface system as claimed in claim 1 wherein said command action includes activities associated with the control of movable autonomous device such as aerial or land based drones.

5. A computer brain interface system as claimed in any previous claims where multiple users are provided with corresponding brain monitoring systems and visual display devices with corresponding command actions for the multi user control of an environment.

6. A computer brain interface system as claimed in any previous claim wherein said display device simultaneously displays multiple separate perspectives of an environment.

7. A method of displaying a plurality of flicker regions in an extended reality display, each of the plurality of flicker regions being configured to provide a steady state visually evoked potential for a brain computer interface, the method comprising: receiving a three dimensional map for at least a portion of an environment visible in the extended reality display, the three dimensional map including position information about two or more objects in the scene; selecting a location for each of the plurality of flicker regions in the scene, each of the locations being associated with one of the two or more objects;determining appearance information for each of the plurality of flicker regions at the selected location, the appearance information including a shape of the flicker region based on a perspective associated with the associated object of the two or more objects; and displaying the plurality of flicker regions at the location with the determined appearance, each of the plurality of flicker regions having a different display frequency to allow the brain -computer interface to determine which of the plurality of flicker regions is being viewed by a user of the brain-computer interface.

8. The method according to claim 7, wherein the appearance information of each of the plurality of flicker regions includes a size.

9. The method according to either of claim 7 or 8, wherein the appearance information of each of the plurality of flicker regions includes a colour.

10. The method according to claim 8, wherein the size of flicker region is determined based on a size of the associated object.

11. The method according to claim 9, wherein the colour of the flicker region is selected based on a colour of the associated object.

12. The method according to claim 7, wherein the size of each of the plurality of flicker regions mimics depth cues, at the location, from the three dimensional map.

13. The method according to claim 7, wherein determining the appearance information of at least one of the plurality of flicker regions includes determining the shape of the flicker region so the flicker region appears to be positioned on a real world surface in the scene.

14. The method according to claim 7, wherein at least one of the plurality of flicker regions rotates based on a location of the user.

15. The method according to claim 7, further comprising: determining that the user of the brain-computer interface has selected one of the plurality of flicker regions by focusing on the selected flicker region; and applying an action by the brain-computer interface associated with the selected flicker region.

16. A method of displaying a plurality of flicker regions, the method comprising:positioning the plurality of flicker regions in an extended reality environment visible to a user, at least two of the plurality of flicker regions being located at different depths from the user; determining appearance information for each of the plurality of flicker regions based on a location of the user relative to each of the plurality of flicker regions; displaying each of the flicker regions in the augmented reality environment; and determining a selection of one of the plurality of flicker regions using steady state visually evoked potentials detected by a sensor pad on the user.

17. The method according to claim 16 wherein the appearance information is a shape of each of the flicker regions.

18. The method according to claim 16, wherein positioning the plurality of flicker regions includes positioning each of the plurality of flicker regions on a linked object in the augmented reality environment.

19. The method according to claim 18, wherein the appearance information is further determined according to a shape of the linked object.

20. The method according to any one of claims 16 to 19, wherein the appearance information of each of the plurality of flicker regions includes a size.

21. The method according to claim 20, wherein the size of flicker region is determined based on a size of the associated object.

22. The method according to any one of claims 16 to 21, wherein the appearance information of each of the plurality of flicker regions includes a colour.

23. The method according to any one of claims 16 to 22, wherein the shape of each of the plurality of flicker regions mimics depth cues, at the location, from the three dimensional map.

24. The method according to any one of claims 16 to 23, wherein determining the appearance information of at least one of the plurality of flicker regions includes determining the shape of the flicker region so the flicker region appears to be positioned on a real world surface in the scene.

25. The method according to any one of claims 16 to 24, further comprising: applying an action by the brain-computer interface associated with the selected flicker region.

26. A system for display a plurality of flicker regions, the system comprising a processing system configured to: position the plurality of flicker regions in an extended reality environment visible to a user, at least two of the plurality of flicker regions being located at different depths from the user; determine appearance information for each of the plurality of flicker regions based on a location of the user relative to each of the plurality of flicker regions; display each of the flicker regions in the augmented reality environment; and determine a selection of one of the plurality of flicker regions using steady state; and visually evoked potentials detected by a sensor pad on the user.

27. A method of controlling multiple spatially movable robot devices via mulitple users, the method including: providing each user with a virtual or augmented reality visual device and brain seaming device for monitoring the SSVEP responses of each user to a corresponding virtual or augmented reality; displaying a series of flicker controls for controlling the robot devices in the simultated virtual or augmented reality; and decoding the SSVEP responses of the users to the flicker controls activate actions by the robot devices; and providing a control system so that multiple users can simultaneously control multiple robot devices to accomplish a task.

28. A method as claimed in claim 27 which includes displaying to a user a visual perspective of a robot device.

29. A method as claimed in claim 28, wherein the visual perspective is displayed in a sub window of the virtual or augmented reality experienced by the user.

30. A method as claimed in claim 27 wherein the multiple users operate in a master / slave relationship with one user maintaining overall control of the robots.

31. A system for allowing multiple users to control multiple movable robot devices to accomplish a task, the system including: for each user, a virtual or augmented reality display unit for wearing by the user and a corresponding brain monitoring system for monitoring and decoding the SSVEP responses to the visual information displayed to the user; a series of spatially movable robot devices; anda virtual or augmented reality environment, including flicker control activations for the mulitple users, the flicker control activations having corresponding robot control activations, upon detection of activation by the users by the brain monitoring system.

32. A system as claimed in claim 31 wherein said movable robot devices include visual environment monitoring devices, with the visual feed being displayed to the user via the augmented reality display.

33. A system as claimed in claim 32, wherein the visual feed is displayed in a sub window of the virtual or augmented reality experienced by the user.

Images