A brain-computer interaction paradigm, implementation method, brain-computer interaction method and system

Abstract

The present invention relates to the field of brain-computer interface technology, and discloses a brain-computer interaction paradigm, implementation method, brain-computer interaction method and system. The brain-computer interaction paradigm presents an electroencephalogram (EEG) stimulation interface to the user's stimulation attention eye, and simultaneously presents an environmental video to the user's video attention eye; the EEG stimulation interface and the environmental video are switched in distance based on the user's active and passive attention selection mechanism. The brain-computer interaction method has an observation mode and a control mode, and switches between the observation mode and the control mode based on the user's eye movements. In the observation mode, the first-person perspective video and the third-person perspective video are synchronously tiled from left to right on the main canvas; in the control mode, the user's EEG features are acquired in real time. When the EEG feature is judged to be a control-state EEG feature, the corresponding EEG stimulation category is determined and the control instruction corresponding to the EEG stimulation category is output. The present invention can limit user misoperation to the greatest extent and realize high-performance immersive brain-computer interaction in complex scenarios.

Description

Technical Field

[0001] The present invention relates to the field of brain-computer interface technology, and in particular to a brain-computer interaction paradigm, implementation method, brain-computer interaction method and system. Background Art

[0002] Brain-computer interface (BCI) technology is a technology that establishes a direct communication pathway between the human brain and external devices without relying on conventional brain information output pathways such as peripheral nerves and muscle tissue. Currently, BCI systems, especially non-invasive BCI systems that are less invasive, less risky, and easier to operate, can achieve intelligent control of a variety of external devices such as wheelchairs, robotic arms, drones, and unmanned vehicles. With the expansion of application scenarios of brain-computer interaction systems and the increase in the control dimensions of controlled objects, the requirements for the flexibility and convenience of BCI systems are becoming increasingly higher. Traditional BCI systems have the following disadvantages during the interaction process: (1) In terms of display, the system needs to use a display screen to present the stimulation interface, and the interactive device is less portable; (2) In terms of disturbance, users are easily disturbed by the surrounding environmental information during use, and the interactive immersion is low, especially in noisy environments, which can easily affect user decision-making; (3) In terms of feedback, the traditional single interface can only present limited environmental information. During the brain-computer interaction control process, the utilization rate of the scene information sent back to the controlled object is low; (4) In terms of comfort, the user's field of view under the traditional display screen is limited to the screen area. In order to ensure a good EEG induction effect of visual stimulation, the user needs to maintain a fixed posture for a long time, which not only increases the user's fatigue, but also increases the user's mental workload.

[0003] Extended Reality (XR) refers to the use of computers to combine the real world with the virtual world to build a virtual environment for human-computer interaction. It is an immersive interactive technology that can create an immersive experience for the experiencer. XR technology includes Virtual Reality (VR), Augmented Reality (AR) and Mixed Reality (MR) technology, which can be applied to various fields such as entertainment, education, medicine and military. With the continuous iteration and update of XR technology, more and more research teams are combining BCI technology with XR technology for rehabilitation training. However, nowadays, as the XR-BCI system gradually moves from simulation training in virtual scenes to immersive control in actual scenes, the real environment information that users need to pay attention to is increasing, the complexity of user control tasks is constantly increasing, and the requirements for the flexibility and accuracy of the interactive system are getting higher and higher. The existing XR-BCI system has become increasingly difficult to meet the complex needs of users. Take the brain-controlled drone system based on steady-state visually evoked potential (SSVEP) in XR mode as an example. Figure 1 As shown in the figure, in terms of environmental information presentation, existing XR-BCI systems can only control drones based on single-perspective information. During the interaction process, users can only perform first-person perspective control based on the video sent back by the drone's front-facing camera, or perform third-person perspective control based on the drone's real flight environment captured by other auxiliary equipment. It is impossible to simultaneously or switch between presenting multi-perspective environmental information that facilitates users to observe and judge the status of the controlled object. As portable XR-BCI systems gradually move from indoors to outdoors, the complexity of tasks that users need to complete is also gradually increasing. Therefore, as an important basis for decision-making and judgment during the user's immersive interaction process, how to provide users with more multi-dimensional reference environmental information in XR mode has become an important issue that needs to be solved urgently in today's XR-BCI systems.

[0004] Most importantly, in existing XR-BCI systems, the interface presentation mode including visual induced stimulation and returned video stream is mostly in surround mode or overlay mode. In surround mode, Figure 2 As shown in (a), the returned video is located in the center of the field of view, and the flickering stimulus is spread all around the video stream. During the interaction process, the user must first focus on the central environmental information, and then shift their gaze to look at the corresponding stimulus to output the control command after making a control decision. In this mode, the user's control decision output has a certain lag, and the user cannot pay attention to the real-time environmental information in time when looking at the stimulus. When the environmental information changes rapidly, it is easy to cause danger. In the overlay mode, as shown in the attached figure, Figure 2As shown in (b), in the case of near-eye immersive display in XR mode, the stimulation will cover a large area of ​​the returned video frame, which will cause the user to have a certain degree of incomprehension of the environment, and also lead to user misjudgment, increasing the risk of peripheral manipulation. Taking the first-person perspective control of a brain-controlled drone as an example, if the stimulation coverage area happens to be the obstacle in front, and the user fails to make a timely decision to change the route to avoid it due to information omission, it will cause the drone to crash, which will pose certain risks to the equipment, environment, and personnel. Therefore, whether it is possible to develop a brain-computer interaction visual stimulation paradigm in XR mode, using the binocular independent rendering mode of the immersive display device, that allows users to take into account both stimulation and environmental information at the same time, without occlusion or overlap, has become another important issue that needs to be solved in the XR-BCI system.

[0005] At the same time, in the traditional XR-BCI interaction process, users are unable to focus on areas of interest in environmental feedback information, and the utilization rate of scene information is low. If users can selectively highlight and amplify key environmental information during the interaction process, it can greatly reduce misjudgments and improve the accuracy of brain-computer decision output. In addition, most existing XR-BCIs are brain-computer synchronous interaction modes. On the one hand, brain-computer interaction stimuli are all synchronized with the video, and on the other hand, brain-computer interaction commands are all synchronized with the stimulation output. In this mode, on the one hand, the continuous appearance of stimulation will cause redundant interference to the user and affect the user's decision-making. On the other hand, when the user has no control intention and is not looking at the visual stimulation, brain control commands will also be randomly output. This operation will bring greater control risks, and the randomly output erroneous commands will affect the normal operation process of the controlled device, reducing the efficiency of brain-computer interaction. Today, XR display devices are constantly optimizing and iterating in terms of posture sensing, visual sensing, and spatial interaction. Therefore, whether it is possible to combine the perception function of XR with the EEG decoding technology of BCI to obtain the information that the user focuses on, and based on the user's attention selection, build an asynchronous brain-computer interaction system with better flexibility, higher utilization of visual information, and fewer erroneous control decision outputs to achieve selective output of brain control commands has become an important issue that needs to be solved urgently. Summary of the Invention

[0006] The purpose of the present invention is to provide a brain-computer interaction paradigm, implementation method, brain-computer interaction method and system. The brain-computer interaction paradigm is based on an XR display device, and multi-classification EEG stimulation and real-time feedback environmental video are independently presented to the user's monocular field of view, realizing separate independent display of stimulation and video in XR mode, and building a stimulation, video penetrating, unobstructed and uncovered visual experience for the user based on the user's binocular competition mechanism and active and passive attention selection mechanism. The brain-computer interaction method combines the multimodal perception function of the XR display device with the asynchronous interaction technology of BCI, so that the user can freely select the required key information and freely output interaction instructions according to the user's control requirements, solving the problem that the existing XR-BCI system control instructions cannot be selectively output. The present invention solves the problems of the current XR-BCI system's inability to simultaneously use multi-dimensional perspectives to present the environmental information of the controlled object, the difficulty in highlighting the user's area of ​​interest, and the redundant error output of control decisions, thereby limiting user misoperation to the greatest extent and realizing high-performance immersive brain-computer interaction in complex scenarios.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] In the first aspect, the present invention provides a brain-computer interaction paradigm, which presents an EEG stimulation interface to the user's stimulation attention eye, and simultaneously presents an environmental video to the user's video attention eye; the stimulation attention eye is the eye whose monocular EEG stimulation classification accuracy is closest to the binocular EEG stimulation classification accuracy, and the other eye is the video attention eye; or, when the left eye EEG stimulation classification accuracy is equal to the right eye EEG stimulation classification accuracy, the stimulation attention eye is the user's physiological dominant eye, and the other eye is the video attention eye; the EEG stimulation interface and the environmental video are switched between near and far based on the user's active and passive attention selection mechanism.

[0009] In a second aspect, the present invention provides a method for implementing a brain-computer interaction paradigm, for implementing the brain-computer interaction paradigm provided in the first aspect, the method for implementing the brain-computer interaction paradigm comprising:

[0010] S10. Constructing rendering cameras and display canvases. The rendering cameras include a left camera, a right camera, and a main camera. The display canvases include a left canvas, a right canvas, and a main canvas. The left canvas is mounted on the left camera and is displayed independently only to the user's left eye. The right canvas is mounted on the right camera and is displayed independently only to the user's right eye. The main canvas is mounted on the main camera and is displayed synchronously to both eyes of the user.

[0011] S11. Write an EEG stimulation interface based on the control instructions required by the controlled object, and render the EEG stimulation interface on the left canvas, the right canvas, and the main canvas;

[0012] S12. Perform an offline EEG classification experiment. Based on the EEG induced by the EEG stimulation interface, determine the eye with the highest EEG classification accuracy among the user's left and right eyes. Mark it as the stimulation attention eye, and mark the other eye as the video attention eye.

[0013] S13. Presenting the EEG stimulation interface to the stimulation eye and presenting the environmental video to the video eye;

[0014] S14. When the user actively focuses on the electroencephalographic stimulation interface, the electroencephalographic stimulation interface approaches the eyes in a direction perpendicular to the plane where the eyes are located, and the environmental video moves away from the eyes in a direction perpendicular to the plane where the eyes are located; when the user actively focuses on the environmental video, the environmental video moves toward the eyes in a direction perpendicular to the plane where the eyes are located, and the electroencephalographic stimulation interface moves away from the eyes in a direction perpendicular to the plane where the eyes are located.

[0015] As a possible implementation, S12 includes:

[0016] S120. When the EEG stimulation interface is only displayed on the left canvas, when the EEG stimulation interface is only displayed on the right canvas, and when the EEG stimulation interface is only displayed on the main canvas, perform an EEG classification offline experiment to obtain the user's left eye EEG classification accuracy, right eye EEG classification accuracy, and binocular EEG classification accuracy respectively;

[0017] S121. Calculate the absolute value of the first difference between the left eye EEG classification accuracy and the binocular EEG classification accuracy, and calculate the absolute value of the second difference between the right eye EEG classification accuracy and the binocular EEG classification accuracy; mark the single eye corresponding to the smaller value of the absolute value of the first difference and the absolute value of the second difference as the stimulus attention eye, and mark the other eye as the video attention eye; if the absolute value of the first difference is equal to the absolute value of the second difference, then mark the user's physiological dominant eye as the stimulus attention eye, and the other eye as the video attention eye.

[0018] As a possible implementation, S13 includes:

[0019] S130. Unify the spatial coordinate systems of the left and right cameras, and set the left and right cameras to move with the user's head posture;

[0020] S131. Mount the left canvas at the origin of the spatial coordinate system where the left camera is located, and mount the right canvas at the origin of the spatial coordinate system where the right camera is located;

[0021] S132. If the left eye is the eye being stimulated and focused, the left camera presents the EEG stimulation interface and the right camera presents the environmental video; if the right eye is the eye being stimulated and focused, the right camera presents the EEG stimulation interface and the left camera presents the environmental video.

[0022] In a third aspect, the present invention provides a brain-computer interaction method. Before online interaction, based on the brain-computer interaction paradigm provided in the first aspect, an offline EEG experiment is performed to obtain n types of control-state EEG features induced when a user gazes at n types of EEG stimulation in an EEG stimulation interface and idle-state EEG features induced when the user gazes at an environmental video. The n types of control-state EEG features corresponding to the n types of EEG stimulation and the idle-state EEG features corresponding to the environmental video are saved as a user EEG classification template.

[0023] During online interaction, the brain-computer interaction method has an observation mode and a control mode. The switching between the observation mode and the control mode is based on the user's eye movements. The brain-computer interaction method includes:

[0024] When switching to observation mode, perform the following operations:

[0025] Obtaining first-person perspective video and third-person perspective video of the controlled object, and synchronously tiledly presenting the first-person perspective video and the third-person perspective video from left to right on the main canvas based on the brain-computer interaction paradigm implementation method provided in the second aspect;

[0026] When switching to the control mode, the brain-computer interaction method includes the following sub-steps:

[0027] S20. Determine the control video;

[0028] S21. Based on the brain-computer interaction paradigm provided in the first aspect, the EEG stimulation interface and the control video are presented to the user's stimulation attention eye and video attention eye respectively;

[0029] S22. Obtain the user's active attention result in real time, and determine whether the user's active attention result is an EEG stimulation interface or a control video. If it is an EEG stimulation interface, execute S23 to S24; if it is a control video, execute S25;

[0030] S23. Move the EEG stimulation interface closer to the eyes along a direction perpendicular to the plane where the eyes are located, and move the control video away from the eyes along a direction perpendicular to the plane where the eyes are located;

[0031] S24. Real-time acquisition of the user's EEG characteristics, matching them with the user's EEG classification template, and determining whether the EEG characteristics belong to the n-type control state EEG characteristics or the idle state EEG characteristics; if they belong to the n-type control state EEG characteristics, then determining the EEG stimulation category corresponding to the EEG characteristics and outputting the control instructions corresponding to the EEG stimulation category; if they belong to the idle state EEG characteristics, then no control instructions are output;

[0032] S25. Move the control video closer to the eyes in a direction perpendicular to the plane where the eyes are located, and move the electroencephalographic stimulation interface away from the eyes in a direction perpendicular to the plane where the eyes are located.

[0033] As a possible implementation method, obtaining the first-person perspective video and the third-person perspective video of the controlled object, and synchronously presenting the first-person perspective video and the third-person perspective video from left to right on the main canvas includes:

[0034] S30. Obtaining first-person perspective video and third-person perspective video of the controlled object;

[0035] S31. Set the main camera not to follow the user's head posture and set the horizontal pixel range of the main canvas equal to the sum of the horizontal pixel ranges of the first-person perspective video and the third-person perspective video, and the vertical pixel range equal to the larger range of the vertical pixel ranges of the first-person perspective video and the third-person perspective video;

[0036] S32. The first-person perspective video and the third-person perspective video are synchronously tiled and presented on the main canvas from left to right, and the horizontal pixel ranges of the first-person perspective video and the third-person perspective video do not overlap.

[0037] As a possible implementation method, determining the control video includes:

[0038] S200. Keep the user's line of sight perpendicular to the plane where the eyes are located, obtain the range of the yaw angle of the user's head posture when the user deflects from the leftmost end of the main canvas to the rightmost end of the main canvas, and establish a field of view mapping relationship between the horizontal pixel coordinates of the user's direct gaze point and the yaw angle of the head posture;

[0039] S201. Determine the switching moment from observation mode to control mode based on the field of view mapping relationship, and the horizontal pixel coordinate value of the user's direct attention point corresponding to the yaw angle of the user's head posture. If the horizontal pixel coordinate value belongs to the horizontal pixel range of the first-person perspective video, then determine that the control video is a first-person perspective video; if the horizontal pixel coordinate value belongs to the horizontal pixel range of the third-person perspective video, then determine that the control video is a third-person perspective video.

[0040] As a possible implementation method, the user's active attention result is obtained in real time, and it is judged whether the user's active attention result is a brain electrical stimulation interface or a control video. Specifically, the first horizontal force direction of the muscles around the eye of the user's stimulating attention eye and the second horizontal force direction of the muscles around the eye of the video attention eye are obtained in real time. When the second horizontal force direction is toward the stimulating attention eye and the first horizontal force direction is the same as the second horizontal force direction, it is judged that the user's active attention result is a brain electrical stimulation interface. Otherwise, it is judged that the user's active attention result is a control video.

[0041] As a possible implementation manner, the user's eye movement is blinking a preset number of times continuously within a preset time.

[0042] In a fourth aspect, the present invention provides a brain-computer interaction system for executing the brain-computer interaction method provided in the third aspect, the brain-computer interaction system comprising:

[0043] The paradigm presentation module runs the brain-computer interaction paradigm and presents the EEG stimulation interface and control video to the user's stimulation attention eye and video attention eye respectively;

[0044] EEG acquisition module, collects the user's EEG signals in real time;

[0045] The signal processing module performs preprocessing, feature extraction and template matching on the collected EEG signals, and generates corresponding control instructions based on the extracted EEG features;

[0046] The attention selection module obtains the user's active attention results in real time and determines whether a control instruction needs to be output. If a control instruction needs to be output, the control instruction generated by the signal processing module is transmitted to the instruction sending module;

[0047] The instruction sending module sends the control instruction to the controlled object, so that the controlled object completes the action corresponding to the control instruction;

[0048] The environmental feedback module is used to display the environmental information of the controlled object to the user in real time in the form of video stream feedback;

[0049] and a mode switching module that switches between control mode and observation mode based on the user's eye movements.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] 1. The brain-computer interaction paradigm proposed in this invention presents multi-classification EEG stimulation and real-time feedback of environmental videos independently to the user's monocular field of view, realizing the separate and independent display of stimulation and video in XR mode. Based on the user's binocular rivalry mechanism and active and passive attention selection mechanism, it constructs a stimulating and video-penetrating visual experience for the user with no obstruction or coverage.

[0052] 2. The brain-computer interaction paradigm proposed in the present invention can realize the free near and far switching of the EEG stimulation interface and the environmental video based on the user's active and passive attention selection mechanism, solving the problems in the current XR-BCI system where the stimulation and environmental information block and overlap each other, the user cannot take both into account at the same time, the scene information utilization rate is low, and the visual stimulation is not obvious. It minimizes the problems of misjudgment and decision delay caused by information omission and observation lag during user interaction, improves the efficiency of brain-computer interaction in XR mode, and realizes portable immersive brain-computer interaction.

[0053] 3. The brain-computer interaction method proposed in this invention saves both the control-state EEG features and the idle-state EEG features into the user's EEG classification template before interaction, enabling asynchronous output of brain-control commands in the presence of environmental interference. Furthermore, the brain-computer interaction method proposed in this invention combines the multimodal perception capabilities of XR display devices with the asynchronous interaction technology of BCI, addressing issues such as the inability of current XR-BCI systems to simultaneously present environmental information about the controlled object from multiple perspectives, difficulty highlighting user areas of interest, and redundant error outputs in control decisions. This minimizes user misoperation and enables high-performance immersive brain-computer interaction in complex scenarios.

[0054] 4. The brain-computer interaction method proposed in this invention fully utilizes the gesture, eye movement, and image sensors of XR devices. It features observation and control modes, both of which can be switched freely. This creates a more flexible, engaging, and flexible brain-computer interaction experience for users, reduces user fatigue, lowers the workload of brain control tasks, and improves the accuracy of brain-computer control decision output. Furthermore, in observation mode, both first-person and third-person perspectives of the controlled object can be presented simultaneously, addressing the single perspective and limited field of view of users in traditional interaction paradigms. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0056] Figure 1 Schematic diagram showing that existing XR-BCI systems can only control controlled objects based on single-view information;

[0057] Figure 2 A schematic diagram of the interface presentation method of visual evoked stimulation and returned video stream of the existing XR-BCI system;

[0058] Figure 3 Schematic diagram of a brain-computer interaction paradigm in an embodiment of the present invention in which the EEG stimulation interface and the environmental video are presented separately to the user's left and right eyes;

[0059] Figure 4 This is a flow chart of a method for implementing a brain-computer interaction paradigm in an embodiment of the present invention;

[0060] Figure 5 This is a schematic diagram of the left canvas, right canvas, and main canvas independently presenting the left eye, right eye, and both eyes in an embodiment of the present invention;

[0061] Figure 6 Schematic diagram of an offline experiment of performing EEG classification on a user based on three presentation modes of the EEG stimulation interface according to an embodiment of the present invention;

[0062] Figure 7 A flow chart of the brain-computer interaction method provided by an embodiment of the present invention;

[0063] Figure 8 This is a schematic diagram of synchronously presenting the first-person perspective video and the third-person perspective video of the controlled object from left to right on the main canvas in an embodiment of the present invention;

[0064] Figure 9 Schematic diagram of presenting the EEG stimulation interface and the control video to the user's stimulation attention eye and video attention eye respectively in an embodiment of the present invention;

[0065] Figure 10 Schematic diagram of normalizing the force deflection degree of the muscles around the user's attention eye and video attention eye in an embodiment of the present invention;

[0066] Figure 11 Schematic diagram of the visual effect presented when the user's active attention result is an electroencephalographic stimulation interface in an embodiment of the present invention;

[0067] Figure 12 Schematic diagram of the brain-computer interaction system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0068] To facilitate a clear description of the technical solutions of the embodiments of the present invention, the words "first" and "second" are used in the embodiments of the present invention to distinguish between identical or similar items with substantially the same functions and effects. For example, the first threshold and the second threshold are merely used to distinguish between different thresholds and do not limit their order. Those skilled in the art will understand that the words "first" and "second" do not limit the quantity or execution order, and the words "first" and "second" do not necessarily mean different.

[0069] It should be noted that, in the present invention, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present invention should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0070] In the present invention, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. The following at least one item (item) or similar expressions thereof refer to any combination of these items, including any combination of single items (items) or plural items (items). For example, at least one item (item) of a, b or c can represent: a, b, c, the combination of a and b, the combination of a and c, the combination of b and c, or the combination of a, b and c, where a, b, c can be single or multiple.

[0071] To address the issue in current XR-BCI systems where the stimulus interface and environmental information obscure and overlap each other, preventing users from maintaining a balanced view, embodiments of the present invention propose a brain-computer interaction paradigm that independently presents multi-classified EEG stimulation and real-time feedback of environmental video to the user's monocular field of view, enabling separate and independent display of stimulation and video in XR mode. Furthermore, the proposed brain-computer interaction paradigm fully utilizes the visual feedback information from the environmental scene. Based on the user's binocular rivalry mechanism and active and passive attention selection mechanism, the user can freely select the stimulus interface or environmental information of interest, creating a visual experience of stimulation and video penetration without obstruction or overlap. Furthermore, building on this extended reality-based brain-computer interface interaction paradigm, embodiments of the present invention also propose a brain-computer interaction method that combines the multimodal perception capabilities of XR display devices with the asynchronous interaction technology of BCI, with both control mode and observation mode. In control mode, the user can freely select the stimulus interface or environmental information of interest and freely output brain control commands based on their control needs. This solves the problem of random output of brain control commands even when the user has no control intention and is not paying attention to the visual stimulus. In observation mode, it is possible to simultaneously present first-person and third-person perspective videos of the controlled object, solving problems such as the current XR-BCI system's inability to simultaneously present the controlled object's environmental information using multiple perspectives, difficulty in highlighting the user's areas of interest, and redundant error outputs in control decisions. This minimizes user misoperations and enables high-performance immersive brain-computer interaction in complex scenarios.

[0072] First, this embodiment provides a brain-computer interaction paradigm, see Figure 3In XR mode, the EEG stimulation interface is presented to the user's attentional eye, while the ambient video is presented to the user's attentional eye. The attentional eye is the eye whose monocular EEG stimulation classification accuracy is closest to that of binocular EEG stimulation, and the other eye becomes the attentional eye for the video. Alternatively, when the EEG stimulation classification accuracy of the user's left eye is equal to that of the right eye, the user's dominant eye is used as the attentional eye, and the other eye becomes the attentional eye for the video. The EEG stimulation interface and the ambient video are switched between near and far based on the user's active and passive attention selection mechanism.

[0073] The brain-computer interaction paradigm proposed in this invention is based on an XR display device, which independently presents multi-classification EEG stimulation and real-time feedback of environmental video to the user's monocular field of view, realizing separate and independent display of stimulation and video in XR mode. Based on the user's binocular rivalry mechanism and active and passive attention selection mechanism, it builds a stimulation and video penetrating, unobstructed and uncovered visual experience for the user, solving the problems of single user perspective and limited field of view under the traditional interaction paradigm. In addition, based on the user's active and passive attention selection mechanism, it can realize free near and far switching of the EEG stimulation interface and environmental video, solving the problems of mutual occlusion and overlap of stimulation and environmental information in the current XR-BCI system, the user cannot take both into account at the same time, the low utilization rate of scene information, and the unclear visual stimulation. It minimizes the problems of misjudgment and decision delay caused by information omission and observation lag during user interaction, improves the efficiency of brain-computer interaction in XR mode, and realizes portable immersive brain-computer interaction.

[0074] In the second aspect, the present invention provides a method for implementing a brain-computer interaction paradigm, for implementing the brain-computer interaction paradigm provided in the first aspect, see Figure 4 , the method for implementing the brain-computer interaction paradigm includes the following steps:

[0075] S10. Construct a rendering camera and a display canvas. The rendering cameras include a left camera, a right camera, and a main camera. The display canvas includes a left canvas, a right canvas, and a main canvas. The left canvas is mounted on the left camera and is displayed independently only to the user's left eye. The right canvas is mounted on the right camera and is displayed independently only to the user's right eye. The main canvas is mounted on the main camera and is displayed synchronously to both eyes of the user.

[0076] As an example, see Figure 5 (a), 5(c), and 5(e) are the spatial coordinate systems of the main camera, left camera, and right camera, respectively. The left canvas, right canvas, and main canvas are mounted on the corresponding rendering cameras, and the center of the display canvas coincides with the origin of the spatial coordinate system of the corresponding camera.

[0077] S11. Write an EEG stimulation interface based on the control instructions required by the controlled object, and render the EEG stimulation interface on the left canvas, the right canvas, and the main canvas.

[0078] As an example, assuming that the control instructions required by the controlled object are four types: rise, fall, turn left, and turn right, the EEG stimulation interface written includes four categories of EEG stimulation, and each EEG stimulation category corresponds to only one control instruction among rise, fall, turn left, and turn right. This embodiment takes eight categories of EEG stimulation as an example, see Figure 5 Figures (b), (d), and (f) are schematic diagrams of rendering the completed EEG stimulation interface, containing eight categories of EEG stimulation (A through H), onto the main, left, and right canvases, respectively. It should be noted that when the EEG stimulation interface is presented independently to one eye, to ensure the user's visual spatial experience, the spatial background remains in the other eye, and the scene display layer is not revoked. This means that the lens is not closed and the field of view is not blacked out.

[0079] S12. Perform an offline EEG classification experiment. Based on the EEG induced by the EEG stimulation interface, determine the single eye with the higher EEG classification accuracy among the user's left and right eyes, and mark it as the stimulation attention eye. The other eye is marked as the video attention eye.

[0080] As a possible implementation, S12 includes:

[0081] S120. When the EEG stimulation interface is only presented on the left canvas, when the EEG stimulation interface is only presented on the right canvas, and when the EEG stimulation interface is only presented on the main canvas, perform EEG classification offline experiments respectively to obtain the user's left eye EEG classification accuracy, right eye EEG classification accuracy, and binocular EEG classification accuracy respectively.

[0082] In order to ensure that the user can obtain the best EEG classification effect when the EEG stimulation interface is presented to one eye and the environment video is presented to the other eye, the embodiment of the present invention proposes to conduct an EEG classification offline experiment on the user, obtain the single eye with the higher EEG classification accuracy of the user's left and right eyes, mark it as the stimulation attention eye, and mark the other eye as the video attention eye. Figure 6 , for the three presentation modes of the EEG stimulation interface, namely, the EEG stimulation interface is presented only on the main canvas, only on the left canvas, and only on the right canvas, an EEG classification offline experiment is conducted on the user. It should be noted that in the three EEG classification offline experiments, variables such as stimulation coding, interface parameters, and user posture are kept consistent, and the EEG classification accuracy of the user's three EEG classification offline experiments is recorded. For example, the EEG classification accuracy recorded when the presentation mode is presented only on the main canvas, only on the left canvas, and only on the right canvas is recorded as Acc respectively. m 、Acc l 、Acc r .

[0083] S121. Calculate the absolute value of the first difference between the left eye EEG classification accuracy and the binocular EEG classification accuracy, and calculate the absolute value of the second difference between the right eye EEG classification accuracy and the binocular EEG classification accuracy; mark the single eye corresponding to the smaller value of the absolute value of the first difference and the absolute value of the second difference as the stimulus attention eye, and mark the other eye as the video attention eye; if the absolute value of the first difference is equal to the absolute value of the second difference, then mark the user's physiological dominant eye as the stimulus attention eye, and the other eye as the video attention eye.

[0084] For example, compare |Acc m ―Acc l |with|Acc m ―Acc r |The size of |Acc m ―Acc l |<|Acc m ―Acc r |, then the user's left eye is marked as the stimulus attention eye, and the right eye is marked as the video attention eye; if |Acc m ―Acc l |>|Acc m ―Acc r |, then the user's right eye is marked as the stimulus attention eye, and the left eye is marked as the video attention eye. It should be noted that if |Acc m ―Acc l |=|Acc m ―Acc r |, the user's dominant eye is marked as the stimulus-focused eye, and the other eye is marked as the video-focused eye. Various methods exist in the prior art for determining the dominant eye, such as the hole method, finger method, arm test method, and Worth four-point instrument method. Any method for determining the dominant eye can achieve the objectives of the present invention and is not specifically limited herein.

[0085] S13. Presenting the EEG stimulation interface to the stimulation eye and presenting the environmental video to the video eye;

[0086] As a possible implementation, S13 includes:

[0087] S130. Unify the spatial coordinate systems of the left and right cameras, and set the left and right cameras to move with the user's head posture; Figure 3As shown in (a), the spatial coordinate systems of the left and right cameras are unified. That is, the X-axis, Y-axis, and Z-axis of the left camera's spatial coordinate system are aligned with the X-axis, Y-axis, and Z-axis of the right camera's spatial coordinate system. The left and right cameras follow the user's head posture, ensuring that the EEG stimulation interface and environmental video are always directly in front of the user's monocular field of view. This further ensures that the user's EEG stimulation effect is better and the user's observation of environmental information is clearer in monocular mode.

[0088] S131. Mount the left canvas at the origin of the spatial coordinate system where the left camera resides, and mount the right canvas at the origin of the spatial coordinate system where the right camera resides. That is, the center of the left canvas coincides with the origin of the spatial coordinate system where the left camera resides, and the center of the right canvas coincides with the origin of the spatial coordinate system where the right camera resides.

[0089] This setup ensures that users can evenly capture both the EEG stimulation interface and the surrounding video. The left and right cameras are positioned at the same depth in terms of field of view, and the centers of the left and right eye images overlap. Due to binocular rivalry, the EEG stimulation interface and the surrounding video will randomly alternate within the user's field of view at the same distance. Due to the human eye's brief visual lingering, a transparent, penetrating visual experience of the EEG stimulation interface and the surrounding video appears within the user's field of view. This means that both the EEG stimulation interface and the transmitted surrounding video may be clearly visible within the same gaze area.

[0090] S132. If the left eye is the eye being stimulated and focused, the left camera presents the electroencephalogram stimulation interface and the right camera presents the environmental video; if the right eye is the eye being stimulated and focused, the right camera presents the electroencephalogram stimulation interface and the left camera presents the environmental video.

[0091] For example, assuming that the left eye is the stimulation attention eye and the right eye is the video attention eye, it proves that the EEG induction effect of the user's left eye is better in the monocular stimulation mode, then the EEG stimulation interface is rendered to the left canvas and displayed to the user's left eye field of view, and the environmental video is rendered to the right canvas and displayed to the user's right eye field of view.

[0092] S14. When the user actively focuses on the electroencephalographic stimulation interface, the electroencephalographic stimulation interface approaches the eyes in a direction perpendicular to the plane where the eyes are located, and the environmental video moves away from the eyes in a direction perpendicular to the plane where the eyes are located; when the user actively focuses on the environmental video, the environmental video moves toward the eyes in a direction perpendicular to the plane where the eyes are located, and the electroencephalographic stimulation interface moves away from the eyes in a direction perpendicular to the plane where the eyes are located.

[0093] As an example, when different images are presented to the user's two eyes, if the user focuses on the image presented in front of the field of view of one eye, the response to the image in the user's visual perception will be stronger, which is the active attention effect; when the image in a certain eye flickers strongly or is closer to the eye, the response to the image in the user's visual perception will also be stronger, which is the passive attention effect. Therefore, during the interaction process, the user can freely choose to look at the EEG stimulation interface or the environmental video based on the active and passive attention mechanisms. If the user needs to output brain control commands, he or she should focus on the EEG stimulation interface in the stimulating eye. At this time, the EEG stimulation interface will also move in the direction close to the eye, the user's EEG induced features will be more obvious, and the EEG classification accuracy will be higher.

[0094] In a third aspect, the present invention provides a brain-computer interaction method. Before online interaction, based on the brain-computer interaction paradigm provided in the first aspect, an offline EEG experiment is performed to obtain n types of control-state EEG features induced when a user gazes at n types of EEG stimulation in an EEG stimulation interface and idle-state EEG features induced when the user gazes at an environmental video. The n types of control-state EEG features corresponding to the n types of EEG stimulation and the idle-state EEG features corresponding to the environmental video are saved as a user EEG classification template.

[0095] During online interaction, the brain-computer interaction method features an observation mode and a control mode, with users freely selecting between them based on whether they have a need for control. Specifically, switching between observation and control modes is based on the user's eye movements. As a possible implementation, eye movements can be captured using eye sensors, using imaging technology to capture the movements of the muscles surrounding the eyes, such as blinking, closing, opening, and looking in different directions. Exemplarily, the user's eye movement for mode switching involves blinking a preset number of times within a preset time, for example, three consecutive blinks within two seconds. In observation mode, the user monitors and makes decisions based on a multi-dimensional video feed of the controlled object. When the user focuses on key information and needs to output brain control commands, they can switch to control mode. In control mode, the user outputs asynchronous brain control commands based on the brain-computer interaction paradigm provided in the first aspect, achieving time-sharing control of the controlled object. When the control operation ends and there is no longer any control intent, the user can also switch to observation mode.

[0096] This invention makes full use of the posture, eye movement, image and other sensor devices of XR devices, and has observation mode and control mode, creating a brain-computer interaction experience with better flexibility, higher freedom of choice and greater fun for users, reducing user fatigue, reducing the load of brain control tasks, and improving the accuracy of brain-computer control decision output.

[0097] See also Figure 7 , brain-computer interaction methods include:

[0098] When switching to observation mode, perform the following operations:

[0099] See also Figure 8 , obtaining a first-person perspective video and a third-person perspective video of the controlled object, and synchronously tiling the first-person perspective video and the third-person perspective video from left to right on the main canvas based on the brain-computer interaction paradigm implementation method provided in the second aspect;

[0100] See also Figure 7 As a possible implementation method, the above operations include:

[0101] S30. Obtaining first-person perspective video and third-person perspective video of the controlled object;

[0102] As an example, the first-person perspective video of the controlled object is obtained via the Real Time Streaming Protocol (RTSP). The third-person perspective video is obtained by calling the front camera of the XR display device. Here, the controlled object is defaulted to the user's foreground perspective. At the same time, the front camera of the XR device is turned on so that the user can observe his or her surroundings.

[0103] S31. Set the main camera not to follow the user's head posture and set the horizontal pixel range of the main canvas equal to the sum of the horizontal pixel ranges of the first-person perspective video and the third-person perspective video, and the vertical pixel range equal to the larger range of the vertical pixel ranges of the first-person perspective video and the third-person perspective video;

[0104] As an example, read the pixel resolution X1*Y1 of the first-person perspective video and the pixel resolution X2*Y2 of the third-person perspective video. Set the horizontal pixel range X of the main canvas m =X1+X2. Set the vertical pixel range Y of the main canvas m =max(Y1, Y2).

[0105] S32. The first-person perspective video and the third-person perspective video are synchronously tiled and presented on the main canvas from left to right, and the horizontal pixel ranges of the first-person perspective video and the third-person perspective video do not overlap.

[0106] As an example, since the horizontal pixel range X of the main canvas has been set in S31 m =X1+X2. Set the vertical pixel range Y of the main canvas m =max(Y1, Y2), therefore, when the first-person perspective video and the third-person perspective video are tiled to the main canvas, the horizontal pixel ranges of the two perspective videos are continuous and non-overlapping.

[0107] The brain-computer interaction method provided by the present invention can simultaneously present first-person perspective video and third-person perspective video of the controlled object, solving the problems of single user perspective and limited field of view under the traditional interaction paradigm.

[0108] See also Figure 7 ,When switching to the control mode, the brain-computer interaction method includes the following sub-steps:

[0109] S20. Determine the control video;

[0110] As a possible implementation, determining the control video includes:

[0111] S200. The user maintains his or her line of sight perpendicular to the plane of the eyes, obtains the range of the yaw angle of the user's head posture when the user deflects from the leftmost end of the main canvas to the rightmost end of the main canvas, and establishes a field of view mapping relationship between the horizontal pixel coordinates of the user's gaze point and the head posture yaw angle;

[0112] As an example, the range of the yaw angle θ of the user's head posture when the user deflects from the leftmost end of the main canvas to the rightmost end of the main canvas is obtained through the posture sensor of the XR device [θ min ,θ max ], where θ min Indicates the yaw angle of the user's head posture when looking at the leftmost end of the main canvas, θ max Indicates the yaw angle of the user's head posture when looking at the rightmost end of the main canvas. Based on the horizontal pixel range of the main canvas (0, X m ], the field of view mapping relationship between the horizontal pixel coordinate Hor_M of the user's attention point and the head posture yaw angle θ is established as follows:

[0113]

[0114] S201. Determine the switching moment from observation mode to control mode based on the field of view mapping relationship, and the horizontal pixel coordinate value of the user's direct attention point corresponding to the yaw angle of the user's head posture. If the horizontal pixel coordinate value belongs to the horizontal pixel range of the first-person perspective video, then determine that the control video is a first-person perspective video; if the horizontal pixel coordinate value belongs to the horizontal pixel range of the third-person perspective video, then determine that the control video is a third-person perspective video.

[0115] As an example, the user's head posture yaw angle at the moment of switching from observation mode to control mode is acquired by the posture sensor. Based on the mapping relationship determined in S200, the horizontal pixel coordinates of the user's direct gaze and attention point are calculated. Because the horizontal pixel ranges of the two perspective videos are continuous and non-overlapping, the user's direct gaze and attention point can only fall into one of the first-person perspective video and the third-person perspective video. If it falls into the first-person perspective video area, the first-person perspective video is used as the control video; if it falls into the third-person perspective video area, the third-person perspective video is used as the control video.

[0116] S21. Based on the brain-computer interaction paradigm provided in the first aspect, the EEG stimulation interface and the control video are presented to the user's stimulation attention eye and video attention eye respectively;

[0117] See also Figure 9 The EEG stimulation interface and control video are presented to the user's stimulation eye and video eye respectively. At this time, a transparent, penetrating, unobstructed and uncovered visual experience of the EEG stimulation interface and control video will appear in the user's field of view.

[0118] S22. Obtain the user's active attention result in real time, and determine whether the user's active attention result is a brain electrical stimulation interface or a control video, specifically: obtain the first horizontal force direction of the muscles around the eye of the user's stimulating attention eye and the second horizontal force direction of the muscles around the eye of the video attention eye in real time; when the second horizontal force direction is toward the stimulating attention eye, and the first horizontal force direction is the same as the second horizontal force direction, determine that the user's active attention result is a brain electrical stimulation interface; otherwise, determine that the user's active attention result is a control video.

[0119] See also Figure 10 As an example, the eye sensors of the XR device are used to obtain the horizontal force directions of the muscles around the user's attention eye and video eye, and the force deflection of the muscles around both eyes is normalized from left to right. For example, the force deflection of the muscles around the user's attention eye is normalized to α∈[0,1], and the force deflection of the muscles around the user's video eye is normalized to β∈[0,1]. When the user focuses their attention on the image within the field of view of one eye, the user's left and right eyes will maintain a certain degree of co-directional movement. Therefore, assuming that the left eye is the stimulating eye, only when α∈[0,0.5) and β∈[0,0.5) do the two eyes move in the same direction and both move to the left. In this case, the user's active attention result is the EEG stimulation interface. Assuming that the right eye is the stimulating eye, only when α∈(0.5,1] and β∈(0.5,1]) do the two eyes move in the same direction and both move to the right. In this case, the user's active attention result is also the EEG stimulation interface. In other cases, the user's active attention result is judged to be the control video.

[0120] If it is a brain electrical stimulation interface, execute S23 to S24; if it is a control video, execute S25;

[0121] S23. Move the EEG stimulation interface closer to the eyes along a direction perpendicular to the plane where the eyes are located, and move the control video away from the eyes along a direction perpendicular to the plane where the eyes are located;

[0122] As an example, based on the user's passive attention mechanism, the display canvas corresponding to the eye being stimulated is dynamically moved. Assuming that the eye being stimulated is the left eye, the left canvas presenting the EEG stimulation interface is translated along the Z-axis depth direction of the left camera's spatial coordinate system, toward the eye; the right canvas presenting the control video is translated along the Z-axis depth direction of the right camera's spatial coordinate system, away from the eye. The X-axis and Y-axis positions of the left and right canvases remain unchanged, and the centers of the left and right eye images are still kept coincident, such as Figure 3 As shown in (a) and 3(b), the left eye is stimulated near and the right eye is far away. Since the user's active attention result is the EEG stimulation interface, the actual visual effect after binocular rivalry is as follows Figure 11 As shown, the visual effect is that the EEG stimulation interface is in front and the control video is behind, and the stimulation is a penetrable effect.

[0123] S24. Real-time acquisition of the user's EEG characteristics, matching them with the user's EEG classification template, and determining whether the EEG characteristics belong to the n-type control state EEG characteristics or the idle state EEG characteristics; if they belong to the n-type control state EEG characteristics, then determining the EEG stimulation category corresponding to the EEG characteristics and outputting the control instructions corresponding to the EEG stimulation category; if they belong to the idle state EEG characteristics, then no control instructions are output;

[0124] The brain-computer interaction method proposed in this invention first conducts an offline EEG experiment on the user before online interaction. The EEG stimulation interface and environmental video are presented to the user's stimulation attention eye and video attention eye, respectively. Through the offline EEG experiment, the user obtains the n types of control-state EEG features induced by the n types of EEG stimulation in the EEG stimulation interface and the idle-state EEG features induced by the user's gaze at the environmental video. The n types of control-state EEG features corresponding to the n types of EEG stimulation and the idle-state EEG features corresponding to the environmental video are saved as the user's EEG classification template. This operation can more accurately judge the user's EEG features in subsequent real-time interactions, and realize asynchronous output of brain-control commands in the presence of environmental interference.

[0125] S25. Move the control video closer to the eyes in a direction perpendicular to the plane where the eyes are located, and move the electroencephalographic stimulation interface away from the eyes in a direction perpendicular to the plane where the eyes are located.

[0126] See also Figure 11When the user needs to observe the environment without issuing control commands, they can focus their attention on the control video behind them through the EEG stimulation interface. The EEG stimulation interface becomes transparent, reducing the visual flickering effect of the stimulation block and weakening the induced EEG signal characteristics. The control video area will not be covered by the EEG stimulation interface, ensuring that the user can observe the environment without missing any information within their field of view.

[0127] The brain-computer interaction method proposed in the present invention combines the multimodal perception function of the XR display device with the asynchronous interaction technology of BCI, and has a control mode and an observation mode. In the control mode, the user can freely select the stimulus interface or environmental information of interest, and freely output brain control commands according to their own control needs, which can solve the problem that brain control commands are randomly output when the user has no control intention and does not look at the visual stimulus. In the observation mode, it is possible to simultaneously present the first-person perspective video and the third-person perspective video of the controlled object, solving the problems of the current XR-BCI system's inability to simultaneously use multi-dimensional perspectives to present the environmental information of the controlled object, the difficulty in highlighting the user's area of ​​interest, and the redundant error output of control decisions. It limits user misoperation to the greatest extent and realizes high-performance immersive brain-computer interaction in complex scenarios.

[0128] In a fourth aspect, the present invention provides a brain-computer interaction system for executing the brain-computer interaction method provided in the third aspect, see Figure 12 , the brain-computer interaction system includes:

[0129] The paradigm presentation module runs the brain-computer interaction paradigm and presents the EEG stimulation interface and control video to the user's stimulation attention eye and video attention eye respectively;

[0130] As an example, the paradigm presentation module is developed based on the Unity3D platform and C# language, runs through the stimulation host, and is presented based on the extended reality display device (the display device in the present invention is a split-screen head-mounted XR display device with gesture sensing, eye tracking, and a front-facing camera, and the display device is connected to the stimulation host). The display device model used in this example is the HTC VIVE Pro Eye, with a maximum field of view of 110° and a refresh rate of 90Hz. The left and right eyes are each equipped with an OLED display with a resolution of 1440*1600, and a binocular resolution of 2880*1600; the stimulation host to which the display device is connected is a DELL workstation with an Intel Xeon 6226R Golden CPU@3.9GHz*2 CPU and an NVIDIA GeForce RTX3090 graphics card.

[0131] The EEG acquisition module acquires the user's EEG signals in real time. As an example, the EEG acquisition module acquires EEG signals based on the Neuroscan SynAmps2 (64-lead) EEG acquisition system.

[0132] The signal processing module performs preprocessing, feature extraction, and template matching on the collected EEG signals, generating corresponding control instructions based on the extracted EEG features. As an example, the signal processing module is written in Python and runs on the processing host using the PyCharm platform. In this example, the processing host is another Dell workstation with an Intel i9-13900K CPU and an NVIDIA GeForce RTX 3060 graphics card.

[0133] The attention selection module obtains the user's active attention results in real time and determines whether it is necessary to output a control instruction. When a control instruction needs to be output, the control instruction generated by the signal processing module is transmitted to the instruction sending module. As an example, the attention selection module is developed based on the Unity platform and C# language and runs through the stimulation host.

[0134] The instruction sending module sends the control instruction to the controlled object, so that the controlled object completes the action corresponding to the control instruction. As an example, the instruction sending module is implemented based on the Transmission Control Protocol (TCP), and can also be implemented based on protocols such as UDP and WebSocket. The host computer runs on the stimulation host.

[0135] The environmental feedback module is used to display the environmental information of the controlled object to the user in an immersive real-time manner in the form of a video stream. As an example, the environmental feedback module obtains the first-person perspective video of the controlled object based on the Real Time Streaming Protocol (RTSP), and obtains the user's third-person perspective video by calling the front camera of the XR display device. The video acquisition and display method is based on the Unity platform and developed in C# language, and runs through the stimulation host.

[0136] And a mode switching module that switches between control mode and observation mode based on the user's eye movements; as an example, the mode switching module is developed based on the Unity platform and C# language and runs via the stimulation host.

[0137] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art can understand and implement other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the drawings, etc. In the specification, the word "comprising" does not exclude other components or steps, and "one" or "an" does not exclude multiple situations. A single processor or other unit can implement several functions listed in the specification. Certain measures are recorded in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0138] Although the present invention has been described with reference to specific features and embodiments thereof, it will be apparent that various modifications and combinations thereof may be made without departing from the spirit and scope of the invention. Accordingly, this specification and drawings are merely illustrative of the present invention and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the invention. It will be apparent that various modifications and variations of the present invention may be made by those skilled in the art without departing from the spirit and scope of the invention. Thus, the present invention is intended to include such modifications and variations as fall within the scope of the invention and its equivalents.

Claims

1. A method for implementing a brain-computer interaction paradigm, characterized in that: The method for implementing the brain-computer interaction paradigm includes: S10. Constructing a rendering camera and display canvas. The rendering camera includes a left camera, a right camera, and a main camera. The display canvas includes a left canvas, a right canvas, and a main canvas. The left canvas is mounted on the left camera and displays only to the user's left eye. The right canvas is mounted on the right camera and displays only to the user's right eye. The main canvas is mounted on the main camera and displays to both eyes simultaneously. S11. Write an EEG stimulation interface based on the control instructions required by the controlled object and render the EEG stimulation interface to the left canvas, right canvas, and main canvas; S12 performs an offline EEG classification experiment. According to the EEG stimulation interface, the user's EEG induced results are determined to determine the user's left and right eyes with a high EEG classification accuracy. The single eye is marked as the stimulation attention eye, and the other eye is marked as the video attention eye. S13. The EEG stimulation interface is presented to the stimulation attention eye, and the environmental video is presented to the video attention eye; S14. When the user actively focuses on the electroencephalographic stimulation interface, the electroencephalographic stimulation interface approaches the eyes in a direction perpendicular to the plane where the eyes are located, and the environmental video moves away from the eyes in a direction perpendicular to the plane where the eyes are located; when the user actively focuses on the environmental video, the environmental video approaches the eyes in a direction perpendicular to the plane where the eyes are located, and the electroencephalographic stimulation interface moves away from the eyes in a direction perpendicular to the plane where the eyes are located.

2. The method for implementing the brain-computer interaction paradigm according to claim 1, characterized in that: The S12 includes: S120. Performing an offline EEG classification experiment when the EEG stimulation interface is displayed only on the left canvas, when the EEG stimulation interface is displayed only on the right canvas, and when the EEG stimulation interface is displayed only on the main canvas, respectively, to obtain the user's left eye EEG classification accuracy, right eye EEG classification accuracy, and binocular EEG classification accuracy; S121. Calculate the absolute value of the first difference between the left eye EEG classification accuracy and the binocular EEG classification accuracy, and calculate the absolute value of the second difference between the right eye EEG classification accuracy and the binocular EEG classification accuracy; mark the single eye corresponding to the smaller value between the absolute value of the first difference and the absolute value of the second difference as the stimulation attention eye, and mark the other eye as the video attention eye; if the absolute value of the first difference is equal to the absolute value of the second difference, then mark the user's physiological dominant eye as the stimulation attention eye, and the other eye as the video attention eye.

3. The method for implementing the brain-computer interaction paradigm according to claim 1, characterized in that: The S13 includes: S130 unifies the spatial coordinate system of the left and right cameras, and sets the left and right cameras to move with the user's head posture; S131. Mounting the left canvas at the origin of the spatial coordinate system where the left camera is located, and mounting the right canvas at the origin of the spatial coordinate system where the right camera is located; S132. If the left eye is the eye being stimulated, the left camera presents the electroencephalogram stimulation interface and the right camera presents the environmental video; if the right eye is the eye being stimulated, the right camera presents the electroencephalogram stimulation interface and the left camera presents the environmental video.

4. A brain-computer interaction paradigm, characterized in that: The method for implementing the brain-computer interaction paradigm described in any one of claims 1 to 3 is applied; the brain-computer interaction paradigm is: presenting the EEG stimulation interface to the user's stimulation attention eye, and presenting the environmental video to the user's video attention eye at the same time; the stimulation attention eye is the eye whose monocular EEG stimulation classification accuracy is closest to the binocular EEG stimulation classification accuracy, and the other eye is the video attention eye; or, when the left eye EEG stimulation classification accuracy is equal to the right eye EEG stimulation classification accuracy, the stimulation attention eye is the user's physiological dominant eye, and the other eye is the video attention eye; the EEG stimulation interface and the environmental video are switched between near and far based on the user's active and passive attention selection mechanism.

5. A brain-computer interaction method, characterized in that: Before online interaction, based on the brain-computer interaction paradigm described in claim 4, an offline EEG experiment is performed to obtain n types of control-state EEG features induced when the user looks at n types of EEG stimulation in the EEG stimulation interface and idle-state EEG features induced when the user looks at the environmental video, and the n types of control-state EEG features corresponding to the n types of EEG stimulation and the idle-state EEG features corresponding to the environmental video are saved as a user EEG classification template; During online interaction, the brain-computer interaction method has an observation mode and a control mode, and switching between the observation mode and the control mode is performed based on the user's eye movements. The brain-computer interaction method includes: When switching to observation mode, perform the following operations: Obtain a first-person perspective video and a third-person perspective video of the controlled object, and synchronously tile the first-person perspective video and the third-person perspective video from left to right on the main canvas based on the brain-computer interaction paradigm implementation method described in claim 2; When switching to the control mode, the brain-computer interaction method includes the following sub-steps: S20. Determine the control video; S21. Based on the brain-computer interaction paradigm of claim 4, the EEG stimulation interface and the control video are presented to the user's stimulation attention eye and video attention eye respectively; S22 real-time acquisition of the user's active attention results, determine the user's active attention results for the EEG stimulation interface or control video, if the EEG stimulation interface, then execute S23 to S24; if the control video, execute S25; S23. Move the EEG stimulation interface closer to the eyes along a direction perpendicular to the plane where the eyes are located, and move the control video away from the eyes along a direction perpendicular to the plane where the eyes are located; S24. Real-time acquisition of the user's EEG characteristics, matching the user's EEG classification template, determining whether the EEG characteristics belong to n-type control state EEG characteristics or idle state EEG characteristics; if they belong to n-type control state EEG characteristics, then determining the EEG stimulation category corresponding to the EEG characteristics, and outputting the control instructions corresponding to the EEG stimulation category; if they are idle state EEG characteristics, then no control instructions are output; S25. Move the control video closer to the eyes along a direction perpendicular to the plane where the eyes are located, and move the electroencephalographic stimulation interface away from the eyes along a direction perpendicular to the plane where the eyes are located.

6. The brain-computer interaction method according to claim 5, characterized in that: Obtaining a first-person perspective video and a third-person perspective video of the controlled object, and synchronously tiling the first-person perspective video and the third-person perspective video from left to right on the main canvas based on the brain-computer interaction paradigm implementation method of claim 1, including: S30 obtains the controlled object's first-person perspective video and third-person perspective video; S31. Set the main camera not to follow the user's head posture and set the horizontal pixel range of the main canvas equal to the sum of the horizontal pixel ranges of the first-person perspective video and the third-person perspective video, and the vertical pixel range equal to the larger range of the vertical pixel ranges of the first-person perspective video and the third-person perspective video; S32. The first-person perspective video and the third-person perspective video are synchronously tiled and presented on the main canvas from left to right, with the horizontal pixel ranges of the first-person perspective video and the third-person perspective video not overlapping.

7. The brain-computer interaction method according to claim 5, characterized in that: The determining of the control video includes: S200. Require the user to maintain their line of sight perpendicular to the plane of their eyes, obtain the range of the yaw angle of the user's head posture as they shift from gazing at the leftmost end of the main canvas to gazing at the rightmost end of the main canvas, and establish a field of view mapping relationship between the horizontal pixel coordinates of the user's gaze point and the yaw angle of their head posture; S201. Determine the horizontal pixel coordinate value of the user's direct attention point corresponding to the yaw angle of the user's head posture at the moment of switching from the observation mode to the control mode based on the field of view mapping relationship. If the horizontal pixel coordinate value belongs to the horizontal pixel range of the first-person perspective video, then determine that the control video is a first-person perspective video; if the horizontal pixel coordinate value belongs to the horizontal pixel range of the third-person perspective video, then determine that the control video is a third-person perspective video.

8. The brain-computer interaction method according to claim 5, characterized in that: The user's active attention result is obtained in real time, and it is determined whether the user's active attention result is a brain electrical stimulation interface or a control video. Specifically, the first horizontal force direction of the muscles around the eye of the user's stimulating attention eye and the second horizontal force direction of the muscles around the eye of the video attention eye are obtained in real time. When the second horizontal force direction is toward the stimulating attention eye, and the first horizontal force direction is the same as the second horizontal force direction, it is determined that the user's active attention result is a brain electrical stimulation interface. Otherwise, it is determined that the user's active attention result is a control video.

9. The brain-computer interaction method according to claim 5, characterized in that: The user's eye movement is blinking a preset number of times continuously within a preset time.

10. A brain-computer interaction system, characterized in that: For executing the brain-computer interaction method according to any one of claims 5 to 9, the brain-computer interaction system comprises: The paradigm presentation module runs the brain-computer interaction paradigm and presents the EEG stimulation interface and control video to the user's stimulation attention eye and video attention eye respectively; EEG acquisition module, collects the user's EEG signals in real time; The signal processing module performs preprocessing, feature extraction and template matching on the collected EEG signals, and generates corresponding control instructions based on the extracted EEG features; The attention selection module obtains the user's active attention results in real time and determines whether a control instruction needs to be output. If a control instruction needs to be output, the control instruction generated by the signal processing module is transmitted to the instruction sending module; An instruction sending module sends the control instruction to the controlled object, so that the controlled object completes the action corresponding to the control instruction; The environmental feedback module transmits the environmental information of the controlled object to the user in real time in the form of a video stream for immersive display; and a mode switching module that switches between control mode and observation mode based on the user's eye movements.

Images