Brain-computer interaction normal form, implementation method, brain-computer interaction method and system

Abstract

The invention relates to the technical field of brain-computer interfaces, and discloses a brain-computer interaction normal form, an implementation method, a brain-computer interaction method and a brain-computer interaction system. The brain-computer interaction normal form presents the brain electrical stimulation interface to a user stimulation attention eye, and presents the environment video to a user video attention eye at the same time; and performing far-near switching between the brain electrical stimulation interface and the environment video based on the active and passive attention selection mechanism of the user. The brain-computer interaction method has an observation mode and a control mode, and switching between the observation mode and the control mode is carried out based on eye actions of a user. In the observation mode, the first-person view angle video and the third-person view angle video are synchronously tiled and presented to the main canvas from left to right; in the control mode, electroencephalogram characteristics of the user are obtained in real time, and when it is judged that the electroencephalogram characteristics are control-state electroencephalogram characteristics, the corresponding electroencephalogram stimulation types are determined, and control instructions corresponding to the electroencephalogram stimulation types are output. According to the method, the misoperation of the user can be limited to the maximum extent, and high-performance immersive brain-computer interaction in a complex scene is realized.

Description

Technical Field

[0001] The present invention relates to the field of brain-computer interface technology, and in particular to a brain-computer interaction paradigm, an implementation method, a brain-computer interaction method and a system. Background Art

[0002] Brain-computer Interface (BCI) technology is a technology that can establish a direct communication channel between the human brain and external devices without relying on conventional brain information output pathways such as peripheral nerves and muscle tissue. At present, based on the BCI system, especially the non-invasive BCI system with less trauma, less risk and simpler operation, it is possible to achieve intelligent control of a variety of external devices such as wheelchairs, robotic arms, drones and unmanned vehicles. With the expansion of the application scenarios of brain-computer interaction systems and the increase in the control dimensions of the controlled objects, the requirements for the flexibility and convenience of BCI systems are becoming higher and higher. Traditional BCI systems have the following disadvantages during the interaction process: (1) In terms of display, the system needs to use a display screen to present the stimulation interface, and the interactive device is less portable; (2) In terms of disturbance, users are easily disturbed by the surrounding environment information during use, and the interactive immersion is low, especially in noisy environments, which can easily affect user decision-making; (3) In terms of feedback, the traditional single interface can only present limited environmental information. During the brain-computer interaction control process, the utilization rate of the scene information sent back by the controlled object is low; (4) In terms of comfort, the user's field of view under the traditional display screen is limited to the screen area. In order to ensure a good EEG induction effect of visual stimulation, the user needs to maintain a fixed posture for a long time, which not only increases the user's fatigue, but also increases the user's mental workload.

[0003] Extended Reality (XR) refers to the use of computers to combine the real world with the virtual world to build a virtual environment for human-computer interaction. It is an immersive interactive technology that can create an immersive experience for the experiencer. XR technology includes virtual reality (VR), augmented reality (AR) and mixed reality (MR), which can be applied to entertainment, education, medicine, military and other fields. With the continuous iteration and update of XR technology, more and more research teams are combining BCI technology with XR technology for rehabilitation training. However, nowadays, as the XR-BCI system gradually moves from simulation training in virtual scenes to immersive control in actual scenes, the real environment information that users need to pay attention to is increasing, the complexity of user control tasks is increasing, and the requirements for the flexibility and accuracy of the interactive system are getting higher and higher. The existing XR-BCI system has become increasingly difficult to meet the complex needs of users. Take the brain-controlled drone system based on steady-state visually evoked potential (SSVEP) in XR mode as an example. Figure 1 As shown in the figure, in terms of environmental information presentation, the existing XR-BCI system can only control the drone based on a single perspective. During the interaction, the user can only perform first-person perspective control based on the video sent back by the drone's front-view camera, or perform third-person perspective control based on the drone's real flight environment captured by other auxiliary equipment. It is impossible to simultaneously or switch between presenting multi-perspective environmental information that facilitates the user to observe and judge the state of the controlled object. As portable XR-BCI systems gradually move from indoors to outdoors, the complexity of tasks that users need to complete is also gradually increasing. Therefore, as an important basis for decision-making and judgment in the user's immersive interaction process, how to provide users with more dimensional reference environmental information in XR mode has become an important issue that needs to be solved in today's XR-BCI system.

[0004] Most importantly, in existing XR-BCI systems, the interface presentation mode including visual induced stimulation and returned video stream is mostly in surround mode or overlay mode. Figure 2 As shown in (a), the returned video is located in the center of the field of view, and the flickering stimulus is spread around the video stream. During the interaction process, the user must first pay attention to the central environmental information, and then shift his gaze to look at the corresponding stimulus to output the control command after making a control decision. In this mode, the user's control decision output has a certain lag, and the user cannot pay attention to the real-time environmental information in time when looking at the stimulus, which is easy to cause danger when the environmental information changes rapidly. In the overlay mode, as shown in the attached figure, Figure 2As shown in (b), in the case of near-eye immersive display in XR mode, the stimulus will cover a large area of ​​the returned video frame, which will cause the user to have a certain lack of judgment on the environment, and also cause the user to misjudge, increasing the risk of peripheral control. Taking the first-person perspective control of the brain-controlled drone as an example, if the stimulus coverage area happens to be the obstacle in front, and the user fails to make a timely decision to change the route to avoid it due to information omission, it will cause the drone to collide with it, which will cause certain risks to the equipment, environment and personnel. Therefore, whether it is possible to develop a brain-computer interaction visual stimulation paradigm that allows users to take into account both stimulus and environmental information at the same time, and without occlusion or overlap, in the XR mode, with the help of the binocular independent rendering mode of the immersive display device, has become another important problem that the XR-BCI system needs to solve.

[0005] At the same time, in the traditional XR-BCI interaction process, users cannot focus on the areas of interest in the environmental feedback information, and the utilization rate of scene information is low. If users can selectively highlight and amplify key environmental information during the interaction process, it can greatly reduce misjudgments and improve the accuracy of brain-computer decision output. In addition, most of the existing XR-BCIs are brain-computer synchronous interaction modes. On the one hand, brain-computer interaction stimuli are all synchronized with the video, and on the other hand, brain-computer interaction instructions are all synchronized with the stimulation output. In this mode, on the one hand, the continuous stimulation will cause redundant interference to the user and affect the user's decision-making. On the other hand, when the user has no control intention and does not look at the visual stimulation, the brain control command will also be randomly output. This operation will bring greater control risks, and the randomly output erroneous instructions will affect the normal operation process of the controlled device and reduce the efficiency of brain-computer interaction. Nowadays, XR display devices are constantly optimizing and iterating in terms of posture sensing, visual sensing, and spatial interaction. Therefore, whether it is possible to combine the perception function of XR with the EEG decoding technology of BCI to obtain the information that the user focuses on, and based on the user's attention selection, build an asynchronous brain-computer interaction system with better flexibility, higher utilization of visual information, and fewer erroneous control decision outputs to achieve selective output of brain control commands has become an important issue that needs to be solved urgently. Summary of the invention

[0006] The purpose of the present invention is to provide a brain-computer interaction paradigm, implementation method, brain-computer interaction method and system. The brain-computer interaction paradigm is based on an XR display device, and multi-classification EEG stimulation and real-time feedback environmental video are presented independently to the user's monocular field of view, realizing stimulation and video separation and independent display in XR mode, and constructing a stimulation, video penetration, unobstructed and uncovered visual experience for users based on the user's binocular competition mechanism and active and passive attention selection mechanism. The brain-computer interaction method combines the multimodal perception function of the XR display device with the asynchronous interaction technology of BCI, so that users can freely select the required key information and freely output interaction instructions according to user control requirements, solving the problem that the existing XR-BCI system control instructions cannot be selectively output. The present invention solves the problems that the current XR-BCI system cannot simultaneously use multi-dimensional perspectives to present the environmental information of the controlled object, it is difficult to highlight the user's area of ​​interest, and the redundant error output of control decisions, and limits user misoperation to the greatest extent, and realizes high-performance immersive brain-computer interaction in complex scenes.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] In the first aspect, the present invention provides a brain-computer interaction paradigm, which presents an electroencephalogram (EEG) stimulation interface to the user's stimulation attention eye, and simultaneously presents an environmental video to the user's video attention eye; the stimulation attention eye is the eye whose monocular EEG stimulation classification accuracy is closest to the binocular EEG stimulation classification accuracy, and the other eye is the video attention eye; or, when the left eye EEG stimulation classification accuracy is equal to the right eye EEG stimulation classification accuracy, the stimulation attention eye is the user's physiological dominant eye, and the other eye is the video attention eye; the EEG stimulation interface and the environmental video are switched between near and far based on the user's active and passive attention selection mechanism.

[0009] In a second aspect, the present invention provides a method for implementing a brain-computer interaction paradigm, which is used to implement the brain-computer interaction paradigm provided in the first aspect, and the method for implementing the brain-computer interaction paradigm includes:

[0010] S10. Constructing a rendering camera and a display canvas, wherein the rendering camera includes a left camera, a right camera and a main camera, and the display canvas includes a left canvas, a right canvas and a main canvas; wherein the left canvas is mounted on the left camera and is displayed only to the left eye of the user independently; the right canvas is mounted on the right camera and is displayed only to the right eye of the user independently; the main canvas is mounted on the main camera and is displayed to both eyes of the user synchronously;

[0011] S11. Write an EEG stimulation interface based on the control instructions required by the controlled object, and render the EEG stimulation interface on the left canvas, the right canvas, and the main canvas;

[0012] S12. Perform an offline EEG classification experiment, and determine the single eye with a higher EEG classification accuracy among the left and right eyes of the user according to the EEG induced results of the user by the EEG stimulation interface, and mark it as the stimulation attention eye, and mark the other eye as the video attention eye;

[0013] S13. presenting the EEG stimulation interface to the stimulation attention eye and presenting the environment video to the video attention eye;

[0014] S14. When the user actively focuses on the electroencephalographic stimulation interface, the electroencephalographic stimulation interface approaches the eyes in a direction perpendicular to the plane where the eyes are located, and the environmental video moves away from the eyes in a direction perpendicular to the plane where the eyes are located; when the user actively focuses on the environmental video, the environmental video approaches the eyes in a direction perpendicular to the plane where the eyes are located, and the electroencephalographic stimulation interface moves away from the eyes in a direction perpendicular to the plane where the eyes are located.

[0015] As a possible implementation, S12 includes:

[0016] S120. When the EEG stimulation interface is only presented on the left canvas, when the EEG stimulation interface is only presented on the right canvas, and when the EEG stimulation interface is only presented on the main canvas, respectively perform an EEG classification offline experiment to obtain the user's left eye EEG classification accuracy, right eye EEG classification accuracy, and binocular EEG classification accuracy;

[0017] S121. Calculate the absolute value of the first difference between the left eye EEG classification accuracy and the binocular EEG classification accuracy, and calculate the absolute value of the second difference between the right eye EEG classification accuracy and the binocular EEG classification accuracy; mark the single eye corresponding to the smaller value of the absolute value of the first difference and the absolute value of the second difference as the stimulation attention eye, and mark the other eye as the video attention eye; if the absolute value of the first difference is equal to the absolute value of the second difference, then the user's physiological dominant eye is used as the stimulation attention eye, and the other eye is used as the video attention eye.

[0018] As a possible implementation, S13 includes:

[0019] S130. Unify the spatial coordinate systems of the left camera and the right camera, and set the left camera and the right camera to move with the user's head posture;

[0020] S131. Mount the left canvas at the origin of the space coordinate system where the left camera is located, and mount the right canvas at the origin of the space coordinate system where the right camera is located;

[0021] S132. If the left eye is the eye that is being stimulated and paid attention to, the left camera will present the electroencephalogram stimulation interface, and the right camera will present the environmental video; if the right eye is the eye that is being stimulated and paid attention to, the right camera will present the electroencephalogram stimulation interface, and the left camera will present the environmental video.

[0022] In a third aspect, the present invention provides a brain-computer interaction method. Before online interaction, based on the brain-computer interaction paradigm provided in the first aspect, an offline EEG experiment is performed to obtain n types of control-state EEG features induced when a user gazes at n types of EEG stimulation in an EEG stimulation interface and idle-state EEG features induced when the user gazes at an environmental video, and the n types of control-state EEG features corresponding to the n types of EEG stimulation and the idle-state EEG features corresponding to the environmental video are saved as a user EEG classification template.

[0023] During online interaction, the brain-computer interaction method has an observation mode and a control mode. The switching between the observation mode and the control mode is performed based on the user's eye movements. The brain-computer interaction method includes:

[0024] When switching to observation mode, perform the following operations:

[0025] Obtain the first-person perspective video and the third-person perspective video of the controlled object, and based on the brain-computer interaction paradigm implementation method provided in the second aspect, synchronously tile the first-person perspective video and the third-person perspective video from left to right on the main canvas;

[0026] When switching to the control mode, the brain-computer interaction method includes the following sub-steps:

[0027] S20. Determine the control video;

[0028] S21. Based on the brain-computer interaction paradigm provided in the first aspect, the EEG stimulation interface and the control video are presented to the user's stimulation attention eye and video attention eye respectively;

[0029] S22. Obtain the user's active attention result in real time, and determine whether the user's active attention result is a brain electrical stimulation interface or a control video. If it is a brain electrical stimulation interface, execute S23 to S24; if it is a control video, execute S25;

[0030] S23. Move the EEG stimulation interface closer to the eyes along a direction perpendicular to the plane where the eyes are located, and move the control video away from the eyes along a direction perpendicular to the plane where the eyes are located;

[0031] S24. Obtain the user's EEG characteristics in real time, match them with the user's EEG classification template, and determine whether the EEG characteristics belong to the n-type control state EEG characteristics or the idle state EEG characteristics; if they belong to the n-type control state EEG characteristics, determine the EEG stimulation category corresponding to the EEG characteristics, and output the control instructions corresponding to the EEG stimulation category; if they are idle state EEG characteristics, do not output the control instructions;

[0032] S25. Move the control video closer to the eyes along a direction perpendicular to the plane where the eyes are located, and move the electroencephalographic stimulation interface away from the eyes along a direction perpendicular to the plane where the eyes are located.

[0033] As a possible implementation method, a first-person perspective video and a third-person perspective video of the controlled object are obtained, and the first-person perspective video and the third-person perspective video are synchronously tiled from left to right to present on the main canvas, including:

[0034] S30. Obtaining the first-person perspective video and the third-person perspective video of the controlled object;

[0035] S31. Set the main camera not to move with the user's head posture, and set the horizontal pixel range of the main canvas to be equal to the sum of the horizontal pixel ranges of the first-person perspective video and the third-person perspective video, and the vertical pixel range to be equal to the larger range of the vertical pixel ranges of the first-person perspective video and the third-person perspective video;

[0036] S32. The first-person perspective video and the third-person perspective video are synchronously tiled and presented on the main canvas from left to right, and the horizontal pixel ranges of the first-person perspective video and the third-person perspective video do not overlap.

[0037] As a possible implementation method, determining the control video includes:

[0038] S200. Make the user keep his sight perpendicular to the plane where the eyes are located, obtain the range of the yaw angle of the head posture when the user deflects from the leftmost end of the main canvas to the rightmost end of the main canvas, and establish a field of view mapping relationship between the horizontal pixel coordinates of the user's direct attention point and the yaw angle of the head posture;

[0039] S201. Determine the switching moment from observation mode to control mode based on the field of view mapping relationship, and the horizontal pixel coordinate value of the user's direct attention point corresponding to the yaw angle of the user's head posture. If the horizontal pixel coordinate value belongs to the horizontal pixel range of the first-person perspective video, then determine that the control video is a first-person perspective video; if the horizontal pixel coordinate value belongs to the horizontal pixel range of the third-person perspective video, then determine that the control video is a third-person perspective video.

[0040] As a possible implementation method, the user's active attention result is obtained in real time, and it is determined whether the user's active attention result is an electroencephalogram stimulation interface or a video control method. Specifically, the first horizontal force direction of the muscles around the eye of the user's stimulating attention eye and the second horizontal force direction of the muscles around the eye of the video attention eye are obtained in real time. When the second horizontal force direction is toward the stimulating attention eye, and the first horizontal force direction is the same as the second horizontal force direction, it is determined that the user's active attention result is an electroencephalogram stimulation interface. Otherwise, it is determined that the user's active attention result is a video control method.

[0041] As a possible implementation manner, the eye movement of the user is blinking a preset number of times continuously within a preset time.

[0042] In a fourth aspect, the present invention provides a brain-computer interaction system for executing the brain-computer interaction method provided in the third aspect, the brain-computer interaction system comprising:

[0043] The paradigm presentation module runs the brain-computer interaction paradigm and presents the EEG stimulation interface and the control video to the user's stimulation attention eye and video attention eye respectively;

[0044] EEG acquisition module, collects the user's EEG signals in real time;

[0045] The signal processing module performs preprocessing, feature extraction and template matching on the collected EEG signals, and generates corresponding control instructions based on the extracted EEG features;

[0046] The attention selection module obtains the user's active attention results in real time and determines whether a control instruction needs to be output. When a control instruction needs to be output, the control instruction generated by the signal processing module is transmitted to the instruction sending module;

[0047] The command sending module sends the control command to the controlled object, so that the controlled object completes the action corresponding to the control command;

[0048] The environmental feedback module is used to display the environmental information of the controlled object to the user in real time in an immersive manner by transmitting the information back in the form of video streams;

[0049] and a mode switching module for switching between control mode and observation mode based on the user's eye movements.

[0050] Compared with the prior art, the beneficial effects of the present invention are:

[0051] 1. The brain-computer interaction paradigm proposed in the present invention presents multi-classification EEG stimulation and real-time feedback of environmental videos independently to the user's monocular field of view, realizing the separate and independent display of stimulation and video in the XR mode, and constructs a stimulating, video-penetrating, unobstructed and uncovered visual experience for the user based on the user's binocular competition mechanism and active and passive attention selection mechanism.

[0052] 2. The brain-computer interaction paradigm proposed in the present invention can realize the free near and far switching of the electroencephalographic stimulation interface and the environmental video based on the user's active and passive attention selection mechanism, which solves the problems in the current XR-BCI system that the stimulation and environmental information block and overlap each other, the user cannot take both into account at the same time, the scene information utilization rate is low, and the visual stimulation is not obvious. It minimizes the problems of misjudgment and decision delays caused by information omissions and observation lags in the user interaction process, improves the efficiency of brain-computer interaction in the XR mode, and realizes portable immersive brain-computer interaction.

[0053] 3. The brain-computer interaction method proposed in the present invention saves the control state EEG features and the idle state EEG features together to the user's EEG classification template before the interaction, which can realize the asynchronous output of brain control commands under environmental interference. At the same time, the brain-computer interaction method proposed in the present invention combines the multimodal perception function of the XR display device with the asynchronous interaction technology of BCI, solving the problems that the current XR-BCI system cannot simultaneously use multi-dimensional perspectives to present the environmental information of the controlled object, it is difficult to highlight the user's area of ​​interest, and the redundant error output of control decisions, thereby limiting user misoperation to the greatest extent and realizing high-performance immersive brain-computer interaction in complex scenarios.

[0054] 4. The brain-computer interaction method proposed in the present invention makes full use of the posture, eye movement, image and other sensor devices of the XR device, and has an observation mode and a control mode. The two modes can be switched freely, creating a more flexible, more free choice, and more interesting brain-computer interaction experience for users, reducing user fatigue, reducing brain control task load, and improving the accuracy of brain-computer control decision output. Moreover, in the observation mode, the first-person perspective video and the third-person perspective video of the controlled object can be presented simultaneously, solving the problem of single user perspective and limited field of view under the traditional interaction paradigm. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0056] Figure 1 A schematic diagram showing that the existing XR-BCI system can only control the controlled object based on single-view information;

[0057] Figure 2 A schematic diagram of the interface presentation method of visual induced stimulation and returned video stream of the existing XR-BCI system;

[0058] Figure 3 This is a schematic diagram of a brain-computer interaction paradigm in an embodiment of the present invention in which the brain electrical stimulation interface and the environmental video are presented separately to the left and right eyes of the user;

[0059] Figure 4 This is a flow chart of a method for implementing a brain-computer interaction paradigm in an embodiment of the present invention;

[0060] Figure 5 This is a schematic diagram of the left canvas, the right canvas and the main canvas independently presenting the left eye, the right eye and both eyes in an embodiment of the present invention;

[0061] Figure 6 It is a schematic diagram of an offline experiment of performing EEG classification on a user according to three presentation modes of the EEG stimulation interface in an embodiment of the present invention;

[0062] Figure 7 A flow chart of a brain-computer interaction method provided by an embodiment of the present invention;

[0063] Figure 8 It is a schematic diagram of synchronously tiling and presenting the first-person perspective video and the third-person perspective video of the controlled object from left to right on the main canvas in an embodiment of the present invention;

[0064] Fig. 9 A schematic diagram of presenting the EEG stimulation interface and the control video to the stimulation attention eye and the video attention eye of the user respectively in an embodiment of the present invention;

[0065] Fig.10 It is a schematic diagram of normalizing the force deflection degree of the muscles around the eye of the user's stimulated attention eye and the video attention eye in an embodiment of the present invention;

[0066] Fig.11 It is a schematic diagram of the visual effect presented when the active attention result of the user in the embodiment of the present invention is the brain electrical stimulation interface;

[0067] Fig.12 Schematic diagram of the brain-computer interaction system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0068] In order to clearly describe the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, words such as "first" and "second" are used to distinguish the same items or similar items with basically the same functions and effects. For example, the first threshold and the second threshold are only used to distinguish different thresholds, and their order is not limited. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0069] It should be noted that, in the present invention, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0070] In the present invention, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The following at least one item (items) or similar expressions thereof refer to any combination of these items, including any combination of single items (items) or plural items (items). For example, at least one item (items) of a, b or c can mean: a, b, c, the combination of a and b, the combination of a and c, the combination of b and c, or the combination of a, b and c, where a, b, c can be single or multiple.

[0071] In view of the problem that the stimulation interface and environmental information in the current XR-BCI system block and cover each other, and the user cannot take both into account, the embodiment of the present invention proposes a brain-computer interaction paradigm, which presents multi-classification EEG stimulation and real-time feedback environmental video to the user's monocular field of view independently, and realizes the stimulation and video separation independent display in XR mode. In addition, the brain-computer interaction paradigm proposed in this embodiment can make full use of the visual feedback information of the environmental scene, and based on the user's binocular competition mechanism and active and passive attention selection mechanism, it can freely select the stimulation interface or environmental information of interest, and build a stimulation, video penetration, unobstructed and uncovered visual experience for the user. In addition, based on the brain-computer interface interaction paradigm based on extended reality, the embodiment of the present invention also proposes a brain-computer interaction method, which combines the multimodal perception function of the XR display device with the asynchronous interaction technology of BCI, and has a control mode and an observation mode. In the control mode, the user can freely select the stimulation interface or environmental information of interest, and freely output brain control instructions according to his own control needs, which can solve the problem that the brain control instructions are randomly output when the user has no control intention and does not look at the visual stimulus. In observation mode, it is possible to simultaneously present the first-person perspective video and the third-person perspective video of the controlled object, solving the problems of the current XR-BCI system's inability to simultaneously use multi-dimensional perspectives to present the controlled object's environmental information, difficulty in highlighting the user's area of ​​interest, and redundant error output of control decisions. This can limit user misoperations to the greatest extent and achieve high-performance immersive brain-computer interaction in complex scenarios.

[0072] In the first aspect, this embodiment provides a brain-computer interaction paradigm, see Figure 3In XR mode, the EEG stimulation interface is presented to the user's stimulation eye, and the environmental video is presented to the user's video eye. The stimulation eye is the eye whose monocular EEG stimulation classification accuracy is closest to the binocular EEG stimulation classification accuracy, and the other eye is the video eye; or, when the user's left eye EEG stimulation classification accuracy is equal to the right eye EEG stimulation classification accuracy, the user's physiological dominant eye is the stimulation eye, and the other eye is the video eye; the EEG stimulation interface and the environmental video are switched far and near based on the user's active and passive attention selection mechanism.

[0073] The brain-computer interaction paradigm proposed in the present invention, based on the XR display device, presents multi-classification EEG stimulation and real-time feedback environmental video to the user's monocular field of view independently, realizing the stimulation and video separated independent display in the XR mode, and constructing the stimulation and video penetrating visual experience without obstruction and coverage for the user based on the user's binocular competition mechanism and active and passive attention selection mechanism, solving the problem of single user perspective and limited field of view under the traditional interaction paradigm. And based on the user's active and passive attention selection mechanism, the free near and far switching of the EEG stimulation interface and the environmental video can be realized, solving the problems of mutual obstruction and coverage of stimulation and environmental information in the current XR-BCI system, the user cannot take both into account at the same time, the low utilization rate of scene information, and the unclear visual stimulation, minimizing the misjudgment and decision delay caused by information omission and observation lag in the user interaction process, improving the efficiency of brain-computer interaction in the XR mode, and realizing portable immersive brain-computer interaction.

[0074] In a second aspect, the present invention provides a method for implementing a brain-computer interaction paradigm, for implementing the brain-computer interaction paradigm provided in the first aspect, see Figure 4 , the method for implementing the brain-computer interaction paradigm includes the following steps:

[0075] S10. Construct a rendering camera and a display canvas. The rendering camera includes a left camera, a right camera and a main camera. The display canvas includes a left canvas, a right canvas and a main canvas. Among them, the left canvas is mounted on the left camera and is only displayed independently to the user's left eye; the right canvas is mounted on the right camera and is only displayed independently to the user's right eye; the main canvas is mounted on the main camera and is displayed synchronously to the user's two eyes.

[0076] As an example, see Figure 5 (a), 5(c), and 5(e) are the spatial coordinate systems of the main camera, left camera, and right camera, respectively. The left canvas, right canvas, and main canvas are mounted on the corresponding rendering cameras, and the center of the display canvas coincides with the origin of the spatial coordinate system of the corresponding camera.

[0077] S11. Write an EEG stimulation interface based on the control instructions required by the controlled object, and render the EEG stimulation interface on the left canvas, the right canvas, and the main canvas.

[0078] As an example, assuming that the control instructions required by the controlled object are four types: rise, fall, turn left, and turn right, the programmed EEG stimulation interface includes four categories of EEG stimulation, and each EEG stimulation category corresponds to only one control instruction among rise, fall, turn left, and turn right. Figure 5 (b), 5(d), and 5(f) are schematic diagrams of rendering the EEG stimulation interface containing eight categories of EEG stimulation from A to H to the main canvas, the left canvas, and the right canvas, respectively. It should be noted that when the EEG stimulation interface is presented independently in one eye, in order to ensure the user's visual space experience, the other eye still retains the spatial background and does not cancel the scene display layer, that is, the lens is not closed and the field of view is not blacked out.

[0079] S12. Perform an offline EEG classification experiment. According to the EEG induction results of the user by the EEG stimulation interface, determine the single eye with higher EEG classification accuracy among the left and right eyes of the user, mark it as the stimulation attention eye, and mark the other eye as the video attention eye.

[0080] As a possible implementation, S12 includes:

[0081] S120. When the EEG stimulation interface is only presented on the left canvas, when the EEG stimulation interface is only presented on the right canvas, and when the EEG stimulation interface is only presented on the main canvas, perform EEG classification offline experiments respectively to obtain the user's left eye EEG classification accuracy, right eye EEG classification accuracy, and binocular EEG classification accuracy respectively.

[0082] In order to ensure that the user can obtain the best EEG classification effect when the EEG stimulation interface is presented to one eye and the environment video is presented to the other eye, the embodiment of the present invention proposes to conduct an EEG classification offline experiment on the user, obtain the single eye with higher EEG classification accuracy among the left and right eyes of the user, mark it as the stimulation attention eye, and mark the other eye as the video attention eye. Figure 6 For the three presentation modes of the EEG stimulation interface, i.e., the EEG stimulation interface is presented only on the main canvas, only on the left canvas, and only on the right canvas, an EEG classification offline experiment is conducted on the user. It should be noted that in the three EEG classification offline experiments, variables such as stimulation coding, interface parameters, and user posture are kept consistent, and the EEG classification accuracy of the user's three EEG classification offline experiments is recorded. For example, the EEG classification accuracy recorded when the presentation mode is presented only on the main canvas, only on the left canvas, and only on the right canvas is recorded as Acc respectively. m 、Acc l 、Acc r .

[0083] S121. Calculate the absolute value of the first difference between the left eye EEG classification accuracy and the binocular EEG classification accuracy, and calculate the absolute value of the second difference between the right eye EEG classification accuracy and the binocular EEG classification accuracy; mark the single eye corresponding to the smaller value of the absolute value of the first difference and the absolute value of the second difference as the stimulation attention eye, and mark the other eye as the video attention eye; if the absolute value of the first difference is equal to the absolute value of the second difference, then the user's physiological dominant eye is used as the stimulation attention eye, and the other eye is used as the video attention eye.

[0084] For example, compare |Acc m ―Acc l |With|Acc m ―Acc r | size, if |Acc m ―Acc l |<|Acc m ―Acc r |, then the user's left eye is marked as the stimulus attention eye, and the right eye is marked as the video attention eye; if |Acc m ―Acc l |>|Acc m ―Acc r |, the user's right eye is marked as the stimulus attention eye, and the left eye is marked as the video attention eye. It should be noted that if |Acc m ―Acc l |=|Acc m ―Acc r |, the user's physiological dominant eye is marked as the stimulus attention eye, and the other eye is marked as the video attention eye. There are many methods for determining the physiological dominant eye in the prior art, such as: hole method, finger method, arm test method, Worth four-point instrument method. Any method for determining the physiological dominant eye can achieve the purpose of the present invention, and no specific limitation is made here.

[0085] S13. presenting the EEG stimulation interface to the stimulation attention eye and presenting the environment video to the video attention eye;

[0086] As a possible implementation, S13 includes:

[0087] S130. Unify the spatial coordinate systems of the left camera and the right camera, and set the left camera and the right camera to move with the user's head posture; Figure 3As shown in (a), the spatial coordinate systems of the left camera and the right camera are unified, that is, the X-axis, Y-axis, and Z-axis of the spatial coordinate system of the left camera are consistent with the X-axis, Y-axis, and Z-axis of the spatial coordinate system of the right camera. The left camera and the right camera move with the user's head posture, ensuring that the EEG stimulation interface and the environmental video are always located in front of the user's monocular field of view, further ensuring that the user's EEG stimulation induction effect is better and the observation of environmental information is clearer in the monocular mode.

[0088] S131. Mount the left canvas at the origin of the space coordinate system where the left camera is located, and mount the right canvas at the origin of the space coordinate system where the right camera is located. That is, the center of the left canvas coincides with the origin of the space coordinate system where the left camera is located; the center of the right canvas coincides with the origin of the space coordinate system where the right camera is located.

[0089] This setting is to ensure that users can evenly capture the EEG stimulation interface and environmental video. At this time, the left and right cameras are at the same depth in the field of view, and the left and right eye images overlap. Based on the binocular competition mechanism, at the same distance, the EEG stimulation interface and environmental video will be randomly presented alternately in the user's field of view. Due to the short visual stay of the human eye, a transparent penetrating visual experience of the EEG stimulation interface and environmental video will appear in the field of view, that is, the EEG stimulation interface and the returned environmental video may be clearly presented in the same gaze area.

[0090] S132. If the left eye is the eye that is being stimulated and focused, the left camera will present the electroencephalographic stimulation interface, and the right camera will present the environmental video; if the right eye is the eye that is being stimulated and focused, the right camera will present the electroencephalographic stimulation interface, and the left camera will present the environmental video.

[0091] Exemplarily, assuming that the left eye is the stimulation attention eye and the right eye is the video attention eye, it proves that the EEG induction effect of the user's left eye is better in the monocular stimulation mode. The EEG stimulation interface is rendered to the left canvas and displayed to the user's left eye field of view, and the environmental video is rendered to the right canvas and displayed to the user's right eye field of view.

[0092] S14. When the user actively focuses on the electroencephalographic stimulation interface, the electroencephalographic stimulation interface approaches the eyes in a direction perpendicular to the plane where the eyes are located, and the environmental video moves away from the eyes in a direction perpendicular to the plane where the eyes are located; when the user actively focuses on the environmental video, the environmental video approaches the eyes in a direction perpendicular to the plane where the eyes are located, and the electroencephalographic stimulation interface moves away from the eyes in a direction perpendicular to the plane where the eyes are located.

[0093] As an example, when the user's two eyes present different pictures, if the user focuses on the picture presented in front of the field of view of one eye, the response of the user's visual perception to the picture will be stronger, which is the active attention effect; when the picture in one eye flickers strongly or is closer to the eye, the response of the user's visual perception to the picture will also be stronger, which is the passive attention effect. Therefore, during the interaction process, the user can freely choose to look at the EEG stimulation interface or the environmental video based on the active and passive attention mechanism. If the user needs to output brain control commands, he should focus on the EEG stimulation interface in the stimulating eye. At this time, the EEG stimulation interface will also move in the direction close to the eye, the user's EEG induced characteristics will be more obvious, and the EEG classification accuracy will be higher.

[0094] In a third aspect, the present invention provides a brain-computer interaction method. Before online interaction, based on the brain-computer interaction paradigm provided in the first aspect, an offline EEG experiment is performed to obtain n types of control-state EEG features induced when a user gazes at n types of EEG stimulation in an EEG stimulation interface and idle-state EEG features induced when the user gazes at an environmental video, and the n types of control-state EEG features corresponding to the n types of EEG stimulation and the idle-state EEG features corresponding to the environmental video are saved as a user EEG classification template.

[0095] During online interaction, the brain-computer interaction method has an observation mode and a control mode, and the user can freely choose the observation mode and the control mode according to whether there is a control demand. Specifically, the switching between the observation mode and the control mode is performed based on the user's eye movements. As a possible implementation method, the user's eye movements can be obtained using eye sensors, that is, the movement of the muscles around the user's eyes can be obtained through image technology, such as blinking, closing eyes, opening eyes, looking in different directions, etc. Exemplarily, the user's eye movement for mode switching is to blink a preset number of times continuously within a preset time, for example, blinking 3 times continuously within 2 seconds. In the observation mode, the user performs real-time monitoring and judgment based on the multi-dimensional perspective video of the controlled object. When the user pays attention to the key information and needs to output the brain control command, he can choose to switch to the control mode; in the control mode, the user performs asynchronous brain control command output based on the brain-computer interaction paradigm provided in the first aspect to realize time-sharing control of the controlled object. When the control operation ends without control intention, the user can also choose to switch to the observation mode.

[0096] The present invention makes full use of the posture, eye movement, image and other sensor devices of the XR device, and has observation mode and control mode, creating a brain-computer interaction experience with better flexibility, higher freedom of choice and greater fun for users, reducing user fatigue, reducing the load of brain control tasks, and improving the accuracy of brain-computer control decision output.

[0097] See also Figure 7 , brain-computer interaction methods include:

[0098] When switching to observation mode, perform the following operations:

[0099] See also Figure 8 , obtaining a first-person perspective video and a third-person perspective video of the controlled object, and synchronously tiling the first-person perspective video and the third-person perspective video from left to right to the main canvas based on the brain-computer interaction paradigm implementation method provided in the second aspect;

[0100] See also Figure 7 As a possible implementation method, the above operations include:

[0101] S30. Obtaining the first-person perspective video and the third-person perspective video of the controlled object;

[0102] As an example, the first-person perspective video of the controlled object is obtained through the Real Time Streaming Protocol (RTSP). The third-person perspective video is obtained by calling the front camera of the XR display device. Here, the controlled object is assumed to be in the user's foreground perspective. At the same time, turning on the front camera of the XR device also makes it easier for the user to observe the surrounding environment.

[0103] S31. Set the main camera not to move with the user's head posture, and set the horizontal pixel range of the main canvas to be equal to the sum of the horizontal pixel ranges of the first-person perspective video and the third-person perspective video, and the vertical pixel range to be equal to the larger range of the vertical pixel ranges of the first-person perspective video and the third-person perspective video;

[0104] As an example, read the pixel resolution X1*Y1 of the first-person perspective video and the pixel resolution X2*Y2 of the third-person perspective video. Set the horizontal pixel range X of the main canvas m =X1+X2. Set the vertical pixel range Y of the main canvas m =max(Y1,Y2).

[0105] S32. The first-person perspective video and the third-person perspective video are synchronously tiled and presented on the main canvas from left to right, and the horizontal pixel ranges of the first-person perspective video and the third-person perspective video do not overlap.

[0106] As an example, since the horizontal pixel range X of the main canvas has been set in S31 m =X1+X2. Set the vertical pixel range Y of the main canvas m =max(Y1, Y2), therefore, when the first-person perspective video and the third-person perspective video are tiled to the main canvas, the horizontal pixel ranges of the two perspective videos are continuous and non-overlapping.

[0107] The brain-computer interaction method provided by the present invention can realize the simultaneous presentation of first-person perspective video and third-person perspective video of the controlled object, solving the problems of single user perspective and limited field of view under the traditional interaction paradigm.

[0108] See also Figure 7 , when switched to the control mode, the brain-computer interaction method includes the following sub-steps:

[0109] S20. Determine the control video;

[0110] As a possible implementation manner, determining the control video includes:

[0111] S200. Make the user keep his sight perpendicular to the plane where the eyes are located, obtain the range of the yaw angle of the head posture when the user deflects from the leftmost end of the main canvas to the rightmost end of the main canvas, and establish a field of view mapping relationship between the horizontal pixel coordinates of the user's direct attention point and the yaw angle of the head posture;

[0112] As an example, the posture sensor of the XR device is used to obtain the change range of the head posture yaw angle θ when the user deflects from the leftmost end of the main canvas to the rightmost end of the main canvas [θ min ,θ max ], where θ min Indicates the yaw angle of the user's head posture when looking at the leftmost end of the main canvas, θ max Indicates the yaw angle of the user's head posture when looking at the rightmost end of the main canvas. Based on the horizontal pixel range of the main canvas (0, X m ], the field of view mapping relationship between the horizontal pixel coordinate Hor_M of the user's attention point and the head posture yaw angle θ is established as follows:

[0113]

[0114] S201. Determine the switching moment from observation mode to control mode based on the field of view mapping relationship, and the horizontal pixel coordinate value of the user's direct attention point corresponding to the yaw angle of the user's head posture. If the horizontal pixel coordinate value belongs to the horizontal pixel range of the first-person perspective video, then determine that the control video is a first-person perspective video; if the horizontal pixel coordinate value belongs to the horizontal pixel range of the third-person perspective video, then determine that the control video is a third-person perspective video.

[0115] As an example, the posture sensor obtains the yaw angle of the user's head posture at the moment of switching from the observation mode to the control mode, and calculates the horizontal pixel coordinate value of the user's direct attention point based on the mapping relationship determined in S200. Since the horizontal pixel ranges of the two perspective videos are continuous and non-overlapping, the user's direct attention point can only fall into one of the first-person perspective video and the third-person perspective video. If it falls into the first-person perspective video area, the first-person perspective video is used as the control video; if it falls into the third-person perspective video area, the third-person perspective video is used as the control video.

[0116] S21. Based on the brain-computer interaction paradigm provided in the first aspect, the EEG stimulation interface and the control video are presented to the user's stimulation attention eye and video attention eye respectively;

[0117] See also Fig. 9 The EEG stimulation interface and control video are presented to the user's stimulation attention eye and video attention eye respectively. At this time, a transparent, penetrating, unobstructed and uncovered visual viewing experience of the EEG stimulation interface and control video will appear in the user's field of vision.

[0118] S22. Obtain the user's active attention result in real time, and determine whether the user's active attention result is an electroencephalogram stimulation interface or a control video, specifically: obtain in real time the first horizontal force direction of the muscles around the eye of the user's stimulating attention eye and the second horizontal force direction of the muscles around the eye of the video attention eye; when the second horizontal force direction is toward the stimulating attention eye, and the first horizontal force direction is the same as the second horizontal force direction, determine that the user's active attention result is an electroencephalogram stimulation interface; otherwise, determine that the user's active attention result is a control video.

[0119] See also Fig.10 As an example, the horizontal force direction of the muscles around the user's attention eye and video eye is obtained through the eye sensor of the XR device, and the force deflection degree of the muscles around the eyes of both eyes is normalized from left to right. For example, the force deflection degree of the muscles around the user's attention eye is normalized to α∈[0,1], and the force deflection degree of the muscles around the user's video eye is normalized to β∈[0,1]. When the user focuses on the image in the field of view of one eye, the user's left and right eyes will maintain a certain degree of same-direction movement. Therefore, assuming that the left eye is the stimulating attention eye, only when α∈[0,0.5) and β∈[0,0.5) does it mean that both eyes move in the same direction and move to the left. At this time, the user's active attention result is the EEG stimulation interface. Assuming that the right eye is the stimulating attention eye, only when α∈(0.5,1] and β∈(0.5,1] does it mean that both eyes move in the same direction and move to the right. At this time, the user's active attention result is also the EEG stimulation interface. In other cases, the user's active attention result is judged as the control video.

[0120] If it is a brain electrical stimulation interface, execute S23 to S24; if it is a control video, execute S25;

[0121] S23. Move the EEG stimulation interface closer to the eyes along a direction perpendicular to the plane where the eyes are located, and move the control video away from the eyes along a direction perpendicular to the plane where the eyes are located;

[0122] As an example, based on the user's passive attention mechanism, the display canvas corresponding to the stimulated eye is dynamically moved. Assuming that the stimulated eye is the left eye, the left canvas presenting the EEG stimulation interface is translated along the Z-axis depth direction of the left camera space coordinate system, toward the eye; the right canvas presenting the control video is translated along the Z-axis depth direction of the right camera space coordinate system, away from the eye. The X-axis and Y-axis positions of the left and right canvases remain unchanged, and the centers of the left and right eye images are still kept overlapping, such as Figure 3 As shown in (a) and 3(b), the left eye is stimulated near and the right eye is far. Since the user's active attention result is the EEG stimulation interface, the actual visual effect after binocular competition is as follows Fig.11 As shown, the visual effect is that the EEG stimulation interface is in front and the control video is in the back, and the stimulation is a penetrable effect.

[0123] S24. Obtain the user's EEG characteristics in real time, match them with the user's EEG classification template, and determine whether the EEG characteristics belong to the n-type control state EEG characteristics or the idle state EEG characteristics; if they belong to the n-type control state EEG characteristics, determine the EEG stimulation category corresponding to the EEG characteristics, and output the control instructions corresponding to the EEG stimulation category; if they are idle state EEG characteristics, do not output the control instructions;

[0124] The brain-computer interaction method proposed in the present invention first performs an offline EEG experiment on the user before online interaction, presents the EEG stimulation interface and the environmental video to the user's stimulation attention eye and video attention eye respectively, obtains the n types of control state EEG features induced when the user looks at the n types of EEG stimulation in the EEG stimulation interface and the idle state EEG features induced when the user looks at the environmental video through the offline EEG experiment, and saves the n types of control state EEG features corresponding to the n types of EEG stimulation and the idle state EEG features corresponding to the environmental video as the user EEG classification template. In this way, the user's EEG features can be judged more accurately in subsequent real-time interaction, and the asynchronous output of brain control commands can be realized under environmental interference.

[0125] S25. Move the control video closer to the eyes along a direction perpendicular to the plane where the eyes are located, and move the electroencephalographic stimulation interface away from the eyes along a direction perpendicular to the plane where the eyes are located.

[0126] See also Fig.11When the user needs to observe the environment without outputting control commands, the user can focus on the control video behind through the front EEG stimulation interface. The EEG stimulation interface presents a transparent effect, reduces the visual effect of the stimulation block flickering, and weakens the result of EEG signal feature induction. The control video area will not be covered by the EEG stimulation interface, ensuring that the user has no missing observation of environmental information in the field of vision.

[0127] The brain-computer interaction method proposed in the present invention combines the multimodal perception function of the XR display device with the asynchronous interaction technology of BCI, and has a control mode and an observation mode. In the control mode, the user can freely select the stimulus interface or environmental information of interest, and freely output brain control commands according to their own control needs, which can solve the problem that brain control commands are randomly output when the user has no control intention and does not look at the visual stimulus. In the observation mode, the first-person perspective video and the third-person perspective video of the controlled object can be presented simultaneously, solving the problems that the current XR-BCI system cannot simultaneously use multi-dimensional perspectives to present the environmental information of the controlled object, it is difficult to highlight the user's area of ​​interest, and the redundant error output of the control decision, etc., which minimizes user misoperation and realizes high-performance immersive brain-computer interaction in complex scenes.

[0128] In a fourth aspect, the present invention provides a brain-computer interaction system for executing the brain-computer interaction method provided in the third aspect, see Fig.12 , the brain-computer interaction system includes:

[0129] The paradigm presentation module runs the brain-computer interaction paradigm and presents the EEG stimulation interface and the control video to the user's stimulation attention eye and video attention eye respectively;

[0130] As an example, the paradigm presentation module is developed based on the Unity3D platform and C# language, runs through the stimulation host, and is presented based on the extended reality display device (the display device in the present invention is a left-eye split-screen head-mounted XR display device with gesture sensing, eye tracking and a front camera, and the display device is connected to the stimulation host). The display device model used in this example is HTC VIVE Pro Eye, with a maximum field of view of 110° and a refresh rate of 90Hz. Each left eye is equipped with an OLED display with a resolution of 1440*1600, and a binocular resolution of 2880*1600; the stimulation host connected to the display device is a DELL workstation, the CPU model is Intel Xeon 6226R Golden CPU@3.9GHz*2, and the graphics card model is NVIDIA GeForce RTX3090.

[0131] The EEG acquisition module acquires the user's EEG signals in real time; as an example, the EEG acquisition module acquires EEG signals based on the Neuroscan SynAmps2 (64-lead) EEG acquisition system.

[0132] The signal processing module performs preprocessing, feature extraction and template matching on the collected EEG signals, and generates corresponding control instructions based on the extracted EEG features; as an example, the signal processing module is written in Python and runs on the processing host via the PyCharm platform. In this example, the processing host is another DELL workstation with an Intel i9-13900K CPU and an NVIDIA GeForce RTX 3060 graphics card.

[0133] The attention selection module obtains the user's active attention results in real time and determines whether it is necessary to output a control instruction. When a control instruction is required, the control instruction generated by the signal processing module is transmitted to the instruction sending module. As an example, the attention selection module is developed based on the Unity platform and C# language and runs via the stimulation host.

[0134] The instruction sending module sends the control instruction to the controlled object, so that the controlled object completes the action corresponding to the control instruction; as an example, the instruction sending module is implemented based on the Transmission Control Protocol (TCP), and can also be implemented based on UDP, WebSocket and other protocols, and the host computer runs on the stimulation host.

[0135] The environmental feedback module is used to display the environmental information of the controlled object to the user in an immersive way in real time in the form of video stream feedback; as an example, the environmental feedback module obtains the first-person video of the controlled object based on the Real Time Streaming Protocol (RTSP), and obtains the user's third-person video by calling the front camera of the XR display device. The video acquisition and display method is based on the Unity platform and developed in C# language, and runs through the stimulation host.

[0136] And a mode switching module switches between the control mode and the observation mode based on the user's eye movements; as an example, the mode switching module is developed based on the Unity platform and C# language and runs via the stimulation host.

[0137] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art may understand and implement other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the drawings, etc. In the specification, the word "comprising" does not exclude other components or steps, and "one" or "an" does not exclude multiple situations. A single processor or other unit can implement several functions listed in the specification. Certain measures are recorded in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0138] Although the present invention has been described in conjunction with specific features and embodiments thereof, it is apparent that various modifications and combinations may be made thereto without departing from the spirit and scope of the present invention. Accordingly, this specification and the accompanying drawings are merely exemplary illustrations of the present invention and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present invention. Obviously, those skilled in the art may make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, the present invention is intended to include such modifications and variations if they fall within the scope of the present invention and its equivalents.

Claims

1. A brain-computer interaction paradigm, characterized in that: The electroencephalogram (EEG) stimulation interface is presented to the user's stimulation attention eye, and the environmental video is presented to the user's video attention eye at the same time; the stimulation attention eye is the eye whose monocular EEG stimulation classification accuracy is closest to the binocular EEG stimulation classification accuracy, and the other eye is the video attention eye; or, when the left eye EEG stimulation classification accuracy is equal to the right eye EEG stimulation classification accuracy, the stimulation attention eye is the user's physiological dominant eye, and the other eye is the video attention eye; the EEG stimulation interface and the environmental video are switched between near and far based on the user's active and passive attention selection mechanism.

2. A method for implementing a brain-computer interaction paradigm, characterized in that: For realizing the brain-computer interaction paradigm of claim 1, the method for realizing the brain-computer interaction paradigm comprises: S10. Constructing a rendering camera and a display canvas, wherein the rendering camera includes a left camera, a right camera and a main camera, and the display canvas includes a left canvas, a right canvas and a main canvas; wherein the left canvas is mounted on the left camera and is displayed independently only to the left eye of the user; the right canvas is mounted on the right camera and is displayed independently only to the right eye of the user; and the main canvas is mounted on the main camera and is displayed synchronously to both eyes of the user; S11. Writing an EEG stimulation interface based on the control instructions required by the controlled object, and rendering the EEG stimulation interface onto the left canvas, the right canvas, and the main canvas; S12. Perform an offline EEG classification experiment, and determine the single eye with a higher EEG classification accuracy among the left and right eyes of the user according to the EEG induced results of the user by the EEG stimulation interface, and mark it as the stimulation attention eye, and the other eye is marked as the video attention eye; S13. presenting the electroencephalographic stimulation interface to the stimulation attention eye, and presenting the environmental video to the video attention eye; S14. When the user actively focuses on the electroencephalographic stimulation interface, the electroencephalographic stimulation interface approaches the eyes in a direction perpendicular to the plane where the eyes are located, and the environmental video moves away from the eyes in a direction perpendicular to the plane where the eyes are located; when the user actively focuses on the environmental video, the environmental video approaches the eyes in a direction perpendicular to the plane where the eyes are located, and the electroencephalographic stimulation interface moves away from the eyes in a direction perpendicular to the plane where the eyes are located.

3. The method for implementing the brain-computer interaction paradigm according to claim 2, characterized in that: The S12 includes: S120. When the EEG stimulation interface is only presented on the left canvas, when the EEG stimulation interface is only presented on the right canvas, and when the EEG stimulation interface is only presented on the main canvas, respectively perform an EEG classification offline experiment to obtain the user's left eye EEG classification accuracy, right eye EEG classification accuracy, and binocular EEG classification accuracy; S121. Calculate the absolute value of the first difference between the left eye EEG classification accuracy and the binocular EEG classification accuracy, and calculate the absolute value of the second difference between the right eye EEG classification accuracy and the binocular EEG classification accuracy; mark the single eye corresponding to the smaller value of the absolute value of the first difference and the absolute value of the second difference as the stimulation attention eye, and mark the other eye as the video attention eye; if the absolute value of the first difference is equal to the absolute value of the second difference, then the user's physiological dominant eye is used as the stimulation attention eye, and the other eye is used as the video attention eye.

4. The method for implementing the brain-computer interaction paradigm according to claim 2, characterized in that: The S13 includes: S130. Unify the spatial coordinate systems of the left camera and the right camera, and set the left camera and the right camera to move with the user's head posture; S131. Mount the left canvas at the origin of the space coordinate system where the left camera is located, and mount the right canvas at the origin of the space coordinate system where the right camera is located; S132. If the left eye is the eye that is being stimulated and focused, the left camera will present the electroencephalographic stimulation interface, and the right camera will present the environmental video; if the right eye is the eye that is being stimulated and focused, the right camera will present the electroencephalographic stimulation interface, and the left camera will present the environmental video.

5. A brain-computer interaction method, characterized in that: Before online interaction, based on the brain-computer interaction paradigm described in claim 1, obtain n types of control-state EEG features induced when the user looks at n types of EEG stimulation in the EEG stimulation interface and idle-state EEG features induced when the user looks at the environmental video through an offline EEG experiment, and save the n types of control-state EEG features corresponding to the n types of EEG stimulation and the idle-state EEG features corresponding to the environmental video as a user EEG classification template; During online interaction, the brain-computer interaction method has an observation mode and a control mode, and switching between the observation mode and the control mode is performed based on the user's eye movements. The brain-computer interaction method includes: When switching to observation mode, perform the following operations: Obtain a first-person perspective video and a third-person perspective video of the controlled object, and based on the brain-computer interaction paradigm implementation method described in claim 2, synchronously tile the first-person perspective video and the third-person perspective video from left to right on the main canvas; When switching to the control mode, the brain-computer interaction method includes the following sub-steps: S20. Determine the control video; S21. Based on the brain-computer interaction paradigm of claim 1, presenting the EEG stimulation interface and the control video to the user's stimulation attention eye and video attention eye respectively; S22. Obtain the user's active attention result in real time, and determine whether the user's active attention result is a brain electrical stimulation interface or a control video. If it is a brain electrical stimulation interface, execute S23 to S24; if it is a control video, execute S25; S23. Move the electroencephalographic stimulation interface closer to the eyes in a direction perpendicular to the plane where the eyes are located, and move the control video away from the eyes in a direction perpendicular to the plane where the eyes are located; S24. Acquire the user's EEG characteristics in real time, match them with the user's EEG classification template, and determine whether the EEG characteristics belong to the n-type control state EEG characteristics or the idle state EEG characteristics; if they belong to the n-type control state EEG characteristics, determine the EEG stimulation category corresponding to the EEG characteristics, and output the control instructions corresponding to the EEG stimulation category; if they are idle state EEG characteristics, do not output the control instructions; S25. Move the control video closer to the eyes along a direction perpendicular to the plane where the eyes are located, and move the electroencephalographic stimulation interface away from the eyes along a direction perpendicular to the plane where the eyes are located.

6. The brain-computer interaction method according to claim 5, characterized in that: Obtaining a first-person perspective video and a third-person perspective video of the controlled object, and synchronously tiling the first-person perspective video and the third-person perspective video from left to right to the main canvas based on the brain-computer interaction paradigm implementation method of claim 2, including: S30. Obtaining the first-person perspective video and the third-person perspective video of the controlled object; S31. Set the main camera not to move with the user's head posture, and set the horizontal pixel range of the main canvas to be equal to the sum of the horizontal pixel ranges of the first-person perspective video and the third-person perspective video, and the vertical pixel range to be equal to the larger range of the vertical pixel ranges of the first-person perspective video and the third-person perspective video; S32. The first-person perspective video and the third-person perspective video are synchronously tiled and presented on the main canvas from left to right, and the horizontal pixel ranges of the first-person perspective video and the third-person perspective video do not overlap.

7. The brain-computer interaction method according to claim 5, characterized in that: The determining of the control video includes: S200. Make the user keep his sight perpendicular to the plane where the eyes are located, obtain the range of the yaw angle of the head posture when the user deflects from the leftmost end of the main canvas to the rightmost end of the main canvas, and establish a field of view mapping relationship between the horizontal pixel coordinates of the user's direct attention point and the yaw angle of the head posture; S201. Determine the switching moment from the observation mode to the control mode based on the field of view mapping relationship, and the horizontal pixel coordinate value of the user's direct attention point corresponding to the yaw angle of the user's head posture. If the horizontal pixel coordinate value belongs to the horizontal pixel range of the first-person perspective video, then determine that the control video is the first-person perspective video; if the horizontal pixel coordinate value belongs to the horizontal pixel range of the third-person perspective video, then determine that the control video is the third-person perspective video.

8. The brain-computer interaction method according to claim 5, characterized in that: The user's active attention result is obtained in real time, and it is determined whether the user's active attention result is a brain electrical stimulation interface or a control video. Specifically, the first horizontal force direction of the muscles around the eye of the user's stimulating attention eye and the second horizontal force direction of the muscles around the eye of the video attention eye are obtained in real time. When the second horizontal force direction is toward the stimulating attention eye, and the first horizontal force direction is the same as the second horizontal force direction, it is determined that the user's active attention result is a brain electrical stimulation interface. Otherwise, it is determined that the user's active attention result is a control video.

9. The brain-computer interaction method according to claim 5, characterized in that: The user's eye movement is blinking a preset number of times continuously within a preset time.

10. A brain-computer interaction system, characterized in that: Used to execute the brain-computer interaction method according to any one of claims 5 to 9, the brain-computer interaction system comprises: The paradigm presentation module runs the brain-computer interaction paradigm and presents the EEG stimulation interface and the control video to the user's stimulation attention eye and video attention eye respectively; EEG acquisition module, collects the user's EEG signals in real time; The signal processing module performs preprocessing, feature extraction and template matching on the collected EEG signals, and generates corresponding control instructions based on the extracted EEG features; The attention selection module obtains the user's active attention results in real time, determines whether a control instruction needs to be output, and transmits the control instruction generated by the signal processing module to the instruction sending module when a control instruction needs to be output; An instruction sending module sends the control instruction to the controlled object so that the controlled object completes the action corresponding to the control instruction; The environmental feedback module transmits the environmental information of the controlled object to the user in real time in the form of video stream for immersive display; and a mode switching module for switching between control mode and observation mode based on the user's eye movements.

Images