Virtual reality interaction method based on brain-computer interface and eye movement technology and edge device

Abstract

The invention discloses a virtual reality interaction method based on a brain-computer interface and an eye movement technology and edge equipment. The virtual reality interaction method based on the brain-computer interface and the eye movement technology comprises the steps that a virtual reality scene is output, and the virtual reality scene comprises at least one virtual object; obtaining eye movement data of interaction between the user and the virtual reality scene, and determining a target object interacted by the user from the at least one virtual object based on the eye movement data; acquiring electroencephalogram data associated with the eye movement data, and determining a control intention of the user on the target object based on the electroencephalogram data; and executing a control operation based on the control intention.

Description

Technical Field

[0001] The present invention relates to the technical fields of human intelligence, virtual reality, etc., and in particular to a virtual reality interaction method and edge device based on brain-computer interface and eye movement technology. Background Art

[0002] Virtual reality interaction technology (VR) mainly achieves immersive experience through head-mounted devices, handles, motion capture devices and other devices. However, the virtual reality interaction method of related technologies is not simple enough, requiring users to perform complex interaction actions, and cannot well understand the user's interaction intentions. Summary of the invention

[0003] The embodiments of the present application are intended to solve at least one of the technical problems in the related art to a certain extent. To this end, the purpose of the embodiments of the present application is to propose a virtual reality interaction method, system, device, edge device, electronic device, storage medium and program product based on brain-computer interface and eye movement technology.

[0004] An embodiment of the present application provides a virtual reality interaction method based on a brain-computer interface and eye movement technology, the method comprising: outputting a virtual reality scene, wherein the virtual reality scene includes at least one virtual object; obtaining eye movement data of a user interacting with the virtual reality scene, and determining a target object with which the user interacts from the at least one virtual object based on the eye movement data; obtaining electroencephalogram (EEG) data associated with the eye movement data, and determining the user's control intention for the target object based on the EEG data; and performing a control operation based on the control intention.

[0005] Exemplarily, the control operation includes: a control operation for executing on the target object.

[0006] Exemplarily, the control operation is performed based on the control intention, including: when the control intention indicates that the user wants to view the target object, controlling the external device to turn toward the target object; or when the control intention indicates that the user wants to reach the location of the target object, controlling the external device to move with the target object as the end point.

[0007] Exemplarily, when the virtual reality scene is a smart home control scene, the target object is a smart device virtual icon, and the eye movement data indicates that the user is looking at the smart device virtual icon; and performing the control operation based on the control intention includes: when the control intention indicates that the user wants to control the smart device corresponding to the smart device virtual icon, controlling the startup state of the smart device.

[0008] Exemplarily, when the virtual reality scene is an assisted communication scene, the target object is a control on a virtual operation interface, and the eye movement data indicates that the user is gazing at the control on the virtual operation interface; performing a control operation based on the control intention includes: when the control intention indicates that the user wants to input information through the controls on the virtual operation interface, controlling the controls on the virtual operation interface to be in a selected confirmed state or controlling the sending of commands through the controls on the virtual operation interface.

[0009] Exemplarily, when the virtual reality scene is a medical rehabilitation scene, the target object is rehabilitation training content, the eye movement data indicates that the user selects rehabilitation training content or performs training based on rehabilitation training content, and the EEG data indicates the user's degree of relaxation with respect to the rehabilitation training content; and based on the control intention, performing a control operation includes: when the control intention indicates that the user is dissatisfied with the training difficulty of the rehabilitation training content, controlling the adjustment of the training difficulty of the rehabilitation training content.

[0010] Exemplarily, when the virtual reality scene is a virtual expo navigation scene, the target object is a virtual exhibit, and the eye movement data indicates that the user selects the virtual exhibit; performing a control operation based on the control intention includes: when the control intention indicates that the user wants to learn about the virtual exhibit, controlling the output of description information of the virtual exhibit.

[0011] Exemplarily, the control operation includes: a control operation for switching virtual reality scenes.

[0012] Exemplarily, the EEG data represents the user's emotional response to the target object; and performing the control operation based on the control intention includes: controlling the switching of the virtual reality scene when the control intention represents that the user's emotional response to the target object is abnormal.

[0013] Exemplarily, determining the user's control intention over the target object based on the EEG data includes: determining, based on the EEG data, whether EEG fluctuations are induced during the user's interaction with the target object through eye movement interaction; if it is determined that EEG fluctuations are induced, determining that the user has a control intention over the target object.

[0014] Another embodiment of the present application provides a virtual reality interaction system based on brain-computer interface and eye movement technology, the system comprising: a data acquisition device for collecting eye movement data and EEG data of the user; a virtual reality interaction device for executing the steps of the method described in any of the above embodiments.

[0015] Another embodiment of the present application provides a virtual reality interaction device based on brain-computer interface and eye movement technology, the device comprising: an output module, used to output a virtual reality scene, wherein the virtual reality scene includes at least one virtual object; a first determination module, used to obtain eye movement data of a user interacting with the virtual reality scene, and determine a target object with which the user interacts from the at least one virtual object based on the eye movement data; a second determination module, used to obtain electroencephalogram (EEG) data associated with the eye movement data, and determine the user's control intention over the target object based on the EEG data; and a control module, used to perform a control operation based on the control intention.

[0016] Another embodiment of the present application provides an edge device, which is used to perform edge computing, and the edge device can execute the above-mentioned virtual reality interaction method, or the edge device is integrated with the above-mentioned virtual reality interaction system, or the edge device is integrated with the above-mentioned virtual reality interaction device.

[0017] Another embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method described in any one of the above embodiments are implemented.

[0018] Another embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method described in any one of the above embodiments are implemented.

[0019] Another embodiment of the present application provides a computer program product, which includes instructions. When the instructions are executed by a processor of a computer device, the computer device can perform the steps of the method described in any of the above embodiments.

[0020] In the above implementation, a virtual reality scene is output, wherein the virtual reality scene includes at least one virtual object; eye movement data of the user interacting with the virtual reality scene is obtained, and a target object with which the user interacts is determined from the at least one virtual object based on the eye movement data; EEG data associated with the eye movement data is obtained, and the user's control intention for the target object is determined based on the EEG data; and a control operation is performed based on the control intention. The virtual reality interaction method based on brain-computer interface and eye movement technology of the present invention can combine the user's eye movement data and EEG data for virtual interaction, has a simple interaction method, does not require the user to perform complex interaction actions, can well understand the user's interaction intention, and improves the accuracy of interaction control. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1A flowchart of a virtual reality interaction method based on brain-computer interface and eye movement technology provided in an embodiment of the present application;

[0022] Figure 2 A flow chart of a control intention determination method provided in an embodiment of the present application;

[0023] Figure 3 A schematic diagram of a virtual reality interaction system based on brain-computer interface and eye movement technology provided in an embodiment of the present application;

[0024] Figure 4 A schematic diagram of a virtual reality interaction device based on a brain-computer interface and eye movement technology provided in an embodiment of the present application;

[0025] Figure 5 A block diagram of an electronic device provided for an embodiment of the present application. DETAILED DESCRIPTION

[0026] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0027] Virtual reality interaction technology (VR) mainly achieves immersive experience through head-mounted devices, handles, motion capture devices and other devices. However, the virtual reality interaction method of related technologies is not simple enough, requiring users to perform complex interaction actions, and cannot well understand the user's interaction intentions.

[0028] In view of this, the embodiments of the present application provide an optimized virtual reality interaction method based on brain-computer interface and eye movement technology, which can combine the user's eye movement data and EEG data to perform virtual interaction and interaction and synchronization between virtual and reality. The interaction method is simple and does not require the user to perform complex interaction actions. It can well understand the user's interaction intentions and improve the accuracy of interaction control.

[0029] Figure 1 A flowchart of a virtual reality interaction method based on brain-computer interface and eye movement technology provided in an embodiment of the present application.

[0030] like Figure 1 As shown, the virtual reality interaction method 100 based on brain-computer interface and eye movement technology provided in an embodiment of the present application includes steps S110-S140.

[0031] Step S110: output a virtual reality scene, where the virtual reality scene includes at least one virtual object.

[0032] Based on mixed reality technology, virtual scenes are superimposed on real-world scenes. Users can see both virtual scenes and real scenes through a virtual reality eye tracker. Users use eye movement technology to track changes in virtual scenes in real time, while extracting and decoding feature values ​​of acquired EEG data, converting the extracted features into recognizable command signals, and then precisely controlling target objects in the real world. This control can be intuitively observed in real scenes, realizing the interaction and synchronization between virtual and reality.

[0033] Step S120, acquiring eye movement data of the user interacting with the virtual reality scene, and determining a target object with which the user interacts from at least one virtual object based on the eye movement data.

[0034] Step S130, acquiring EEG data associated with the eye movement data, and determining the user's control intention on the target object based on the EEG data.

[0035] Step S140: executing a control operation based on the control intention.

[0036] Exemplarily, a user may interact in a virtual reality scene, such as performing interactive operations on virtual objects in the virtual reality scene. The virtual reality scene may include multiple virtual objects, which may, for example, receive interactive operations and respond to interactive operations. For example, a user may interact with a virtual object through eye movement technology, collect interactive eye movement data, analyze the eye movement data to determine the target object with which the user interacts, such as analyzing the eye movement data to obtain analysis results such as the user's gaze information, gaze duration, and gaze trajectory, and determine the target object based on the analysis results.

[0037] When the user performs eye movement interaction, the associated EEG data can also be obtained. The associated EEG data indicates that the EEG data changes when the eye movement interaction is performed. After obtaining the EEG data, the EEG data is subjected to signal processing and feature value extraction, and the extracted features are converted into recognizable command signals to understand the user's control intention. For example, the user tracks the stimulus or task presented in the virtual reality scene through a virtual reality eye tracker, and controls the movement of the virtual scene or real device (which can be an external device referred to elsewhere in the text) through the EEG data feature extraction results, such as thresholds, emotions, stress states, or thoughts. Interaction in the virtual reality scene is achieved based on the combination of eye movement data and EEG data. At the same time, the scene superposition based on augmented reality can be used for interaction and synchronization between virtual and reality.

[0038] The embodiments of the present application are virtual reality interaction methods based on brain-computer interface and eye movement technology, which can combine the user's eye movement data and EEG data for virtual interaction. The interaction method is simple and does not require the user to perform complex interaction actions. It can well understand the user's interaction intentions and improve the accuracy of interaction control.

[0039] Figure 2 A flowchart of a method for determining control intention provided in an embodiment of the present application.

[0040] like Figure 2 As shown, the control intention determination method provided in the embodiment of the present application includes, for example, steps S210-S220.

[0041] Step S210: determining, based on the EEG data, whether EEG fluctuations are induced in the process of the user interacting with the target object through eye movement interaction.

[0042] Step S220: If it is determined that EEG fluctuations are induced, it is determined that the user has the intention to control the target object.

[0043] For example, by analyzing the EEG data, it can be found out whether EEG fluctuations are induced when the user interacts with the target object through eye movement interaction. If the degree of EEG fluctuation is large or exceeds a certain degree, it is considered that EEG fluctuations are successfully induced. At this time, it can be determined that the user has the intention to control the target object.

[0044] For example, in a virtual reality scene, users can interact with the target object through eye movement, which can bring specific visual stimulation to the user. At this time, the occipital lobe area produces a corresponding EEG response, which is called Steady-State Visual Evoked Potential (SSVEP). Steady-state visual evoked potential (SSVEP) means that when a fixed frequency of visual stimulation is received, the visual cortex of the human brain will produce a continuous response related to the stimulation frequency. Interaction is achieved by decoding and controlling the collected EEG signals, so that users can interact accurately in virtual reality scenes through eye movement interaction and EEG interaction, without the need for users to perform too many interactive operations, and users who are inconvenient to perform physical interaction can also interact.

[0045] In another example, the control operation includes: a control operation for performing on a target object. For example, in various virtual reality scenarios such as travel scenarios, smart home control scenarios, assisted communication scenarios, virtual exhibition navigation scenarios, medical rehabilitation scenarios, etc., the control operation can be an operation performed on a target object.

[0046] For example, in a travel scene, the target object may be some objects or scenery in the travel scene. For users with motor impairments, the external device may be a wheelchair that the user is riding. When the user uses eye movement technology to look at or watch a target object, the external device can be controlled based on EEG.

[0047] For example, when the control intention obtained by parsing the EEG data indicates that the user wants to watch the target object, the external device is controlled to turn toward the target object. Alternatively, when the control intention indicates that the user wants to reach the location of the target object, the external device is controlled to move with the target object as the end point. At the same time, the real world location after the movement can be seen based on the eye movement device. It can be seen that the interaction is achieved by navigating or selecting the corresponding target object through eye movement technology, and then controlling the movement of the external device based on the brain-computer interface (Brain Computer Interface, BCI) technology.

[0048] For example, when the virtual reality scene is a smart home control scene, the target object is a virtual icon of a smart device, and the eye movement data indicates that the user is looking at the virtual icon of the smart device. Based on the control intention, the control operation is performed, including: when the control intention indicates that the user wants to control the smart device corresponding to the virtual icon of the smart device, controlling the startup state of the smart device.

[0049] For example, a user can look at the icon on the control interface to select the smart device they want to control, and then activate or shut down the device through a specific brain wave pattern. For example, the smart device includes a lamp, and the control interface includes an icon for adjusting the brightness of the light. The user looks at the icon and thinks about adjusting the brightness of the light in his mind, thereby adjusting the brightness of the light based on the steady-state visual evoked potential (SSVEP) technology. For example, the control interface may also include an icon for turning on or off the lamp. The user looks at the icon and thinks about turning the lamp on or off in his mind, and controls the turning on and off of the lamp by analyzing the EEG data.

[0050] Exemplarily, in a case where the virtual reality scene is an assisted communication scene, the target object is a control on a virtual operation interface, and the eye movement data indicates that the user is looking at the control on the virtual operation interface; based on the control intention, a control operation is performed, including: when the control intention indicates that the user wants to input information through the controls on the virtual operation interface, controlling the controls on the virtual operation interface to be in a selected confirmed state or controlling the sending of commands through the controls on the virtual operation interface.

[0051] For example, on an assisted communication platform, a virtual operation interface may include a virtual keyboard, and controls may be keys on the keyboard. The user may select a corresponding key by gazing at the keys on the virtual keyboard to input text. After selecting a key through eye movement, the brain controls the imagination and generates a specific EEG signal. The specific EEG signal is analyzed to know that the user wants to press the key, and the key is confirmed and selected and a command is sent to input text through the key.

[0052] Exemplarily, when the virtual reality scene is a medical rehabilitation scene, the target object is rehabilitation training content, the eye movement data indicates that the user selects rehabilitation training content or conducts training based on rehabilitation training content, and the EEG data indicates the user's degree of relaxation with respect to the rehabilitation training content; based on the control intention, a control operation is performed, including: when the control intention indicates that the user is dissatisfied with the training difficulty of the rehabilitation training content, controlling the adjustment of the training difficulty of the rehabilitation training content.

[0053] For example, during the medical rehabilitation process, patients can select rehabilitation training content through eye movement interaction, or perform training activities based on the selected rehabilitation training content through eye movement interaction, and monitor the patient's attention and relaxation state by collecting EEG data. If the EEG data analysis shows that the user is not satisfied with the difficulty of the rehabilitation training content, such as the user is too nervous or too focused, it means that the training is difficult; if the user is too relaxed or distracted, it may mean that the training is less difficult. At this time, the difficulty of the rehabilitation training in the medical rehabilitation scenario can be adjusted.

[0054] Exemplarily, when the virtual reality scene is a virtual exhibition navigation scene, the target object is a virtual exhibit, and the eye movement data indicates that the user selects the virtual exhibit; based on the control intention, a control operation is performed, including: when the control intention indicates that the user wants to learn about the virtual exhibit, control the output of description information of the virtual exhibit.

[0055] For example, eye movement technology is used to capture the exhibits selected by the user through gaze, and EEG data is collected and analyzed to activate and output detailed information of the exhibits through brain control, such as playing the audio commentary of the exhibits or displaying the text information of the exhibits.

[0056] In another example, the control operation includes: a control operation for switching virtual reality scenes and / or real devices.

[0057] Exemplarily, the EEG data represents the user's emotional response to the target object; based on the control intention, a control operation is performed, including: when the control intention indicates that the user's emotional response to the target object is abnormal, controlling the switching of the virtual reality scene.

[0058] For example, when a user looks at a target object through eye movements, the EEG data indicates that the user's emotional response to the target object is too intense or nervous, which means that the user is not relaxed enough in the current virtual reality scene. The user can imagine switching the virtual reality scene in his brain. After collecting the EEG data, the user's control intention can be obtained by analyzing the EEG data, and the virtual reality scene can be switched based on the control intention, so as to switch to a virtual reality scene that makes the user more relaxed.

[0059] The embodiments of the present application integrate brain-computer interface and eye-control interaction technology in a virtual reality scenario, and control parameters can be customized, such as custom eye movement control parameters, for example, gazing for 3S is considered as selection, blinking twice in a row is considered as exit, etc. The commands of the brain-computer interface BCI can also be customized. For example, when the control intention corresponds to a certain emotional label (relaxation, tension, excitement), the classification label of the control operation can include operation labels such as left, right, and selection. By real-time feedback of the interaction process and adaptive adjustment of eye movement control parameters and brain-computer interface commands, interactive control based on multimodal signal fusion and decoding can be achieved.

[0060] Figure 3 Schematic diagram of a virtual reality interaction system based on brain-computer interface and eye movement technology provided in an embodiment of the present application.

[0061] like Figure 3 As shown, the virtual reality interaction system 300 based on brain-computer interface and eye movement technology provided in the embodiment of the present application includes a data acquisition device 310 and a virtual reality interaction device 320. The data acquisition device 310 is used to collect the user's eye movement data and EEG data, and the virtual reality interaction device 320 is used to execute the above-mentioned virtual reality interaction method based on brain-computer interface and eye movement technology. The virtual reality interaction device 320 may also include external equipment, such as a wheelchair, etc.

[0062] Figure 4 A schematic diagram of a virtual reality interaction device based on brain-computer interface and eye movement technology provided in an embodiment of the present application.

[0063] like Figure 4 As shown, the virtual reality interaction device 400 based on brain-computer interface and eye movement technology provided in the embodiment of the present application includes: an output module 410, a first determination module 420, a second determination module 430 and a control module 440.

[0064] The output module 410 is used to output a virtual reality scene, wherein the virtual reality scene includes at least one virtual object; based on mixed reality technology, the virtual scene is superimposed on the real world scene, and the user can see both the virtual scene and the real scene through a virtual reality eye tracker.

[0065] A first determination module 420 is used to obtain eye movement data of the user interacting with the virtual reality scene, and determine a target object interacted by the user from at least one virtual object based on the eye movement data;

[0066] A second determination module 430 is used to obtain EEG data associated with the eye movement data, and determine the user's control intention on the target object based on the EEG data;

[0067] The control module 440 is used to execute a control operation based on the control intention.

[0068] Exemplarily, the control operation includes: a control operation performed on a target object.

[0069] Exemplarily, the control module 440 is also used to: when the control intention indicates that the user wants to view the target object, control the external device to turn toward the target object; or when the control intention indicates that the user wants to reach the location of the target object, control the external device to move with the target object as the end point.

[0070] Exemplarily, when the virtual reality scene is a smart home control scene, the target object is a smart device virtual icon, and the eye movement data indicates that the user is looking at the smart device virtual icon; the control module 440 is also used to: when the control intention indicates that the user wants to control the smart device corresponding to the smart device virtual icon, control the startup state of the smart device.

[0071] Exemplarily, when the virtual reality scene is an assisted communication scene, the target object is a control on the virtual operation interface, and the eye movement data indicates that the user is looking at the control on the virtual operation interface; the control module 440 is also used to: when the control intention indicates that the user wants to input information through the controls on the virtual operation interface, control the controls on the virtual operation interface to be in a selected confirmed state or control the sending of commands through the controls on the virtual operation interface.

[0072] Exemplarily, when the virtual reality scene is a medical rehabilitation scene, the target object is rehabilitation training content, the eye movement data indicates that the user selects rehabilitation training content or conducts training based on rehabilitation training content, and the EEG data indicates the user's degree of relaxation with respect to the rehabilitation training content; the control module 440 is also used to: when the control intention indicates that the user is dissatisfied with the training difficulty of the rehabilitation training content, control the adjustment of the training difficulty of the rehabilitation training content.

[0073] Exemplarily, when the virtual reality scene is a virtual exhibition navigation scene, the target object is a virtual exhibit, and the eye movement data indicates that the user selects the virtual exhibit; the control module 440 is also used to: when the control intention indicates that the user wants to learn about the virtual exhibit, control the output of the description information of the virtual exhibit.

[0074] Exemplarily, the control operation includes: a control operation for switching virtual reality scenes.

[0075] Exemplarily, the EEG data represents the user's emotional response to the target object; the control module 440 is also used to control the switching of the virtual reality scene when the control intention indicates that the user's emotional response to the target object is abnormal.

[0076] Exemplarily, the second determination module 430 is also used to: determine, based on EEG data, whether EEG fluctuations are induced in the process of the user interacting with the target object through eye movement interaction; if it is determined that EEG fluctuations are induced, determine that the user has control intention over the target object.

[0077] It can be understood that the specific implementation process of the virtual reality interaction device 400 based on brain-computer interface and eye movement technology can refer to the implementation process of the virtual reality interaction method based on brain-computer interface and eye movement technology mentioned above, and will not be repeated here.

[0078] The embodiment of the present application provides an edge device, which is used to perform edge computing. The edge device can perform the virtual reality interaction method described above, or the edge device integrates the virtual reality interaction system described above, or the edge device integrates the virtual reality interaction device described above. Through edge computing, there is no need to send data to a remote data center or cloud for processing, which can reduce data transmission time and network delay caused by data transmission, and can also improve data security.

[0079] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method in any one of the above embodiments are implemented.

[0080] One embodiment of the present application provides a computer program product, which includes instructions. When the instructions are executed by a processor of a computer device, the computer device can perform the steps of the method of any of the above embodiments.

[0081] Figure 5 A block diagram of an electronic device provided for an embodiment of the present application.

[0082] An embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the above-mentioned virtual reality interaction method based on brain-computer interface and eye movement technology is implemented.

[0083] like Figure 5 As shown, for ease of understanding, the embodiment of the present application shows a specific electronic device 500.

[0084] The electronic device 500 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0085] like Figure 5 As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0086] Multiple components in the electronic device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the electronic device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0087] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods described above. For example, in some embodiments, any one or more of the various methods described above may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, any one or more of the various methods described above may be executed. One or more steps. Alternatively, in other embodiments, the computing unit 501 may be configured to perform any one or more of the various methods described above in any other appropriate manner (e.g., by means of firmware).

[0088] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this application, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0089] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0090] In the description of the present application, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0091] In the description of the present application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0092] In addition, the terms "first", "second", etc. used in the embodiments of the present application are only used for descriptive purposes and should not be understood as indicating or implying relative importance, or implicitly indicating the number of technical features indicated in the present embodiment. Therefore, the features defined by the terms "first", "second", etc. in the embodiments of the present application can explicitly or implicitly indicate that at least one of the features is included in the embodiment. In the description of the present application, the word "multiple" means at least two or two or more, such as two, three, four, etc., unless otherwise clearly and specifically defined in the embodiments.

[0093] In this application, unless otherwise clearly specified or limited in the embodiments, the terms "installed", "connected", "connected" and "fixed" etc. appearing in the embodiments should be understood in a broad sense. For example, the connection can be a fixed connection, a detachable connection, or an integrated connection. It can be understood that it can also be a mechanical connection, an electrical connection, etc.; of course, it can also be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal connection of two elements, or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to the specific implementation situation.

[0094] In the present application, unless otherwise clearly specified and limited, a first feature being “above” or “below” a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being “above”, “above”, and “above” a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being “below”, “below”, and “below” a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0095] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A virtual reality interaction method based on brain-computer interface and eye movement technology, characterized in that: The method comprises: Outputting a virtual reality scene, wherein the virtual reality scene includes at least one virtual object; Acquire eye movement data of a user interacting with the virtual reality scene, and determine a target object with which the user interacts from the at least one virtual object based on the eye movement data; Acquire electroencephalogram (EEG) data associated with the eye movement data, and determine the user's control intention on the target object based on the EEG data; Based on the control intention, a control operation is performed.

2. The method according to claim 1, characterized in that: The control operation includes: a control operation performed on the target object.

3. The method according to claim 2, characterized in that The performing of the control operation based on the control intention includes: If the control intention indicates that the user wants to view the target object, controlling the external device to turn toward the target object; or When the control intention indicates that the user wants to reach the location of the target object, the external device is controlled to move with the target object as the end point.

4. The method according to claim 2, characterized in that: In the case where the virtual reality scene is a smart home control scene, the target object is a virtual icon of a smart device, and the eye movement data indicates that the user is gazing at the virtual icon of the smart device; The performing of the control operation based on the control intention includes: When the control intention indicates that the user wants to control the smart device corresponding to the smart device virtual icon, the startup state of the smart device is controlled.

5. The method according to claim 2, characterized in that: In the case where the virtual reality scene is an auxiliary communication scene, the target object is a control on a virtual operation interface, and the eye movement data indicates that the user is looking at the control on the virtual operation interface; and performing a control operation based on the control intention includes: When the control intention indicates that the user wants to input information through the control on the virtual operation interface, the control on the virtual operation interface is controlled to be in a selected confirmed state or a command is sent through the control on the virtual operation interface.

6. The method according to claim 2, characterized in that In the case where the virtual reality scene is a medical rehabilitation scene, the target object is rehabilitation training content, the eye movement data indicates that the user selects rehabilitation training content or performs training based on rehabilitation training content, and the EEG data indicates the user's degree of relaxation with respect to the rehabilitation training content; The performing of the control operation based on the control intention includes: When the control intention indicates that the user is not satisfied with the training difficulty of the rehabilitation training content, the training difficulty of the rehabilitation training content is controlled to be adjusted.

7. The method according to claim 2, characterized in that In the case where the virtual reality scene is a virtual exhibition navigation scene, the target object is a virtual exhibit, and the eye movement data indicates that the user selects the virtual exhibit; and performing a control operation based on the control intention includes: When the control intention indicates that the user wants to learn about the virtual exhibit, the control outputs description information of the virtual exhibit.

8. The method according to claim 1, characterized in that The control operation includes: a control operation for switching virtual reality scenes.

9. The method according to claim 8, characterized in that The EEG data represents the user's emotional response to the target object; and the control operation is performed based on the control intention, including: When the control intention indicates that the user's emotional reaction to the target object is abnormal, the virtual reality scene is controlled to be switched.

10. The method according to any one of claims 1 to 9, characterized in that: The determining the user's control intention on the target object based on the EEG data includes: Based on the EEG data, determining whether EEG fluctuations are induced during the user's interaction with the target object through eye movement interaction; If it is determined that brain wave fluctuations are induced, it is determined that the user has a control intention on the target object.

11. A virtual reality interactive system based on brain-computer interface and eye movement technology, characterized in that: The system comprises: A data collection device, used to collect the user's eye movement data and EEG data; A virtual reality interaction device, used to execute the steps of the method described in any one of claims 1 to 10.

12. A virtual reality interactive device based on brain-computer interface and eye movement technology, characterized in that: The device comprises: An output module, used to output a virtual reality scene, wherein the virtual reality scene includes at least one virtual object; a first determination module, configured to obtain eye movement data of a user interacting with the virtual reality scene, and determine a target object interacted by the user from the at least one virtual object based on the eye movement data; A second determination module is used to obtain EEG data associated with the eye movement data, and determine the user's control intention on the target object based on the EEG data; The control module is used to execute a control operation based on the control intention.

13. An edge device, characterized in that: The edge device is used to perform edge computing, and the edge device can execute the virtual reality interaction method according to any one of claims 1 to 10, or the edge device is integrated with the virtual reality interaction system according to claim 11, or the edge device is integrated with the virtual reality interaction device according to claim 12.

14. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images