VR device control method based on visual brain-computer interface

Abstract

The application relates to a VR device control method based on a visual brain-computer interface. The method comprises the following steps: in response to a panel display request, displaying an input panel in a target area; receiving a first target indication signal sent by a control device; determining a first target visual stimulation area corresponding to a first target visual stimulation coding sequence indicated by the first target indication signal according to the first target visual stimulation coding sequence, and performing a target operation corresponding to the first target visual stimulation area. The method can improve the convenience of user operation in a VR space.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human-computer interaction, in particular to a VR device control method based on a visual brain-computer interface. BACKGROUND

[0002] With the progress of VR (Virtual Reality) technology, VR devices are gradually developing in the direction of simulating more fine and bringing more real experience to users.

[0003] In the prior art, users often need to control VR devices through VR handles. However, the VR handle has problems such as unnatural operation and simple interaction mode in the VR space. For example, when implementing keyboard input, the user needs to select letters on the virtual keyboard in sequence through the handle direction key for input, which is a relatively complex and time-consuming operation process, resulting in user inconvenience. SUMMARY

[0004] Therefore, it is necessary to provide a VR device control method based on a visual brain-computer interface in view of the above technical problems.

[0005] In a first aspect, the present application provides a VR device control method based on a visual brain-computer interface. The method comprises:

[0006] In response to a panel display request, displaying an input panel in a target area, the input panel comprising at least one first visual stimulation area, each first visual stimulation area corresponding to a different operation, and each first visual stimulation area flashing according to a visual stimulation code sequence corresponding to each first visual stimulation area, wherein the target area is an area between a VR device display area and a user, and for any two visual stimulation code sequences, the correlation of the two visual stimulation code sequences is less than a correlation threshold;

[0007] Receiving a first target indication signal sent by a control device, the first target indication signal being an indication signal generated according to a first target electroencephalogram signal, and the first target indication signal being used to indicate a target visual stimulation code sequence corresponding to the first target electroencephalogram signal, wherein the first target electroencephalogram signal is an electroencephalogram signal generated by the user when staring at a target visual stimulation area, and the target visual stimulation area is any first visual stimulation area in the at least one first visual stimulation area;

[0008] According to the target visual stimulation code sequence indicated by the first target indication signal, determining a target visual stimulation area corresponding to the target visual stimulation code sequence, and performing a target operation corresponding to the target visual stimulation area.

[0009] In one embodiment, the input panel comprises a mouse input panel, the mouse input panel comprises a mouse indicating block, each of the first visual stimulating areas is arranged radially with the mouse indicating block as the center, and each of the first visual stimulating areas corresponds to a different moving direction.

[0010] In one embodiment, the performing the target operation corresponding to the target visual stimulating area comprises:

[0011] moving the mouse to a specified position in the display area according to the moving direction corresponding to the first target visual stimulating area;

[0012] moving the mouse indicating block to a specified position in the mouse input panel according to the moving direction corresponding to the first target visual stimulating area;

[0013] redrawing the first visual stimulating area for each moving direction with the mouse indicating block as the center.

[0014] In one embodiment, the moving the mouse to a specified position in the display area according to the moving direction corresponding to the first target visual stimulating area comprises:

[0015] acquiring a continuous moving number corresponding to the moving direction;

[0016] determining a moving length of the mouse according to the continuous moving number, the moving length being positively correlated with the continuous moving number;

[0017] moving the mouse to a specified position according to the moving length.

[0018] In one embodiment, the displaying the input panel in the target area in response to the panel display request comprises:

[0019] displaying an input panel control area in the display area in response to the panel display request, the input panel control area comprising at least one second visual stimulating area, each of the second visual stimulating areas corresponding to a different input panel, and each of the second visual stimulating areas flashing according to a visual stimulating coding sequence corresponding to each of the second visual stimulating areas;

[0020] In a case that a second target indication signal sent by the control device is received, a second target visual stimulation coding sequence indicated by the second target indication signal is determined, a second target visual stimulation region corresponding to the second target visual stimulation coding sequence is determined, and a second target visual stimulation region corresponding input panel is generated in the target region, the second target indication signal is an indication signal generated according to a second target electroencephalogram signal, the second target electroencephalogram signal is an electroencephalogram signal generated by the user when staring at the second target visual stimulation region, and the second target visual stimulation region is any one of the at least one second visual stimulation region.

[0021] In an embodiment, the input panel is any one of a vertical input panel or a parallel input panel, wherein the vertical input panel is perpendicular to the display region, the parallel input panel is parallel to the display region, and the color of the parallel input panel is semi-transparent.

[0022] In a second aspect, the application further provides a VR device control method based on a visual brain-computer interface. The method comprises:

[0023] Receiving a target electroencephalogram signal sent by an electroencephalogram device, the target electroencephalogram signal being generated when the user stares at a target visual stimulation region, the target visual stimulation region being any one of at least one visual stimulation region in an input panel, the input panel being generated in a target region by a VR device in response to a panel display request, each visual stimulation region corresponding to a different operation, and each visual stimulation region flashing according to a visual stimulation coding sequence corresponding to the visual stimulation region, wherein the target region is a region between a display region of the VR device and the user, and for any two visual stimulation coding sequences, the correlation of the two visual stimulation coding sequences is less than a correlation threshold;

[0024] From the reference electroencephalogram signals corresponding to each visual stimulation coding sequence, a target reference electroencephalogram signal matching the target electroencephalogram signal is determined, a target visual stimulation coding sequence corresponding to the target reference electroencephalogram signal is determined according to a preset correspondence between reference electroencephalogram signals and visual stimulation coding sequences, and a target indication signal corresponding to the target visual stimulation coding sequence is generated;

[0025] The target indication signal is sent to the VR device, so that the VR device determines the target visual stimulation region corresponding to the target visual stimulation coding sequence according to the target visual stimulation coding sequence indicated by the target indication signal, and performs a target operation corresponding to the target visual stimulation region.

[0026] In a third aspect, the application further provides a VR device control system based on a visual brain-computer interface, comprising a VR device, an electroencephalogram device, and a control device, wherein

[0027] The VR device is configured to, in response to a panel display request, display an input panel corresponding to the panel display request in a target region, the input panel comprising at least one visual stimulation region, each of the visual stimulation regions corresponding to a different operation, and each of the visual stimulation regions flashing according to a visual stimulation code sequence corresponding to the visual stimulation region, wherein the target region is a region between a VR device display region and a user, and for any two of the visual stimulation code sequences, the correlation between the two visual stimulation code sequences is less than a correlation threshold.

[0028] The electroencephalogram device is configured to detect a target electroencephalogram signal generated by the user when gazing at a target visual stimulation region, and send the target electroencephalogram signal to the control device, the target visual stimulation region being any of the at least one first visual stimulation region.

[0029] The control device is configured to receive the target electroencephalogram signal sent by the electroencephalogram device, determine a target reference electroencephalogram signal corresponding to the target electroencephalogram signal, and determine a target visual stimulation code sequence corresponding to the target reference electroencephalogram signal according to a preset correspondence between reference electroencephalogram signals and visual stimulation code sequences, and generate a target indication signal corresponding to the target visual stimulation code sequence, and send the target indication signal to the VR device.

[0030] The VR device is further configured to receive the target indication signal sent by the control device, determine a target visual stimulation region corresponding to the target visual stimulation code sequence indicated by the target indication signal, and perform a target operation corresponding to the target visual stimulation region.

[0031] In a fourth aspect, the application further provides a VR device control apparatus based on a visual brain-computer interface, the apparatus comprising:

[0032] The display module is configured to, in response to a panel display request, display an input panel in a target region, the input panel comprising at least one first visual stimulation region, each of the first visual stimulation regions corresponding to a different operation, and each of the first visual stimulation regions flashing according to a visual stimulation code sequence corresponding to the first visual stimulation region, wherein the target region is a region between a VR device display region and a user, and for any two of the visual stimulation code sequences, the correlation between the two visual stimulation code sequences is less than a correlation threshold.

[0033] receive a first target indication signal sent by the control device, the first target indication signal being an indication signal generated according to a first target electroencephalogram signal, and the first target indication signal being used to indicate a first target visual stimulation coding sequence corresponding to the first target electroencephalogram signal, wherein the first target electroencephalogram signal is an electroencephalogram signal generated by the user when gazing at a first target visual stimulation area, and the first target visual stimulation area is any one of the at least one first visual stimulation area;

[0034] execute a module for determining a first target visual stimulation area corresponding to the first target visual stimulation coding sequence according to the first target visual stimulation coding sequence indicated by the first target indication signal, and executing a target operation corresponding to the first target visual stimulation area.

[0035] In one of the embodiments, the input panel includes a mouse input panel, the mouse input panel includes a mouse indicating block, each of the first visual stimulation areas is arranged radially with the mouse indicating block as the center, and each of the first visual stimulation areas corresponds to a different moving direction.

[0036] In one of the embodiments, the execution module is further configured to:

[0037] move the mouse to a specified position in the display area according to the moving direction corresponding to the first target visual stimulation area;

[0038] move the mouse indicating block to a specified position in the mouse input panel according to the moving direction corresponding to the first target visual stimulation area;

[0039] redraw the first visual stimulation area for each moving direction with the mouse indicating block as the center.

[0040] In one of the embodiments, the execution module is further configured to:

[0041] obtain a continuous moving number corresponding to the moving direction;

[0042] determine a moving length of the mouse according to the continuous moving number, the moving length being positively correlated with the continuous moving number;

[0043] move the mouse to a specified position along the moving direction according to the moving length.

[0044] In one of the embodiments, the display module is further configured to:

[0045] in response to the panel display request, display an input panel control region in the display region, the input panel control region comprising at least one second visual stimulus region, each of the second visual stimulus regions corresponding to a different input panel, and each of the second visual stimulus regions flashing according to a visual stimulus coding sequence corresponding to each of the second visual stimulus regions;

[0046] in a case where a second target indication signal sent by the control device is received, determining a second target visual stimulus region corresponding to a second target visual stimulus coding sequence indicated by the second target indication signal, and generating an input panel corresponding to the second target visual stimulus region in the target region, the second target indication signal being an indication signal generated according to a second target electroencephalogram signal, the second target electroencephalogram signal being an electroencephalogram signal generated by the user when gazing at the second target visual stimulus region, and the second target visual stimulus region being any one of the at least one second visual stimulus region.

[0047] In one of the embodiments, the input panel is any one of a vertical input panel or a parallel input panel, wherein the vertical input panel is perpendicular to the display region, the parallel input panel is parallel to the display region, and the color of the parallel input panel is semi-transparent.

[0048] In a fifth aspect, the present application further provides a VR device control apparatus based on a visual brain-computer interface, the apparatus comprising:

[0049] a receiving module configured to receive a target electroencephalogram signal sent by an electroencephalogram device, the target electroencephalogram signal being generated by the user when gazing at a target visual stimulus region, the target visual stimulus region being any one of at least one visual stimulus region in an input panel, the input panel being generated by a VR device in a target region in response to a panel display request, each of the visual stimulus regions corresponding to a different operation, and each of the visual stimulus regions flashing according to a visual stimulus coding sequence corresponding to each of the visual stimulus regions, wherein the target region is a region between a display region of the VR device and the user, and for any two of the visual stimulus coding sequences, the correlation between the two visual stimulus coding sequences is less than a correlation threshold;

[0050] a determining module configured to determine a target reference electroencephalogram signal matching the target electroencephalogram signal from reference electroencephalogram signals corresponding to each of the visual stimulus coding sequences, determine a target visual stimulus coding sequence corresponding to the target reference electroencephalogram signal according to a preset correspondence between reference electroencephalogram signals and visual stimulus coding sequences, and generate a target indication signal corresponding to the target visual stimulus coding sequence;

[0051] The sending module is configured to send the target indication signal to the VR device, so that the VR device determines a target visual stimulation area corresponding to a target visual stimulation coding sequence indicated by the target indication signal according to the target visual stimulation coding sequence, and performs a target operation corresponding to the target visual stimulation area.

[0052] In a sixth aspect, the present application provides a computer device. The computer device comprises a memory and a processor. The memory stores a computer program. The processor executes the computer program to implement any of the above methods.

[0053] In a seventh aspect, the present application provides a computer readable storage medium. The computer readable storage medium stores a computer program. The computer program is executed by a processor to implement any of the above methods.

[0054] In an eighth aspect, the present application provides a computer program product. The computer program product comprises a computer program. The computer program is executed by a processor to implement any of the above methods.

[0055] The above VR device control method, device, system, computer device and storage medium based on visual brain-computer interface, according to the user's panel display request, display an input panel in the area between the VR device display area and the user. The input panel comprises at least one first visual stimulation area, and each first visual stimulation area flashes according to a different visual stimulation coding sequence. The correlation of any two visual stimulation coding sequences is less than a correlation threshold. When the user gazes at any first target visual stimulation area in the first visual stimulation area, a first target brain electrical signal is generated. The VR device receives a first target indication signal generated by the control device according to the first target brain electrical signal, and determines a target operation to be performed according to the first target indication signal. The embodiments of the present application display the input panel in the space between the VR device display area and the user, and make the VR device perform a target operation corresponding to a visual stimulation area according to the brain electrical signal generated by the user when gazing at the first visual stimulation area in the input panel. Therefore, the user can control the VR device in the VR space without a VR handle, and the operation convenience of the user can be improved. The use of visual stimulation coding sequences with low correlation between each other can also improve the accuracy of distinguishing brain electrical signals induced according to different visual stimulation coding sequences. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 An application environment diagram of the VR device control method based on visual brain-computer interface in one embodiment;

[0057] Figure 2 A flowchart of the VR device control method based on visual brain-computer interface in one embodiment;

[0058] Figure 3 A schematic diagram of a target region in an embodiment;

[0059] Figure 4 A schematic diagram of a target region in an embodiment;

[0060] Figure 5 A comparison diagram of various visual stimulus encoding sequences in an embodiment;

[0061] Figure 6 A schematic diagram of a mouse input panel in an embodiment;

[0062] Figure 7 A flowchart of step 206 in an embodiment;

[0063] Figure 8 A schematic diagram of a mouse indicating block and movement of each first visual stimulus region in an embodiment;

[0064] Figure 9 A flowchart of step 704 in an embodiment;

[0065] Figure 10 A flowchart of step 202 in an embodiment;

[0066] Figure 11 A schematic diagram of the relative positions of each input panel and display region in an embodiment;

[0067] Figure 12 A schematic diagram of a vertical input panel in an embodiment;

[0068] Figure 13 A schematic diagram of a parallel input panel in an embodiment;

[0069] Figure 14 A schematic diagram of an embedded input panel in an embodiment;

[0070] Figure 15 A flowchart of a VR device control method based on a visual brain-computer interface in an embodiment;

[0071] Figure 16 A block diagram of a VR device control apparatus based on a visual brain-computer interface in an embodiment;

[0072] Figure 17 A block diagram of a VR device control apparatus based on a visual brain-computer interface in an embodiment;

[0073] Figure 18 An internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0074] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0075] The VR device control method based on the visual brain-computer interface provided by the embodiments of the present application can be applied to, but is not limited to, the application environment as shown in Figure 1 The VR device 102 communicates with the control device 104, and the control device 104 communicates with the electroencephalogram device 106. When the VR device 102 receives a panel display request, the VR device 102 displays an input panel corresponding to the panel display request in a target area. The user generates a corresponding electroencephalogram signal by gazing at a visual stimulation area on the input panel. The electroencephalogram device 106 collects the electroencephalogram signal of the user and sends the electroencephalogram signal to the control device 104. The control device 104 analyzes the electroencephalogram signal and determines an indication signal corresponding to the electroencephalogram signal. The control device 104 sends the indication signal to the VR device 102. The VR device 102 performs a corresponding target operation according to the indication signal. The control device 104 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc.

[0076] In one embodiment, as shown in Figure 2 , a VR device control method based on a visual brain-computer interface is provided. The present embodiment takes the VR device 102 in Figure 1 as an example to illustrate the method, which includes the following steps:

[0077] Step 202, in response to a panel display request, an input panel is displayed in a target area. The input panel includes at least one first visual stimulation area. Each first visual stimulation area corresponds to a different operation, and each first visual stimulation area flashes according to a visual stimulation coding sequence corresponding to each first visual stimulation area. The target area is the area between the VR device display area and the user. For any two visual stimulation coding sequences, the correlation of the two visual stimulation coding sequences is less than a correlation threshold.

[0078] In the embodiments of the present application, the user sends a panel display request to the VR device, so that the VR device generates and displays an input panel in the target area. The panel display request can be sent through a VR handle or other control methods, and the input panel can be a keyboard input panel (for example, a keyboard input panel composed of 40 target stimulus blocks), a mouse input panel (for example, a mouse input panel composed of multiple target stimulus blocks), or a user-defined input panel, etc. The embodiments of the present application do not make specific limitations on this.

[0079] The target area refers to the area between the display area of the VR device and the user. The display area refers to the display screen generated in the VR virtual space when the VR device only displays a two-dimensional picture (for example, when a normal video, text, image or webpage is viewed through the VR device, etc.). Referring to FIG. 1, after the user enters the VR space, the user has a viewpoint in the virtual space, and at the same time, the display area of the VR device is displayed parallel to the user's field of view at a certain distance from the user. At this time, the target area refers to the three-dimensional space between the display area of the VR device and the user's viewpoint. Figure 3

[0080] When there is no display area in the VR device (for example, when the user enters a VR game or other VR application), the target area can refer to a spherical area surrounding the user's viewpoint (see FIG. 2). When the user's viewpoint moves, the target area moves with the user's viewpoint. The input panel can be displayed in any area in the target area, for example, on a plane parallel to the user's field of view, or on a plane perpendicular to the user's field of view, etc. The embodiments of the present application do not make specific limitations on this. Figure 4

[0081] The embodiments of the present application can also be applied to AR devices. When applied to AR devices, the target area can refer to the three-dimensional space between the real object and the user.

[0082] The input panel includes at least one first visual stimulus area, each first visual stimulus area corresponds to a different operation, and different visual stimulus areas flash according to their respective visual stimulus coding sequences, wherein the correlation between any two visual stimulus coding sequences is less than a correlation threshold, so as to improve the accuracy of determining the area where the user is gazing. The correlation threshold can be set by those skilled in the art according to actual needs.

[0083] ​​In addition, the visual stimulation code sequence selected in the embodiments of the present application can also have a bandwidth greater than a bandwidth threshold and a phase change without periodicity, wherein the bandwidth threshold can also be set by those skilled in the art according to actual needs to achieve higher transmission information efficiency and reduce user visual fatigue. Moreover, the visual stimulation code sequence with a phase change without periodicity can also generate a larger number of sequences with a correlation less than a correlation threshold to support an input panel that needs to display dozens of different visual stimulation code sequences at the same time. Figure 5 Comparison of the visual stimulation code sequence in the embodiments of the present application and the M sequence (M-Sequence) and the steady-state visual evoked potential sequence (SSVEP, Steady-State Visual Evoked Potentials) in the prior art.

[0084] The operation corresponding to the first visual stimulation area can be a simple operation composed of one action, such as opening an application, adjusting the volume of the device, inputting a letter, etc., or an operation composed of multiple actions customized by the user, such as opening a certain video in a video application and adjusting the volume to a fixed value, etc., which is not limited in the embodiments of the present application.

[0085] The visual stimulation code sequence corresponding to each visual stimulation area can be pre-set or randomly selected from the visual stimulation code sequence library when the control panel is generated. Each visual stimulation code sequence in the visual stimulation code sequence library has a corresponding reference electroencephalogram signal, which refers to the electroencephalogram signal most likely to be generated by the user when watching the area flickering according to the visual stimulation code sequence. The user can be shown each visual stimulation code sequence in the visual stimulation code sequence library (for example, by making multiple areas flicker according to each visual stimulation code sequence) in advance (for example, when the user first uses the VR device), the reference electroencephalogram signal generated by the user based on different visual stimulation code sequences is collected, and the correspondence between the reference electroencephalogram signal and the visual stimulation code sequence is recorded. In actual application, the control device can determine which area flickering according to which visual stimulation code sequence the user is gazing at according to the electroencephalogram signal of the user (generally, the control device can complete accurate determination after the user gazes at the area for 0.2-0.3s), and then send the determined target visual stimulation code sequence to the VR device, so that the VR device determines which first target visual stimulation area the target visual stimulation code sequence corresponds to, and executes the target operation corresponding to the first target visual stimulation area.

[0086] For example, if the generated input panel includes two first visual stimulation areas (area A and area B), area A corresponds to visual stimulation coding sequence A and corresponds to the operation of flipping a page up, and area B corresponds to visual stimulation coding sequence B and corresponds to the operation of flipping a page down. When the area flashes according to visual stimulation coding sequence A, the induced user brain electrical signal is the reference brain electrical signal A, and when the area flashes according to visual stimulation coding sequence B, the induced user brain electrical signal is the reference brain electrical signal B. The actual process of controlling the VR device is as follows: if the user needs to flip a page down, the user gazes at area B. The electroencephalogram device collects the first target brain electrical signal of the user at this time and sends the first target brain electrical signal to the control device. The control device performs similarity comparison between the first target brain electrical signal and the reference brain electrical signals A and B, respectively. Since the visual stimulation coding sequences used have little correlation with each other, the similarity between the first target brain electrical signal and the reference brain electrical signal B is more likely to be higher. The control device thus determines that the user is gazing at the area flashing according to visual stimulation coding sequence B, and sends a first target indication signal to the VR device according to this, indicating to the VR device that the user is gazing at the area flashing according to visual stimulation coding sequence B. The VR device determines that the area corresponding to visual stimulation coding sequence B is area B, and that the operation corresponding to area B is flipping a page down, and then performs the operation of flipping a page down.

[0087] In step 204, a first target indication signal sent by the control device is received, the first target indication signal is an indication signal generated according to a first target brain electrical signal, and the first target indication signal is used to indicate a first target visual stimulation coding sequence corresponding to the first target brain electrical signal, wherein the first target brain electrical signal is a brain electrical signal generated by the user when gazing at a first target visual stimulation area, and the first target visual stimulation area is any first visual stimulation area in the at least one first visual stimulation area.

[0088] In the embodiments of the present application, when the user determines a target operation to be performed and gazes at a first target visual stimulation area corresponding to the target operation, the electroencephalogram device collects a first target brain electrical signal generated by the user and sends the first target brain electrical signal to the control device. The control device then determines a reference brain electrical signal corresponding to the first target brain electrical signal, and then determines a first target visual stimulation coding sequence corresponding to the first target brain electrical signal according to the correspondence between the reference brain electrical signal and the visual stimulation coding sequence. After the control device generates a first target indication signal used to indicate that the VR device corresponds to the first target visual stimulation coding sequence corresponding to the first target brain electrical signal, the control device sends the first target indication signal to the VR device, and the VR device then receives the first target indication signal.

[0089] In step 206, a first target visual stimulation region corresponding to the first target visual stimulation coding sequence is determined according to the first target visual stimulation coding sequence indicated by the first target indication signal, and a target operation corresponding to the first target visual stimulation region is performed.

[0090] In the embodiments of the present application, after receiving the first target indication signal, the VR device determines a first target visual stimulation region corresponding to the first target visual stimulation coding indicated by the first target indication signal, and performs a target operation corresponding to the first target visual stimulation region, so as to complete the process of controlling the VR device by the user through the brain electrical signals.

[0091] The VR device control method based on the visual brain-computer interface provided in the embodiments of the present application displays an input panel in the region between the VR device display region and the user according to the user's panel display request, the input panel includes at least one first visual stimulation region, and each first visual stimulation region flashes according to a different visual stimulation coding sequence. The correlation of any two visual stimulation coding sequences is less than a correlation threshold. When the user gazes at any first target visual stimulation region in the first visual stimulation region, a first target brain electrical signal is generated. The VR device receives a first target indication signal generated by the control device according to the first target brain electrical signal, and determines a target operation to be performed according to the first target indication signal. In the embodiments of the present application, the input panel is displayed in the space between the VR device display region and the user, and the VR device performs a target operation corresponding to the visual stimulation region according to the brain electrical signal generated by the user when gazing at the first visual stimulation region in the input panel, so that the user can control the VR device in the VR space without a VR handle, and the operation convenience of the user can be improved. The visual stimulation coding sequences with low correlation between each other are used, and the accuracy of distinguishing the brain electrical signals induced according to different visual stimulation coding sequences can be improved.

[0092] In one embodiment, the input panel includes a mouse input panel, the mouse input panel includes a mouse indication block, each first visual stimulation region is arranged radially with the mouse indication block as the center, and each first visual stimulation region corresponds to a different moving direction.

[0093] In the embodiments of the present application, the input panel includes a mouse input panel, the mouse input panel includes a mouse indication block, each first visual stimulation region is arranged radially with the mouse indication block as the center, and each first visual stimulation region corresponds to a different moving direction. Figure 6As shown, when the input panel is a mouse input panel, the mouse input panel can include a mouse indication block for indicating the position of the mouse, and a plurality of first visual stimulation regions for controlling the movement of the mouse in different directions. Among them, the mouse indication block can be marked with a more eye-catching color and does not flicker. When generating the mouse input panel, the mouse indication block can be generated at a random position, or the mouse indication block can be generated at a fixed preset position (such as the center of the mouse input panel). Each first visual stimulation region is arranged radially around the mouse indication block, and the position of each first visual stimulation region on the mouse input panel can be related to the corresponding movement direction. For example, the first visual stimulation region located above the mouse indication block can correspond to the upward movement direction, the first visual stimulation region located below and to the right of the mouse indication block can correspond to the downward and rightward movement direction, and so on, so as to improve the user's operation experience.

[0094] The mouse indication block and each first visual stimulation region can be composed of one or more display blocks. The display blocks belonging to the same first visual stimulation region flicker according to the same visual stimulation code sequence. By composing the first visual stimulation region with multiple display blocks, it is beneficial to dynamically move each first visual stimulation region together with the mouse indication block when the mouse indication block moves, so that the user only needs to gaze at the position where the mouse indication block needs to move to, and the operation of the mouse can be realized, without needing to judge which first visual stimulation region needs to be gazed at according to the real-time position of the mouse indication block. The reason is that the position gazed at by the user must be a display block on the mouse input panel, and when each first visual stimulation region and the mouse indication block dynamically move together, the relative position of each display block constituting each first visual stimulation region and the mouse indication block is actually the movement direction corresponding to the first visual stimulation region where it is located. Referring to Figure 6 As shown, if the initial state of the mouse input panel is A, and the user needs to move the mouse indication block filled with diagonal lines to the position of the display block filled with horizontal lines, the mouse input panel will go through two states B and C. It can be seen that the user only needs to continuously gaze at the position where the mouse indication block needs to move to, and the operation of the mouse and the mouse indication block can be realized, without moving the line of sight during the operation.

[0095] It should be noted that, Figure 6 In the embodiment, the mouse input panel includes eight first visual stimulation regions, and the visual stimulation regions A-H correspond to the upward left movement, the upward movement, the upward right movement, the right movement, the downward right movement, the downward movement, the downward left movement, and the left movement, respectively. However, in fact, the number of first visual stimulation regions can be increased or decreased according to actual needs, and the embodiments of the present application do not make specific limitations.

[0096] The VR device control method based on the visual brain-computer interface provided in the embodiments of the present application can set a mouse indicating block and first visual stimulation areas corresponding to respective moving directions on a mouse input panel, and the first visual stimulation areas are arranged radially with the mouse indicating block as the center, so that the user can see the real-time position of the mouse on the mouse input panel when operating the mouse, and thus the convenience of user operation can be improved.

[0097] In one embodiment, as shown in FIG. 2, in step 206, the target operation corresponding to the first target visual stimulation area is performed, including: Figure 7

[0098] In step 702, the mouse is moved to a specified position in the display area according to the moving direction corresponding to the first target visual stimulation area.

[0099] In step 704, the mouse indicating block is moved to a specified position in the mouse input panel according to the moving direction corresponding to the first target visual stimulation area.

[0100] In step 706, the first visual stimulation area for each mouse moving direction is redrawn with the mouse indicating block as the center.

[0101] In the embodiments of the present application, after receiving the first target indication signal, the VR device can move the mouse on the display area and move the mouse indicating block on the mouse input panel simultaneously. For example, a preset distance by which the mouse and the mouse indicating block are moved each time can be set in advance, and when the first target indication signal is received, the mouse and the mouse indicating block are moved to a specified position by the preset distance according to the direction indicated by the first target indication signal; or, the distance by which the mouse and the mouse indicating block are moved each time can be increased according to the user's habit of operating the mouse in the real space, when the mouse is continuously moved in the same direction, and the like, which is not limited in the embodiments of the present application.

[0102] Since the mouse indicating block on the mouse input panel should reflect the real-time position of the mouse on the display area, the proportional relationship between the distance by which the mouse is moved and the distance by which the mouse indicating block is moved can also be set according to the proportional relationship between the display area and the mouse input panel. For example, when the size proportion of the display area and the mouse input panel is 8:1, the proportional relationship between the distance by which the mouse is moved each time and the distance by which the mouse indicating block is moved each time can also be 8:1.

[0103] After the mouse indicating block is moved, the first visual stimulation area for each mouse moving direction can also be redrawn according to the position of the mouse indicating block after being moved. For details, refer to Figure 8 ​As shown, after the mouse moves to the right and up, the first visual stimulation area for 8 moving directions can be regenerated around the mouse indicating block to achieve the effect that each first visual stimulation area moves together with the mouse indicating block, so that the user only needs to gaze at the position to which the mouse needs to move when operating the mouse to move, and the user can operate the mouse without moving the line of sight, thereby improving the operation experience of the user.

[0104] It should be noted that the visual stimulation code sequence corresponding to each first visual stimulation area after regeneration can be the same as or different from the original visual stimulation code sequence, and the embodiments of the present application do not make a specific limitation.

[0105] The VR device control method based on the visual brain-computer interface provided by the embodiments of the present application can move the mouse according to the mouse moving direction indicated by the first target indicating signal, move the mouse indicating block on the mouse input panel, and redraw the first visual stimulation area for each mouse moving direction with the mouse indicating block after moving as the center. Therefore, each first visual stimulation area can move together with the mouse indicating block, so that the user only needs to gaze at the position to which the mouse needs to move in the process of following the movement of the mouse indicating block, and the user can operate the mouse without moving the line of sight, thereby improving the convenience of user operation. At the same time, since the mouse indicating block moves continuously, the phenomenon that the mouse indicating block jumps quickly and cannot be stable can also be avoided.

[0106] In one embodiment, as shown in Figure 9 In step 704, the mouse is moved to a specified position in the display area according to the moving direction corresponding to the first target visual stimulation area, including:

[0107] In step 902, the number of continuous movements corresponding to the mouse moving direction is obtained.

[0108] In step 904, the moving length of the mouse is determined according to the number of continuous movements, and the moving length is positively correlated with the number of continuous movements.

[0109] In step 906, the mouse is moved to a specified position along the mouse moving direction according to the moving length.

[0110] In this embodiment, the VR device may contain a counter that records the number of times the mouse moves continuously in the same direction. The counter records the direction of the mouse's last movement (hereinafter referred to as the historical direction) and the number of consecutive movements in that direction (i.e., the number of consecutive movements). When the current mouse movement direction is the same as the historical direction, the number of consecutive movements is incremented by one; when the current mouse movement direction is different from the historical direction, the number of consecutive movements is reset. When moving the mouse, the length of the mouse movement can be determined based on the number of consecutive movements recorded in the counter. For example, a unit movement length can be preset, and the length of the mouse movement can be determined by multiplying the number of consecutive movements by the unit movement length. Alternatively, other methods can be used to determine the length of the mouse movement, as long as the length of the mouse movement and the number of consecutive movements are positively correlated. This embodiment does not specifically limit this method.

[0111] For example, taking the product of the number of consecutive moves and the unit movement length as the mouse's current movement length, after the VR device determines that the mouse should move to the right, it compares the current mouse movement direction with the historical direction in the counter. If the historical direction was leftward, meaning the current mouse movement direction is different from the historical direction, the VR device resets the historical direction to rightward and resets the consecutive move count to 1 (or another initial value), thus determining the current mouse movement length as 1 unit movement length, and moves the mouse 1 unit movement length to the right to the specified position. The next time the mouse is moved, if the VR device still determines that the mouse should move to the right, meaning the current mouse movement direction is the same as the historical direction, the VR device increments the consecutive move count by 1 (currently 2), thus determining the current mouse movement length as 2 units movement length, and moves the mouse 2 units movement length to the right to the specified position... and so on, until the user finishes inputting.

[0112] It should be noted that the process of moving the mouse pointer on the mouse control panel can also refer to the process of moving the mouse in the display area described above, and will not be repeated here in the embodiments of this application.

[0113] The VR device control method based on a visual brain-computer interface provided in this application determines the movement length of the mouse according to the number of consecutive movements corresponding to the mouse movement direction, and makes the movement length and the number of consecutive movements positively correlated. Then, the mouse is moved according to the movement length, which can speed up the movement speed of the mouse when the user moves the mouse continuously in the same direction, and improve the convenience of user operation.

[0114] In one embodiment, such as Figure 10 As shown, in step 202, in response to the panel display request, an input panel is displayed in the target area, including:

[0115] At step 1002, in response to the panel display request, display an input panel control region in the display region, the input panel control region including at least one second visual stimulation region, each second visual stimulation region corresponding to a different input panel, and each second visual stimulation region flashing according to a visual stimulation coding sequence corresponding to the second visual stimulation region.

[0116] At step 1004, in a case where a second target indication signal sent by the control device is received, determine a second target visual stimulation region corresponding to a second target visual stimulation coding sequence according to the second target visual stimulation coding sequence indicated by the second target indication signal, and generate an input panel corresponding to the second target visual stimulation region in the target region, the second target indication signal being an indication signal generated according to a second target electroencephalogram signal, the second target electroencephalogram signal being an electroencephalogram signal generated by the user when gazing at the second target visual stimulation region, and the second target visual stimulation region being any one of the at least one second visual stimulation region.

[0117] In the embodiments of the present application, after receiving the panel display request sent by the user, the VR device can display, in the display region, the second visual stimulation region corresponding to each input panel allowed to be generated by the user in the current interface of the VR device. For example, when there is a text input interface in the display region, the second visual stimulation region corresponding to the keyboard input panel and the mouse input panel can be displayed in the display region; when there is no text input interface in the display region, the second visual stimulation region corresponding to the mouse input panel and the shortcut key input panel defined by the user can be displayed in the display region, and the like, which are not limited in the embodiments of the present application.

[0118] Each second visual stimulation region flashes according to a different visual stimulation coding sequence, the control device generates a second target indication signal by judging which visual stimulation coding sequence induces the electroencephalogram signal of the user, so as to make the VR device generate the corresponding input panel. The control device determines the second target indication signal, and the process of interaction between the control device and the electroencephalogram device and the VR device can refer to the related description of the control device determining the first target indication signal in the foregoing embodiments, which will not be described herein again.

[0119] The VR device control method based on the visual brain-computer interface provided in the embodiments of the present application can generate each second visual stimulation region, so that the user can send a panel display request to the VR device by gazing at the second visual stimulation region, thereby further reducing the operation of the user to the VR device through other devices and improving the convenience of the user in the VR space.

[0120] In one embodiment, the input panel is any of a vertical input panel or a parallel input panel, wherein the vertical input panel is perpendicular to the display area, the parallel input panel is parallel to the display area, and the parallel input panel is semi-transparent in color.

[0121] In the embodiments of the present application, the input panel can be a vertical input panel or a parallel input panel. Alternatively, the input panel can be arranged inside the display area as an embedded input panel. Referring to Figure 11 FIG. 1 shows that, taking the plane where the display area is located as the yz plane, a three-dimensional coordinate system is constructed in the VR space, and the vertical input panel can be arranged in the xy plane or the xz plane, or other planes parallel to the xy plane or the xz plane, that is, perpendicular to the display area. Referring to Figure 12 FIG. 2 shows one example of a vertical input panel arranged in the xy plane.

[0122] The parallel input panel can be parallel to the yz plane. Referring to Figure 13 FIG. 3 shows that, when the input panel is a parallel input panel, the parallel input panel can be arranged to be semi-transparent, so that the user can see the content displayed in the display area through the parallel input panel.

[0123] The embedded input panel can be arranged in the yz plane. Referring to Figure 14 FIG. 4 shows that the size of the embedded input panel should be smaller than the size of the display area.

[0124] The VR device control method based on the visual brain-computer interface provided in the embodiments of the present application can generate a vertical input panel or a parallel input panel. The vertical input panel can more efficiently utilize the idle space between the display area and the user, and the vertical input panel is more in line with the input habits of the user in the real space. The parallel input panel can enable the user to see the content in the display area while inputting, which facilitates the user to check the input content, and thus improves the convenience of user operation while utilizing the idle space in the VR space.

[0125] In one embodiment, as Figure 15 FIG. 5 shows that a VR device control method based on a visual brain-computer interface is provided. The embodiments take the control device 104 in Figure 1 as an example for illustration, which includes the following steps:

[0126] At step 1502, the target electroencephalogram signal transmitted by the electroencephalogram device is received, the target electroencephalogram signal is generated when the user gazes at the target visual stimulation area, the target visual stimulation area is any one of the at least one visual stimulation area in the input panel, the input panel is generated in the target area by the VR device in response to the panel display request, the input panel includes at least one visual stimulation area, each visual stimulation area corresponds to a different operation, and each visual stimulation area flashes according to the visual stimulation coding sequence corresponding to each visual stimulation area, wherein the target area is the area between the VR device display area and the user, and for any two visual stimulation coding sequences, the correlation of the two visual stimulation coding sequences is less than the correlation threshold.

[0127] At step 1504, the target reference electroencephalogram signal matched with the target electroencephalogram signal is determined from the reference electroencephalogram signal corresponding to each visual stimulation coding sequence, and the target visual stimulation coding sequence corresponding to the target reference electroencephalogram signal is determined according to the preset correspondence between the reference electroencephalogram signal and the visual stimulation coding sequence, and the target indication signal corresponding to the target visual stimulation coding sequence is generated.

[0128] At step 1506, the target indication signal is sent to the VR device, so that the VR device determines the target visual stimulation area corresponding to the target visual stimulation coding sequence indicated by the target indication signal according to the target visual stimulation coding sequence indicated by the target indication signal, and performs the target operation corresponding to the target visual stimulation area.

[0129] In the embodiment of the application, the electroencephalogram device collects the target electroencephalogram signal generated when the user gazes at the target visual stimulation area in the input panel of the VR device, and sends the target electroencephalogram signal to the control device. The control device has pre-recorded reference electroencephalogram signals corresponding to each visual stimulation coding sequence, and the acquisition method of the reference electroencephalogram signal can refer to the related description of the foregoing embodiment, which will not be repeated here. The control device determines the target reference electroencephalogram signal most similar to the target electroencephalogram signal from each reference electroencephalogram signal by matching the target electroencephalogram signal and each reference electroencephalogram signal, and determines the target visual stimulation coding sequence corresponding to the target reference electroencephalogram signal, and then generates the target indication signal for informing the VR device of the target visual stimulation coding sequence, and sends the target indication signal to the VR device. The interaction process of the control device, the electroencephalogram device and the VR device, and the operations performed by the control device, the electroencephalogram device and the VR device can refer to the related description of the foregoing embodiment, which will not be repeated here.

[0130] The VR device control method based on the visual brain-computer interface provided in the embodiments of the present application displays an input panel in the region between the VR device display region and the user according to the user's panel display request, the input panel comprises at least one first visual stimulation region, and each first visual stimulation region flashes according to a different visual stimulation code sequence. The correlation of any two visual stimulation code sequences is less than a correlation threshold. When the user gazes at any first target visual stimulation region in the first visual stimulation region, a first target electroencephalogram signal is generated. The control device receives the target electroencephalogram signal, determines a target reference electroencephalogram signal corresponding to the target electroencephalogram signal, and determines a target visual stimulation code sequence corresponding to the target reference electroencephalogram signal, and then informs the VR device of the target visual stimulation code sequence through a first target indication signal, so that the VR device determines a target operation to be performed according to the target visual stimulation region corresponding to the target visual stimulation code sequence. The embodiments of the present application display the input panel in the space between the VR device display region and the user, and make the VR device perform a target operation corresponding to the visual stimulation region according to the electroencephalogram signal generated when the user gazes at the first visual stimulation region in the input panel, so that the user can control the VR device in the VR space without a VR handle, and the operation convenience of the user can be improved. The use of visual stimulation code sequences with low correlation between each other can also improve the accuracy of distinguishing electroencephalogram signals induced according to different visual stimulation code sequences.

[0131] It should be understood that, although each step in the flowchart involved in each of the above-described embodiments is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in each of the above-described embodiments can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps.

[0132] In one embodiment, a VR device control system based on a visual brain-computer interface is provided, the system comprising a VR device, an electroencephalogram device, and a control device, wherein:

[0133] The VR device is configured to display, in response to a panel display request, an input panel corresponding to the panel display request in a target area, the input panel including at least one visual stimulation area, each visual stimulation area corresponding to a different operation, and each visual stimulation area flashing according to a visual stimulation code sequence corresponding to the visual stimulation area, wherein the target area is an area between a VR device display area and a user, and for any two visual stimulation code sequences, the correlation of the two visual stimulation code sequences is less than a correlation threshold.

[0134] The EEG device is configured to detect a target EEG signal generated by a user when the user gazes at a target visual stimulation area, and send the target EEG signal to the control device, the target visual stimulation area being any one of the at least one first visual stimulation area.

[0135] The control device is configured to receive the target EEG signal sent by the EEG device, determine a target reference EEG signal corresponding to the target EEG signal, and determine a target visual stimulation code sequence corresponding to the target reference EEG signal according to a preset correspondence between reference EEG signals and visual stimulation code sequences, and generate a target indication signal corresponding to the target visual stimulation code sequence, and send the target indication signal to the VR device.

[0136] The VR device is further configured to receive the target indication signal sent by the control device, determine a target visual stimulation area corresponding to the target visual stimulation code sequence indicated by the target indication signal, and perform a target operation corresponding to the target visual stimulation area.

[0137] In the embodiments of the present application, the interaction process of the VR device, the control device and the EEG device, and the specific operations performed by the VR device, the control device and the EEG device in the VR device control process based on the visual brain-computer interface can be referred to the related description of the foregoing embodiments, which will not be repeated here.

[0138] The VR device control system based on the visual brain-computer interface provided in the embodiments of the present application displays an input panel in the region between the VR device display region and the user according to a panel display request of the user, the input panel includes at least one first visual stimulation region, and each first visual stimulation region flashes according to a different visual stimulation coding sequence. The correlation of any two visual stimulation coding sequences is less than a correlation threshold. When the user gazes at any first target visual stimulation region in the first visual stimulation region, a first target electroencephalogram signal is generated. The VR device receives a first target indication signal generated by the control device according to the first target electroencephalogram signal, and determines a target operation to be performed according to the first target indication signal. The embodiments of the present application display the input panel in the space between the VR device display region and the user, and make the VR device perform a target operation corresponding to the visual stimulation region according to the electroencephalogram signal generated when the user gazes at the first visual stimulation region in the input panel, so that the user can control the VR device in the VR space without a VR handle, and the operation convenience of the user can be improved. The use of visual stimulation coding sequences with low correlation between each other can also improve the accuracy of distinguishing the electroencephalogram signals induced according to different visual stimulation coding sequences.

[0139] Based on the same inventive concept, the embodiments of the present application also provide a visual brain-computer interface based VR device control apparatus for implementing the visual brain-computer interface based VR device control method described above. The implementation scheme for solving the problem provided by the apparatus is similar to the implementation scheme described in the above method, so the specific limitations in one or more visual brain-computer interface based VR device control apparatus embodiments provided below can be referred to the limitations of the visual brain-computer interface based VR device control method described above, which will not be described here again.

[0140] In one embodiment, as shown in Figure 16 A visual brain-computer interface based VR device control apparatus is provided, including: a display module 1602, a receiving module 1604, and an execution module 1606, wherein:

[0141] The display module 1602 is configured to display an input panel in a target region in response to a panel display request, the input panel including at least one first visual stimulation region, each first visual stimulation region corresponding to a different operation, and each first visual stimulation region flashing according to a visual stimulation coding sequence corresponding to the first visual stimulation region, wherein the target region is a region between a VR device display region and a user, and for any two visual stimulation coding sequences, the correlation of the two visual stimulation coding sequences is less than a correlation threshold;

[0142] The receiving module 1604 is configured to receive a first target indication signal sent by the control device, the first target indication signal being an indication signal generated according to a first target electroencephalogram signal, and the first target indication signal being used to indicate a first target visual stimulation coding sequence corresponding to the first target electroencephalogram signal, wherein the first target electroencephalogram signal is an electroencephalogram signal generated by the user when gazing at a first target visual stimulation area, and the first target visual stimulation area is any first visual stimulation area in the at least one first visual stimulation area;

[0143] The executing module 1606 is configured to determine a first target visual stimulation area corresponding to the first target visual stimulation coding sequence according to the first target visual stimulation coding sequence indicated by the first target indication signal, and perform a target operation corresponding to the first target visual stimulation area.

[0144] The VR device control apparatus based on a visual brain-computer interface provided by the embodiments of the present application can display an input panel in a region between a display region of a VR device and a user according to a panel display request of the user, the input panel includes at least one first visual stimulation area, and each first visual stimulation area flashes according to a different visual stimulation coding sequence. The correlation of any two visual stimulation coding sequences is less than a correlation threshold. The user generates a first target electroencephalogram signal when gazing at any first target visual stimulation area in the first visual stimulation area. The VR device receives a first target indication signal generated by a control device according to the first target electroencephalogram signal, and determines a target operation to be performed according to the first target indication signal. The embodiments of the present application display the input panel in the space between the display region of the VR device and the user, and make the VR device perform a target operation corresponding to a visual stimulation area according to the electroencephalogram signal generated by the user when gazing at the first visual stimulation area in the input panel, so that the user can control the VR device in the VR space without a VR handle, and the operation convenience of the user can be improved. The use of visual stimulation coding sequences with low correlation between each other can also improve the accuracy of distinguishing electroencephalogram signals induced according to different visual stimulation coding sequences.

[0145] In one of the embodiments, the input panel includes a mouse input panel, the mouse input panel includes a mouse indication block, each first visual stimulation area is arranged radially with the mouse indication block as the center, and each first visual stimulation area corresponds to a different moving direction.

[0146] In one of the embodiments, the executing module 1606 is further configured to:

[0147] move the mouse to a specified position in the display region according to the moving direction corresponding to the first target visual stimulation area;

[0148] moving the mouse pointer in the mouse input panel to a specified position according to the moving direction corresponding to the first target visual stimulus area;

[0149] redrawing the first visual stimulus area for each moving direction with the mouse pointer as the center.

[0150] In one embodiment, the execution module 1606 is further configured to:

[0151] acquiring a continuous moving number corresponding to the moving direction;

[0152] determining a moving length of the mouse according to the continuous moving number, the moving length being positively correlated with the continuous moving number;

[0153] moving the mouse to a specified position according to the moving length along the moving direction.

[0154] In one embodiment, the display module 1602 is further configured to:

[0155] displaying an input panel control area in the display area in response to a panel display request, the input panel control area including at least one second visual stimulus area, each second visual stimulus area corresponding to a different input panel, and each second visual stimulus area flashing according to a visual stimulus coding sequence corresponding to the second visual stimulus area;

[0156] In a case where a second target indication signal sent by a control device is received, determining a second target visual stimulus area corresponding to a second target visual stimulus coding sequence indicated by the second target indication signal according to the second target visual stimulus coding sequence, and generating an input panel corresponding to the second target visual stimulus area in the target area, the second target indication signal being an indication signal generated according to a second target electroencephalogram signal, the second target electroencephalogram signal being an electroencephalogram signal generated by a user when gazing at a second target visual stimulus area, and the second target visual stimulus area being any one of the at least one second visual stimulus area.

[0157] In one embodiment, the input panel is any one of a vertical input panel or a parallel input panel, wherein the vertical input panel is perpendicular to the display area, the parallel input panel is parallel to the display area, and the color of the parallel input panel is semi-transparent.

[0158] In one embodiment, as shown in Figure 17 Fig. 1, a VR device control apparatus based on a visual brain-computer interface is provided, including a receiving module 1702, a determining module 1704, and a sending module 1706.

[0159] The receiving module 1702 is configured to receive a target electroencephalogram signal sent by the electroencephalogram device, the target electroencephalogram signal being generated when the user gazes at a target visual stimulation area, the target visual stimulation area being any one of at least one visual stimulation area in an input panel, the input panel being generated in a target area by the VR device in response to a panel display request, each of the visual stimulation areas corresponding to a different operation, and each of the visual stimulation areas flashing according to a visual stimulation coding sequence corresponding to each of the visual stimulation areas, wherein the target area is an area between a display area of the VR device and the user, and for any two visual stimulation coding sequences, the correlation between the two visual stimulation coding sequences is less than a correlation threshold;

[0160] The determining module 1704 is configured to determine a target reference electroencephalogram signal matched with the target electroencephalogram signal from reference electroencephalogram signals corresponding to each of the visual stimulation coding sequences, determine a target visual stimulation coding sequence corresponding to the target reference electroencephalogram signal according to a preset correspondence between reference electroencephalogram signals and visual stimulation coding sequences, and generate a target indication signal corresponding to the target visual stimulation coding sequence.

[0161] The sending module 1706 is configured to send the target indication signal to the VR device, so that the VR device determines the target visual stimulation area corresponding to the target visual stimulation coding sequence indicated by the target indication signal according to the target visual stimulation coding sequence, and performs a target operation corresponding to the target visual stimulation area.

[0162] The embodiment of the application provides a VR device control device based on a visual brain-computer interface. According to a panel display request of a user, an input panel is displayed in a region between a VR device display region and the user. The input panel includes at least one first visual stimulation region, and each first visual stimulation region flashes according to a different visual stimulation coding sequence. The correlation of any two visual stimulation coding sequences is less than a correlation threshold. When the user gazes at any first target visual stimulation region in the first visual stimulation region, a first target electroencephalogram signal is generated. The control device receives the target electroencephalogram signal, determines a target reference electroencephalogram signal corresponding to the target electroencephalogram signal, and determines a target visual stimulation coding sequence corresponding to the target reference electroencephalogram signal. Then, the VR device is informed of the target visual stimulation coding sequence through a first target indication signal, so that the VR device determines a target operation to be performed according to a target visual stimulation region corresponding to the target visual stimulation coding sequence. The embodiment of the application displays the input panel in the space between the VR device display region and the user, and makes the VR device perform a target operation corresponding to the visual stimulation region according to the electroencephalogram signal generated when the user gazes at the first visual stimulation region in the input panel, so that the user can control the VR device in the VR space without a VR handle, and the operation convenience of the user can be improved. The visual stimulation coding sequences with low correlation between each other are used, and the accuracy of distinguishing the electroencephalogram signals induced according to different visual stimulation coding sequences can be improved.

[0163] The modules in the VR device control device based on the visual brain-computer interface can be all or partially realized by software, hardware and a combination thereof. The modules can be embedded in or independent of a processor in a computer device in a hardware form, or stored in a memory in the computer device in a software form, so as to be called and executed by a processor to perform the operations corresponding to the modules.

[0164] In one embodiment, a computer device can be a server, and an internal structure diagram of the computer device can be as shown in Figure 18 The computer device includes a processor, a memory and a network interface connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement a VR device control method based on a visual brain-computer interface.

[0165] Those skilled in the art can understand that Figure 18The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0166] In an embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor implementing the steps in the above method embodiments when executing the computer program.

[0167] In an embodiment, a computer readable storage medium is provided, storing a computer program, and the computer program implementing the steps in the above method embodiments when executed by a processor.

[0168] In an embodiment, a computer program product is provided, including a computer program, and the computer program implementing the steps in the above method embodiments when executed by a processor.

[0169] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties.

[0170] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0171] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0172] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A VR device control method based on a visual brain-computer interface, characterized in that, The method includes: In response to a panel display request, an input panel is displayed within a target area. The input panel includes at least one first visual stimulus area, each corresponding to a different operation, and each first visual stimulus area flashes according to a visual stimulus coding sequence corresponding to that first visual stimulus area. The target area is the area between the VR device display area and the user. For any two visual stimulus coding sequences, the correlation between the two visual stimulus coding sequences is less than a correlation threshold. The input panel includes a mouse input panel, which includes a mouse pointer block. Each first visual stimulus area is arranged radially around the mouse pointer block, and each first visual stimulus area corresponds to a different movement direction. The direction of each first visual stimulus area relative to the mouse pointer block is the same as the movement direction corresponding to each first visual stimulus area. The system receives a first target indication signal sent by a control device. The first target indication signal is an indication signal generated based on a first target EEG signal. The first target indication signal is used to indicate a first target visual stimulus encoding sequence corresponding to the first target EEG signal. The first target EEG signal is an EEG signal generated when the user gazes at a first target visual stimulus area. The first target visual stimulus area is any one of the at least one first visual stimulus areas. Based on the first target visual stimulus coding sequence indicated by the first target indication signal, determine the first target visual stimulus region corresponding to the first target visual stimulus coding sequence, and execute the target operation corresponding to the first target visual stimulus region. Wherein, when the input panel is the mouse input panel, the execution of the target operation corresponding to the first target visual stimulus area includes: Move the mouse to the specified position in the display area according to the movement direction corresponding to the first target visual stimulus area; Move the mouse pointer block to the designated position in the mouse input panel according to the movement direction corresponding to the first target visual stimulation area; Centered on the mouse pointer block, the first visual stimulus area is redrawn for each movement direction.

2. The method according to claim 1, characterized in that, Moving the mouse to a designated position in the display area according to the movement direction corresponding to the first target visual stimulus area includes: Obtain the number of consecutive movements corresponding to the stated direction of movement; The movement length of the mouse is determined based on the number of consecutive movements, and the movement length is positively correlated with the number of consecutive movements. Move the mouse to the designated position along the movement direction according to the stated movement length.

3. The method according to claim 1 or 2, characterized in that, The step of displaying an input panel in the target area in response to a panel display request includes: In response to a panel display request, an input panel control area is displayed in the display area. The input panel control area includes at least one second visual stimulation area, each second visual stimulation area corresponds to a different input panel, and each second visual stimulation area flashes according to the visual stimulation coding sequence corresponding to each second visual stimulation area. Upon receiving a second target indication signal sent by the control device, the second target visual stimulus region corresponding to the second target visual stimulus coding sequence is determined according to the second target visual stimulus coding sequence indicated by the second target indication signal, and an input panel corresponding to the second target visual stimulus region is generated within the target region. The second target indication signal is an indication signal generated based on a second target EEG signal, the second target EEG signal is an EEG signal generated when the user gazes at the second target visual stimulus region, and the second target visual stimulus region is any one of the at least one second visual stimulus region.

4. The method according to claim 3, characterized in that, The input panel is either a vertical input panel or a parallel input panel, wherein the vertical input panel is perpendicular to the display area, the parallel input panel is parallel to the display area, and the color of the parallel input panel is semi-transparent.

5. A VR device control method based on a visual brain-computer interface, characterized in that, The method includes: The system receives target EEG signals sent by an EEG device, which are generated when the user gazes at a target visual stimulus area. The target visual stimulus area is any one of at least one visual stimulus area in an input panel. The input panel is generated within the target area by the VR device in response to a panel display request. Each visual stimulus area corresponds to a different operation, and each visual stimulus area flashes according to its corresponding visual stimulus coding sequence. The target area is the area between the VR device display area and the user. For any two visual stimulus coding sequences, the correlation between the two visual stimulus coding sequences is less than a correlation threshold. The input panel includes a mouse input panel, which includes a mouse pointer block. Each visual stimulus area is arranged radially around the mouse pointer block, and each visual stimulus area corresponds to a different movement direction. The direction of each visual stimulus area relative to the mouse pointer block is the same as the movement direction corresponding to each visual stimulus area. From the reference EEG signals corresponding to each visual stimulus coding sequence, a target reference EEG signal that matches the target EEG signal is determined. Based on the preset correspondence between the reference EEG signal and the visual stimulus coding sequence, a target visual stimulus coding sequence corresponding to the target reference EEG signal is determined, and a target indication signal corresponding to the target visual stimulus coding sequence is generated. The target indication signal is sent to the VR device so that the VR device determines the target visual stimulus region corresponding to the target visual stimulus encoding sequence according to the target visual stimulus encoding sequence indicated by the target indication signal, and performs the target operation corresponding to the target visual stimulus region. Wherein, when the input panel is the mouse input panel, the execution of the target operation corresponding to the target visual stimulus area includes: Move the mouse to the specified position in the display area according to the movement direction corresponding to the target visual stimulus area; Move the mouse pointer block to the designated position in the mouse input panel according to the movement direction corresponding to the target visual stimulation area; Centered on the mouse pointer block, the visual stimulus area for each movement direction is redrawn.

6. A VR device control system based on a visual brain-computer interface, characterized in that, This includes VR devices, EEG devices, and control devices, among which, The VR device is configured to respond to a panel display request and display an input panel corresponding to the panel display request within a target area. The input panel includes at least one visual stimulation area, each visual stimulation area corresponding to a different operation, and each visual stimulation area flashes according to a visual stimulation coding sequence corresponding to that visual stimulation area. The target area is the area between the VR device display area and the user. For any two visual stimulation coding sequences, the correlation between the two visual stimulation coding sequences is less than a correlation threshold. The input panel includes a mouse input panel, which includes a mouse pointer block. Each visual stimulation area is arranged radially around the mouse pointer block, each visual stimulation area corresponding to a different movement direction, and the direction of each visual stimulation area relative to the mouse pointer block is the same as the movement direction corresponding to that visual stimulation area. The EEG device is used to detect the target EEG signal generated when the user gazes at the target visual stimulation area, and to send the target EEG signal to the control device. The target visual stimulation area is any one of the at least one visual stimulation areas. The control device is used to receive the target EEG signal sent by the EEG device, determine the target reference EEG signal corresponding to the target EEG signal, and determine the target visual stimulus coding sequence corresponding to the target reference EEG signal according to the preset correspondence between the reference EEG signal and the visual stimulus coding sequence, generate the target indication signal corresponding to the target visual stimulus coding sequence, and send the target indication signal to the VR device. The VR device is further configured to receive a target indication signal sent by the control device, determine the target visual stimulus region corresponding to the target visual stimulus encoding sequence according to the target visual stimulus encoding sequence indicated by the target indication signal, and perform the target operation corresponding to the target visual stimulus region. Wherein, when the input panel is the mouse input panel, the execution of the target operation corresponding to the target visual stimulus area includes: Move the mouse to the specified position in the display area according to the movement direction corresponding to the target visual stimulus area; Move the mouse pointer block to the designated position in the mouse input panel according to the movement direction corresponding to the target visual stimulation area; Centered on the mouse pointer block, the visual stimulus area for each movement direction is redrawn.

7. A VR device control device based on a visual brain-computer interface, characterized in that, The device includes: A display module is used to respond to a panel display request and display an input panel within a target area. The input panel includes at least one first visual stimulus area, each first visual stimulus area corresponding to a different operation, and each first visual stimulus area flashes according to a visual stimulus coding sequence corresponding to that first visual stimulus area. The target area is the area between the VR device display area and the user. For any two visual stimulus coding sequences, the correlation between the two visual stimulus coding sequences is less than a correlation threshold. The input panel includes a mouse input panel, which includes a mouse pointer block. Each first visual stimulus area is arranged radially around the mouse pointer block, each first visual stimulus area corresponding to a different movement direction, and the direction of each first visual stimulus area relative to the mouse pointer block is the same as the movement direction corresponding to that first visual stimulus area. The receiving module is used to receive a first target indication signal sent by the control device. The first target indication signal is an indication signal generated based on a first target EEG signal, and the first target indication signal is used to indicate a first target visual stimulus encoding sequence corresponding to the first target EEG signal. The first target EEG signal is an EEG signal generated by the user when looking at the first target visual stimulus area, and the first target visual stimulus area is any one of the at least one first visual stimulus areas. The execution module is configured to determine the first target visual stimulus region corresponding to the first target visual stimulus encoding sequence indicated by the first target indication signal, and execute the target operation corresponding to the first target visual stimulus region. Wherein, when the input panel is the mouse input panel, the execution of the target operation corresponding to the first target visual stimulus area includes: Move the mouse to the specified position in the display area according to the movement direction corresponding to the first target visual stimulus area; Move the mouse pointer block to the designated position in the mouse input panel according to the movement direction corresponding to the first target visual stimulation area; Centered on the mouse pointer block, the first visual stimulus area is redrawn for each movement direction.

8. A VR device control device based on a visual brain-computer interface, characterized in that, The device includes: A receiving module is used to receive target EEG signals sent by an EEG device. The target EEG signals are generated when the user gazes at a target visual stimulus area. The target visual stimulus area is any one of at least one visual stimulus area in an input panel. The input panel is generated within the target area by the VR device in response to a panel display request. Each visual stimulus area corresponds to a different operation, and each visual stimulus area flashes according to its corresponding visual stimulus coding sequence. The target area is the area between the VR device display area and the user. For any two visual stimulus coding sequences, the correlation between the two visual stimulus coding sequences is less than a correlation threshold. The input panel includes a mouse input panel, which includes a mouse pointer block. Each visual stimulus area is arranged radially around the mouse pointer block. Each visual stimulus area corresponds to a different movement direction, and the direction of each visual stimulus area relative to the mouse pointer block is the same as the movement direction corresponding to each visual stimulus area. The determination module is used to determine a target reference EEG signal that matches the target EEG signal from the reference EEG signals corresponding to each visual stimulus coding sequence, determine the target visual stimulus coding sequence corresponding to the target reference EEG signal according to the preset correspondence between the reference EEG signal and the visual stimulus coding sequence, and generate a target indication signal corresponding to the target visual stimulus coding sequence. The sending module is used to send the target indication signal to the VR device, so that the VR device determines the target visual stimulus region corresponding to the target visual stimulus encoding sequence according to the target visual stimulus encoding sequence indicated by the target indication signal, and performs the target operation corresponding to the target visual stimulus region. Wherein, when the input panel is the mouse input panel, the execution of the target operation corresponding to the target visual stimulus area includes: Move the mouse to the specified position in the display area according to the movement direction corresponding to the target visual stimulus area; Move the mouse pointer block to the designated position in the mouse input panel according to the movement direction corresponding to the target visual stimulation area; Centered on the mouse pointer block, the visual stimulus area for each movement direction is redrawn.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images