Display method and device of brain-computer interaction interface

Abstract

The invention discloses a display method and device of a brain-computer interaction interface, relates to the technical field of human-computer interaction, and aims to improve the display effect of the brain-computer interaction interface. The method comprises the following steps: a first device generates a brain-computer interaction interface based on obtained interface parameters. Wherein the interface parameters are determined according to the position relation between the observation point and the display screen and the size of the display screen. The first device obtains a target brain wave signal. Wherein the target brain wave signal is generated when the user watches the brain-computer interaction interface. The first device determines one or more control instructions corresponding to the target electroencephalogram signal according to the corresponding relation between the electroencephalogram signal and the control instructions, and sends the one or more control instructions to the controlled device. And the first equipment receives the state information from the controlled equipment. Wherein the state information is used for indicating the operation state of the controlled equipment. And the first device generates a target image according to the state information, and displays the target image in the brain-computer interaction interface.

Description

Technical Field

[0001] The present application relates to the field of human-computer interaction technology, and in particular to a display method and device for a brain-computer interaction interface. Background Art

[0002] Brain-computer interface (BCI) refers to the establishment of a direct path between the human brain and external devices to collect brain wave signals (electroencephalogram, EEG) without relying on peripheral nerves and muscles, thereby realizing information transmission and control. Virtual reality (VR) is a technology that generates a virtual world through computer technology, three-dimensional imaging technology, multimedia simulation technology, display technology, etc. VR devices based on brain-computer interface can provide users with a better user experience.

[0003] However, the display effect of the brain-computer interaction interface based on brain wave signals to control VR devices in the existing technical solutions is poor and needs to be improved. Summary of the invention

[0004] The present invention discloses a display method, system and device for a brain-computer interaction interface, which relate to the technical field of human-computer interaction and are used to improve the display effect of the brain-computer interaction interface.

[0005] In a first aspect, an embodiment of the present application provides a method for displaying a brain-computer interaction interface, comprising: generating a brain-computer interaction interface based on acquired interface parameters. The interface parameters are determined according to the positional relationship between the observation point and the display screen and the size of the display screen. Obtaining a target brain wave signal. The target brain wave signal is generated when the user looks at the brain-computer interaction interface. Determine one or more control instructions corresponding to the target brain wave signal based on the correspondence between the brain wave signal and the control instruction, and send one or more control instructions to the controlled device. Receive status information from the controlled device. The status information is used to indicate the operating status of the controlled device. Generate a target image based on the status information, and display the target image in the brain-computer interaction interface.

[0006] The method is used to determine interface parameters according to the positional relationship between the observation point and the display screen and the size of the display screen, and generate a brain-computer interaction interface according to the interface parameters. A target image is acquired and displayed in the brain-computer interaction interface to achieve that the size of the target image displayed on the brain-computer interaction interface is the same as the size of the target image displayed on the display screen, thereby improving the display effect of the brain-computer interaction interface and improving the user's visual experience.

[0007] In one possible design, the interface parameters include: the pitch angle between the observation point and the display screen, the yaw angle between the observation point and the display screen, the horizontal distance between the observation point and the display screen, the height of the display screen, the width of the display screen, the vertical field of view angle of the brain-computer exchange interface, the horizontal field of view angle of the brain-computer exchange interface, the layer depth of the brain-computer exchange interface, the width of the brain-computer exchange interface, and the height of the brain-computer exchange interface.

[0008] Based on this design, the interface parameters of the brain-computer interaction interface are determined according to the positional relationship between the observation point and the display screen and the parameter information of the display screen, which can improve the display effect of the brain-computer interaction interface and enhance the user's visual experience.

[0009] In one possible design, the interface parameters satisfy:

[0010]

[0011] Scenes h =Object h ;

[0012] Scenes w =Object w ;

[0013] Fov v =Pitch;

[0014] Fov h =Yaw;

[0015] Position z =Distance;

[0016] Among them, Pitch represents the pitch angle between the observation point and the display screen, Yaw represents the yaw angle between the observation point and the display screen, and Object h Indicates the height of the display screen, Object w Indicates the width of the display, Distance indicates the horizontal distance between the observation point and the display, and Scenes w Indicates the width of the brain-computer interface, Scenes h Indicates the height of the brain-computer interface, Fov v Indicates the vertical field of view of the brain-computer interface, Fov h Indicates the horizontal field of view of the brain-computer interface, Position z Depth of layers representing the brain-computer interface.

[0017] In one possible design, the target EEG signal is generated when the user gazes at a stimulation block in the brain-computer interaction interface, and the positional relationship between the stimulation block and the brain-computer interaction interface is the same as the positional relationship between the stimulation block and the display screen.

[0018] In one possible design, before displaying the target image in the brain-computer interaction interface, it also includes: determining the optimization strategy corresponding to the interaction device based on the correspondence between the performance parameters of the brain-computer interaction device and the optimization strategy, and the brain-computer interaction device is used to present the brain-computer interaction interface; optimizing and processing the target image according to the optimization strategy corresponding to the brain-computer interaction device, and displaying the optimized target image on the brain-computer interaction interface.

[0019] With this design, the image of the target object is optimized according to the current performance of the imaging device to avoid problems such as frame drops and freezes, thereby improving the stability of the imaging device.

[0020] In one possible design, the optimization strategy corresponding to the brain-computer interaction device is determined according to the correspondence between the performance parameters of the brain-computer interaction device and the optimization strategy, including: determining the optimization strategy corresponding to the brain-computer interaction device from the correspondence between the performance parameters of the brain-computer interaction device and the optimization strategy according to the usage of the graphics processing unit GPU, central processing unit CPU or memory of the brain-computer interaction device.

[0021] In a second aspect, the embodiment of the present application further provides a display device for a brain-computer interaction interface, including a communication module and a processing module.

[0022] The processing module is used to generate a brain-computer interaction interface based on the obtained interface parameters, and the interface parameters are determined according to the positional relationship between the observation point and the display screen and the size of the display screen. The communication module is used to obtain the target brain wave signal, which is generated when the user looks at the brain-computer interaction interface; the processing module is also used to determine one or more control instructions corresponding to the target brain wave signal based on the correspondence between the brain wave signal and the control instruction, and send one or more control instructions to the controlled device; the communication module is also used to receive status information from the controlled device, and the status information is used to indicate the operating status of the controlled device; the processing module is used to generate a target image based on the status information, and display the target image in the brain-computer interaction interface.

[0023] In one possible design, the interface parameters include: the pitch angle between the observation point and the display screen, the yaw angle between the observation point and the display screen, the horizontal distance between the observation point and the display screen, the height of the display screen, the width of the display screen, the vertical field of view angle of the brain-computer exchange interface, the horizontal field of view angle of the brain-computer exchange interface, the layer depth of the brain-computer exchange interface, the width of the brain-computer exchange interface, and the height of the brain-computer exchange interface.

[0024] In one possible design, the interface parameters satisfy:

[0025]

[0026] Scenesh =Object h ;

[0027] Scenes w =Object w ;

[0028] Fov v =Pitch;

[0029] Fov h =Yaw;

[0030] Position z =Distance;

[0031] Among them, Pitch represents the pitch angle between the observation point and the display screen, Yaw represents the yaw angle between the observation point and the display screen, Object h Indicates the height of the display screen, Object w Indicates the width of the display, Distance indicates the horizontal distance between the observation point and the display, and Scenes w Indicates the width of the brain-computer interface, Scenes h Indicates the height of the brain-computer interface, Fov v Indicates the vertical field of view of the brain-computer interface, Fov h Indicates the horizontal field of view of the brain-computer interface, Position z Depth of layers representing the brain-computer interface.

[0032] In one possible design, the optimization strategy corresponding to the brain-computer interaction device is determined according to the correspondence between the performance parameters of the brain-computer interaction device and the optimization strategy. The processing module is specifically used to determine the optimization strategy corresponding to the brain-computer interaction device from the correspondence between the performance parameters of the brain-computer interaction device and the optimization strategy according to the usage of the graphics processing unit GPU, central processing unit CPU or memory of the brain-computer interaction device.

[0033] In one possible design, before displaying the target image in the brain-computer interaction interface, the processing module is specifically used to: determine the optimization strategy corresponding to the interaction device based on the correspondence between the performance parameters of the brain-computer interaction device and the optimization strategy, and the brain-computer interaction device is used to present the brain-computer interaction interface; optimize and process the target image according to the optimization strategy corresponding to the brain-computer interaction device, and display the optimized target image on the brain-computer interaction interface.

[0034] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a processor, wherein the processor is configured to implement the method of the first aspect and any one of its designs when executing a computer program stored in a memory.

[0035] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of the first aspect and any one of its designs.

[0036] The technical effects brought about by the second to fourth aspects and any one of their designs can refer to the technical effects brought about by the corresponding design in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 It is a structural schematic diagram of a brain-computer interface system in the prior art;

[0039] Figure 2 A schematic diagram of a flow chart of a method for displaying a brain-computer interaction interface provided in an embodiment of the present application;

[0040] Figure 3 A schematic diagram of the structure of a brain-computer interface system provided in an embodiment of the present application;

[0041] Figure 4 A schematic diagram of the structure of a communication system provided in an embodiment of the present application;

[0042] Figure 5 A brain-computer interaction interface provided in an embodiment of the present application;

[0043] Figure 6 A waveform diagram of spectral information provided in an embodiment of the present application;

[0044] Figure 7 Another waveform diagram of spectral information provided in an embodiment of the present application;

[0045] Figure 8 A schematic diagram of the structure of a display device for a brain-computer interaction interface provided in an embodiment of the present application;

[0046] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be described in detail in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0048] Next, the prior art solutions combining brain-computer interface and VR equipment are introduced.

[0049] Figure 1 FIG. 1 is a schematic diagram of the structure of a brain-computer interface system in the prior art. Figure 1 As shown, Figure 1 The brain-computer interface system shown includes a signal acquisition module, a signal processing module, a peripheral control module and an interactive interface module.

[0050] Signal processing methods usually include: signal preprocessing, feature extraction and classification recognition. The peripheral control module can generate control instructions according to the preset strategy and send the control instructions to the corresponding device.

[0051] The interactive interface module can receive visual feedback from other devices and present it on the interactive interface. In the prior art, the way to present pictures on the interactive interface is usually to place flickering stimuli in a rich virtual scene. Or, in a VR scene, the flickering stimuli are superimposed on the background video of the virtual scene.

[0052] However, the display effect of the brain-computer interaction interface of VR devices controlled by brain wave signals in the existing technical solutions is poor and needs to be improved.

[0053] In order to solve the above-mentioned defects, the present application provides a display method and device of a brain-computer interaction interface to improve the display effect of the brain-computer interaction interface.

[0054] In the present application, the method adopted includes: the first device generates a brain-computer interaction interface based on the acquired interface parameters. The interface parameters are determined according to the positional relationship between the observation point and the display screen and the size of the display screen. The first device obtains a target brain wave signal. The target brain wave signal is generated when the user looks at the brain-computer interaction interface. The first device determines one or more control instructions corresponding to the target brain wave signal based on the correspondence between the brain wave signal and the control instruction, and sends one or more control instructions to the controlled device. The first device receives status information from the controlled device. The status information is used to indicate the operating status of the controlled device. The first device generates a target image based on the status information and displays the target image in the brain-computer interaction interface.

[0055] The method is used to determine interface parameters according to the positional relationship between the observation point and the display screen and the size of the display screen, and generate a brain-computer interaction interface according to the interface parameters. A target image is acquired and displayed in the brain-computer interaction interface to achieve that the size of the target image displayed on the brain-computer interaction interface is the same as the size of the target image displayed on the display screen, thereby improving the display effect of the brain-computer interaction interface and improving the user's visual experience.

[0056] In addition, the first device may be included in a computer system for executing the method described in the present application, or may be a processing device in a computer system for executing the method described in the present application, such as a processor or a processing module, etc., which is not specifically limited in the present application.

[0057] Figure 2 A flowchart of a method for displaying a brain-computer interaction interface provided in an embodiment of the present application. Taking the first device as the execution subject as an example, the process may include the following steps:

[0058] S201, the first device generates a brain-computer interaction interface based on the acquired interface parameters, wherein the interface parameters are determined according to the positional relationship between the observation point and the display screen and the size of the display screen.

[0059] In one or more embodiments, the interface parameters include at least one of the following: the pitch angle between the observation point and the display screen, the yaw angle between the observation point and the display screen, the horizontal distance between the observation point and the display screen, the height of the display screen, the width of the display screen, the vertical field of view of the brain-computer exchange interface, the horizontal field of view of the brain-computer exchange interface, the layer depth of the brain-computer exchange interface, the width of the brain-computer exchange interface, and the height of the brain-computer exchange interface. Optionally, the interface parameters also include the resolution of the display.

[0060] It is understandable that the observation point can be the position where the user observes the display screen during the test. Alternatively, the observation point can be a reference point generated according to business needs. Among them, the reference point can be a virtual point or a real point. This application does not make specific restrictions.

[0061] In one or more embodiments, the interface parameters satisfy:

[0062]

[0063] Scenes h =Object h ;

[0064] Scenes w =Object w ;

[0065] Fov v =Pitch;

[0066] Fov h =Yaw;

[0067] Position z =Distance;

[0068] Among them, Pitch represents the pitch angle between the observation point and the display screen, Yaw represents the yaw angle between the observation point and the display screen, and Object h Indicates the height of the display screen, Object w Indicates the width of the display, Distance indicates the horizontal distance between the observation point and the display, and Scenes w Indicates the width of the brain-computer interface, Scenes h Indicates the height of the brain-computer interface, Fov v Indicates the vertical field of view of the brain-computer interface, Fov h Indicates the horizontal field of view of the brain-computer interface, Position z Depth of layers representing the brain-computer interface.

[0069] It can be understood that based on this embodiment, the interface parameters of the brain-computer interaction interface are determined according to the positional relationship between the observation point and the display screen and the parameter information of the display screen, which can improve the display effect of the brain-computer interaction interface and enhance the user's visual experience.

[0070] S202, the first device obtains a target brain wave signal, wherein the target brain wave signal is generated when the user looks at the brain-computer interaction interface.

[0071] As an example, the first device may include a module or device with a function of collecting brain wave signals, and the brain wave device may collect brain wave signals through its own module or device. The first device may collect brain wave signals generated by the user in real time. Furthermore, after the first device collects brain wave signals through its own module or device, it may also perform pre-processing such as filtering on the collected brain wave signals to reduce noise signals of the brain wave signals.

[0072] As another example, the device for collecting brain wave signals can be performed by other devices. For example, Figure 3 A schematic diagram of the structure of a brain-computer interface system provided in an embodiment of the present application is shown in FIG. Figure 3As shown, the signal acquisition device is used to collect brain wave signals. The signal acquisition device can be a portable device or a fixed device, and this application does not make specific limitations. The signal acquisition device can collect brain wave signals generated by the user in real time, and feed back the collected brain wave signals to the first device. At this time, the signal acquisition device can be considered as a device independent of the first device, or a device in an independent device. The first device and the signal acquisition device can be connected through a wired interface and / or a wireless interface. Correspondingly, the signal acquisition device can send brain wave signals to the first device through a wired interface and / or a wireless interface. The first device can receive brain wave signals from the signal acquisition device through a wired interface and / or a wireless interface.

[0073] In one or more embodiments, the target brain wave signal is generated when the user gazes at a stimulation block in the brain-computer interaction interface, and the positional relationship between the stimulation block and the brain-computer interaction interface is the same as the positional relationship between the stimulation block and the display screen. The stimulation block is a module that induces the user to generate corresponding brain waves according to preset rules. The stimulation block can be a virtual module, such as a virtual light module displayed on a display screen or a brain-computer interaction interface. The stimulation block can also be a real device or module, such as a device or module independent of a display screen or a brain-computer interaction interface. For example, Figure 5 As shown, stimulation blocks 1, 2, 3, 4, 5 and 6 are distributed on the brain-computer interaction interface. The stimulation blocks can be used to induce the user to generate corresponding brain wave signals. That is, at least one of stimulation blocks 1, 2, 3, 4, 5 and 6 can be used to induce the user to generate brain wave signals.

[0074] S203, the first device determines one or more control instructions corresponding to the target EEG signal according to the correspondence between the EEG signal and the control instructions, and sends the one or more control instructions to the controlled device. Correspondingly, the controlled device receives the one or more control instructions from the first device.

[0075] In one or more embodiments, the correspondence between brain wave signals and control instructions may be pre-set according to business needs. Optionally, the correspondence between brain wave signals and control instructions may be stored in a table. Among them, one brain wave signal may correspond to multiple control instructions, or one brain wave signal may correspond to one control instruction. In addition, when the correspondence between brain wave signals and control instructions does not include a target brain wave signal, the target brain wave signal may be added to the correspondence between brain wave signals and control instructions according to business needs, and a correspondence between the brain wave signal and one or more control instructions may be constructed.

[0076] As an example, the correspondence between brain wave signals and control instructions may also refer to the correspondence between the signal type of brain wave signals and control instructions. In other words, the correspondence between brain wave signals and control instructions is determined based on the correspondence between the signal type of brain wave signals and control instructions. Accordingly, after the first device acquires the target brain wave signal, it can determine the signal type of the target brain wave signal based on the pre-stored correspondence between brain wave signals and signal types, and determine one or more control instructions corresponding to the target brain wave signal based on the correspondence between the signal type and the control instructions.

[0077] For example, Figure 3 As shown, the brain wave signal decoding module (EEG decoding module) can decode the brain wave information to obtain the signal type corresponding to the brain wave signal. The EEG decoding module can send the signal type corresponding to the brain wave information to the control plug-in through the transmission control protocol (TCP).

[0078] Optionally, the first device and the EEG decoding module can be integrated or as one entity, or the first device includes the EEG decoding module. In this case, the first device and the EEG decoding module can be connected via an internal bus. Accordingly, the first device can directly call the EEG decoding module to decode the brain wave signal, thereby obtaining the signal type corresponding to the brain wave signal. Furthermore, the first device can determine one or more control instructions corresponding to the brain computer signal from the correspondence between the pre-stored signal type and the control instruction according to the signal type corresponding to the brain wave signal. Figure 3 For example, the control plug-in can be represented as the first device, the decoding instruction can be represented as the signal type corresponding to the brain wave signal, and the peripheral control instruction can be represented as the control instruction. The control plug-in can determine that the signal type corresponds to the peripheral control instruction according to the decoding instruction mapping relationship.

[0079] Table 1 is a table of correspondence between a signal type and a control instruction provided in an embodiment of the present application. Table 1 is a table of correspondence between the signal type of the brain wave signal and the control instruction of the drone. As shown in Table 1, when the signal type of the target brain wave signal is identified as 1, the control instruction corresponding to the target brain wave signal may be takeoff. When the signal type of the target brain wave signal is identified as 2, the control instruction corresponding to the target brain wave signal is landing.

[0080] Signal type identification Control instructions Signal type identification Control instructions 1 take off 6 Shift Right 2 landing 7 go ahead 3 Move Up 8 Back 4 Move Down 9 Counterclockwise rotation 5 Shift Left 10 Rotate clockwise

[0081] Table 1

[0082] In one or more embodiments, the first device determines the instruction buffer corresponding to the multiple control instructions according to the correspondence between the control instructions and the instruction buffer, and stores the multiple control instructions in the instruction buffer.

[0083] It is understandable that the type of control instructions that can be cached by the instruction buffer in the present application may correspond to the operating state of the controlled device. That is, when the operating state of the controlled device is different, the instruction buffer may cache different types of control instructions.

[0084] Optionally, the first device sends one or more control instructions to the controlled device.

[0085] In one or more embodiments, after the first device obtains the target control instruction, it can send the target instruction to the controlled device. Accordingly, the controlled device receives the target control instruction from the first device. The controlled device can perform corresponding actions according to the target control instruction. In addition, after receiving the target control instruction, the controlled device can also send a confirmation message to the first device. For example, after receiving the target control instruction, the controlled device can send an acknowledgement character (ACK) to the first device. Figure 3 As shown, the flight device can be represented as a controlled device, and after obtaining the peripheral control instruction, the control plug-in can send the peripheral control instruction to the flight device.

[0086] In one or more embodiments, the first device and the interactive interface may be connected via a named pipe, thereby enabling data communication between the first device and the interactive interface. Figure 3 As shown, the control plug-in can feed back data stream to the interactive interface module through the named pipe. Similarly, the interactive interface module can send status data to the control plug-in through the named pipe.

[0087] Furthermore, before sending the data stream, the first device and the interactive interface may perform preprocessing operations on the data stream based on data processing technology. The operation of preprocessing the data stream may be operations such as compressing, trimming, and encapsulating the data stream. For example, before sending the data stream to the interactive interface, the first device may compress the data stream to reduce the size of the data stream file. For another example, before sending the interactive interface's operating status data stream to the first device, the interactive interface may encapsulate the operating status data stream to ensure the integrity of data transmission.

[0088] In addition, corresponding data buffers can be set for different types of data streams in the named pipe according to business needs. For example, when the interactive interface sends a running status data stream to the first device through the named pipe, the controlled device can store the running status data in the running status data buffer of the named pipe in the form of FIFO. Correspondingly, the first device can obtain the data stream of the running status of the interactive interface from the running status data buffer.

[0089] It is understandable that the first device and the interactive interface can use the same programming language when communicating through a named pipe. The first device and the interactive interface can also use two different programming languages ​​when communicating through a named pipe. That is to say, when the first device sends a data stream to the interactive interface, the process of acquiring, processing, and writing data can be encapsulated in a programming language. Correspondingly, the process of the interactive interface acquiring and reading the data stream can be unpacked in another programming language. This method is used for communication in the present application. Even if the first device and the interactive interface support different programming languages, the first device and the interactive interface can also achieve cross-language communication without the need to convert the formats of different programming languages, so as to improve the communication efficiency between the two devices.

[0090] S204, the first device receives the status information from the controlled device. Correspondingly, the controlled device sends the status information of the device to the first device, wherein the status information is used to indicate the operating status of the controlled device.

[0091] In one or more embodiments, after receiving one or more control instructions from the first device, the controlled device may execute corresponding control instructions based on the one or more control instructions. The specific control instructions executed by the controlled device can be found in step S203, which will not be described here. After executing the control instructions, the controlled device may send the status information of its own device to the first device. Alternatively, the controlled device may send the status information of its own device to the first device in real time during the execution of the control instructions. Alternatively, the controlled device may send the status information of its own device to the first device according to preset rules. For example, the preset rules may be to send status information to the controlled device at a fixed time. The controlled device may also send status information to the first device according to the indication information used to indicate that the status information is sent to the first device.

[0092] In one or more embodiments, the specific implementation manner of the controlled device sending the status information to the first device may refer to the specific implementation manner of the first device sending the control instruction to the controlled device, which will not be repeated here.

[0093] S205: The first device generates a target image according to the state information, and displays the target image in the brain-computer interaction interface.

[0094] In one or more embodiments, after receiving the status information fed back by the controlled device, the first device can generate a corresponding target image according to the status information. For example, still taking the controlled device as a flight device as an example, the flight device can feed back to the first device that the current operating status of its own device is flight. Accordingly, the first device receives the status information from the flight device that the operating status is flight, and the first device can generate an image with the flight status of flight. For another example, the flight device has a shooting function, and the flight device can feed back the picture taken by the device to the first device in real time. Accordingly, the first device receives the shot picture from the flight device, and processes the shot picture based on the parameter information of the brain-computer interaction interface, so as to obtain the target image corresponding to the shot picture.

[0095] In one or more embodiments, after generating the corresponding target image, the first device may display the target image in the brain-computer interaction interface.

[0096] In the present application, in order to improve the user experience, the brain-computer interaction interface can be optimized according to a pre-set strategy before displaying the target image in the brain-computer interaction interface.

[0097] The following is an introduction to the optimization of the brain-computer interaction interface provided by this application:

[0098] In one or more embodiments, the first device may determine the optimization strategy corresponding to the brain-computer interaction device according to the corresponding relationship between the performance parameters of the brain-computer interaction device and the optimization strategy. The brain-computer interaction device is used to present the brain-computer interaction interface. The first device may optimize the target image according to the optimization strategy corresponding to the brain-computer interaction device, and display the optimized target image on the brain-computer interaction interface.

[0099] In one or more embodiments, the first device determines the optimization strategy corresponding to the brain-computer interaction device based on the correspondence between the performance parameters of the brain-computer interaction device and the optimization strategy, including: determining the optimization strategy corresponding to the brain-computer interaction device from the correspondence between the performance parameters of the brain-computer interaction device and the optimization strategy according to the usage of the graphics processing unit GPU, central processing unit CPU or memory of the brain-computer interaction device.

[0100] Optionally, the current performance parameters of the brain-computer interaction device can be obtained by evaluating the performance of the brain-computer interaction device based on auxiliary means. Auxiliary means are, for example, a task manager, a development engine, a performance testing tool, a sensor, etc. Optionally, the performance parameters of the imaging device may include a graphics processing unit (GPU), a central processing unit (CPU), a memory, etc.

[0101] Exemplarily, frame rate test and delay test are performed on flicker stimulus and background based on performance test tools, so as to observe the stable frame rate, minimum frame rate, frame rate drop and operation delay time of the imaging device during operation.

[0102] For example, based on the resource usage of the task manager, it can be determined that the current performance bottleneck may be caused by excessive GPU computing load, limited CPU processing power, or memory overload.

[0103] In one or more embodiments, the brain-computer interaction device can determine the optimization strategy of the imaging device from the correspondence between the performance parameters of the brain-computer interaction device and the optimization strategy according to the usage of the GPU, CPU and memory of the brain-computer interaction device. The usage of the GPU, CPU and memory of the brain-computer interaction device can be, for example, the total amount and current usage of the GPU, CPU and memory of the imaging device, or the usage rate of the GPU, CPU and memory of the imaging device.

[0104] As an example, the optimization strategy of the brain-computer interaction device may include at least one of a static object optimization strategy, a dynamic object optimization strategy, and a resource optimization strategy.

[0105] Among them, the optimization strategy for static objects includes: setting a low level of anti-aliasing effect, selecting a low-dynamic rendering method, simplifying the quality of shadows, setting multiple levels of detail (LOD) rendering methods, etc., to reduce the GPU computing load, thereby avoiding flickering and frame drops. In addition, during static processing, you can set static batch processing to merge multiple static objects into one object, complete the rendering and drawing tasks at one time, so as to reduce the number of rendering times and improve the GPU rendering efficiency, thereby further avoiding flickering and frame drops and improving the reliability of the interactive interface.

[0106] Dynamic object optimization strategies include: turning off dynamic collision detection, turning on offline collision detection mode, reducing the simulation of physical properties of dynamic objects, etc., to reduce CPU workload, reduce data flow delay to avoid freezing problems, and thus improve the smoothness of the scene. In addition, during dynamic processing, you can set dynamic batch processing to combine multiple dynamic objects with the same material and less than the required number of vertices into a single object for a single rendering call, so as to reduce the number of rendering calls, improve CPU utilization efficiency, and thus reduce the frequency of flickering and frame drops.

[0107] Resource optimization strategies include: reducing the number of objects in the picture, simplifying object models, simplifying object textures, reasonably reducing the number of light sources, etc., in order to reduce memory usage, increase the space occupied by device feedback data streams and flickering stimuli, and avoid problems such as high latency in device feedback data streams and flickering stimuli frame drops. In addition, when applied to scenarios with only device status feedback and control interfaces, the device status can be optimized for resources. For example, the data file size can be compressed by reducing the frequency of status feedback, reducing the background image resolution, converting the image format, etc., to balance visual effects and performance requirements, further reduce memory usage, and avoid problems such as high latency in device feedback data streams and flickering stimuli frame drops.

[0108] The method provided by the present application is introduced below by taking Example 1 as an example. In Example 1, a steady state visual evoked potential-braincomputer interface (SSVEP-BCI) is used in a VR environment to control an AirSim virtual drone to shoot, and present the pictures shot by the drone on an imaging device as an example.

[0109] In this embodiment, a control plug-in, an EEG decoding module, a VR device and an AirSim virtual drone are taken as examples.

[0110] The carrier of the control plug-in may be a dynamic link library (DLL) file. The DDL file may be created using C++. The control plug-in may communicate with the EEG decoding module via a transmission control protocol. The communication link between the control plug-in and the VR device may be a local direct communication link created via a named pipe.

[0111] In this embodiment, the control plug-in can call the C++ version image acquisition interface provided by AirSim to obtain the first-person perspective observation image of the AirSim virtual drone, and send the obtained frame image to the VR device through a named pipe.

[0112] The interactive interface program of the VR device can use C# to perform a real-time empty operation on the named pipe. When the named pipe is not empty, the image data in the named pipe is read and presented as the interface background.

[0113] Understandably, the communication between the control plug-in and the VR device runs independently in a multi-threaded loop to present the images observed by the AirSim virtual drone in real time.

[0114] In this embodiment, the EEG decoding module can decode the brain wave signal to obtain a decoding instruction. After obtaining the decoding instruction, the EEG decoding module can send the decoding instruction to the control plug-in through the TCP protocol. The control plug-in can determine the control action corresponding to the decoding instruction according to the correspondence between the preset decoding instruction and the control action.

[0115] After the control plug-in obtains the control action, it can store the control action in the buffer according to the running state of the AirSim virtual drone. For example, when the flag "fly" used to indicate the flight state of the AirSim virtual drone is false, the control action can be stored in buffer A. When the flag "fly" used to indicate the flight state of the AirSim virtual drone is true, the control action can be stored in buffer B.

[0116] When the number of control actions of "take-off" in buffer A is greater than 3, the control plug-in sends the control action of "take-off" to the AirSim virtual drone and marks the flag "fly" as true. Similarly, when the number of control actions of "landing" in buffer A is greater than 3, the control plug-in sends the control action of "landing" to the AirSim virtual drone and marks the flag "fly" as false.

[0117] In this embodiment, buffer B is a 4*5 two-dimensional matrix. The first dimension can be used to represent the speed of the four degrees of freedom of the flying device, front and back, left and right, up and down, clockwise and counterclockwise rotation, and the second dimension can be used to cache five control instructions. The control plug-in can sum all the control actions in buffer B to obtain the target control action, and send the target control action to the AirSim virtual drone.

[0118] It is understandable that the control function implemented by the control plug-in can be encapsulated as a DDL control plug-in so that the interactive interface program of the VR device calls the DDL control plug-in in a multi-threaded manner. Figure 4 This is a schematic diagram of the structure of a communication system provided in an embodiment of the present application. Figure 4 As shown, the functions of acquiring, processing and writing data streams are encapsulated as a control plug-in, and the data stream is sent to the named pipe in the form of a FIFO file. Accordingly, the interactive interface can read the data stream from the named pipe and present the status of the data stream.

[0119] In addition, this application can also use cross-platform visual size standardization technology to adjust the interactive interface of VR devices.

[0120] Figure 5 A brain-computer interaction interface is provided in an embodiment of the present application, such as Figure 5 As shown, Figure 5Take the AirSim virtual drone as an example. Figure 5 The first device generates a target image based on the real-time feedback of the shooting picture of the drone. h The width of the display is 33.57 cm. w The distance between the observation point and the display screen is 59.67cm, the distance between the observation point and the display screen is 70cm, and the resolution of the display screen is 2560*1440. The pitch angle Pitch and yaw angle Yaw of the VR imaging device can be determined according to the real height of the display screen, the real width of the display screen, and the distance between the observation point and the display screen. Among them, the real height of the display screen, the real width of the display screen, the distance between the observation point and the display screen, and the pitch angle and yaw angle of the VR imaging device meet the following conditions:

[0121]

[0122] Furthermore, in this embodiment, the cross-platform visual size standardization technology can be used to achieve the consistency of the visual size of the interactive interface in the VR scene with the display screen. In other words, the parameters of the VR interactive interface can be adjusted according to the parameters of the display screen, so that the resolution of the VR screen is the same as the resolution of the display screen (i.e. 2560*1440), and the display layer width Scenes w The same width as the display screen (i.e. Scenes w =Object w ), display layer height Scenes h Same height as the display screen (i.e. Scenes h =Object h ). The vertical field of view of the VR camera v The same as the elevation angle of the observation point (i.e. Fov v =Pitch), the horizontal field of view of the VR camera Fov h Same as the yaw angle of the observation point (i.e. Fov h =Yaw), the layer depth Position of the interactive interface z The same as the distance between the observation point and the display screen (i.e. Position z =Distance).

[0123] The interactive interface includes stimulation block 1, stimulation block 2, stimulation block 3, stimulation block 4, stimulation block 5 and stimulation block 6. The positional relationship between the stimulation block and the intersection interface is determined according to the positional relationship between the stimulation block and the display.

[0124] In this embodiment, a photocell sensor and a Unity Profiler tool may be used to perform performance evaluation on the interactive interface of a VR device.

[0125] Taking the performance evaluation of the interactive interface of a VR device using a photocell sensor as an example, the photocell sensor is used to collect the spectral information of a specified stimulus block in the interactive interface of the VR device, and the spectral information is plotted as a waveform graph. Figure 6 A waveform diagram of spectral information provided in an embodiment of the present application. Figure 6 As shown, the inter-trial variance of the photoelectric signal waveform of the 12 Hz flickering stimulus is 120.6074. Figure 6 The content shown indicates that the flickering stimulation of the interactive interface of the VR device has poor consistency between trials and there are flickering and frame drops problems.

[0126] Taking the performance evaluation of the interactive interface of the VR device using the Unity Profiler tool as an example, the UnityProfiler tool is used to analyze the resource usage of the running VR device. Among them, the CPU running time of the VR device is 12.11ms, the GPU running time is 2.43ms, and the total memory usage is 8.56GB. The results show that the GPU and CPU usage of the VR device are too high and the memory usage is too large.

[0127] Based on the above performance evaluation results, the performance of the interactive interface of the VR device can be optimized. The performance optimization strategies include:

[0128] Optimize static objects: adjust the scene anti-aliasing level of the VR device's interactive interface to "4xMulti Sampling", set the multi-level-of-detail rendering mode, and enable the "Static" attribute of all static objects in the VR device's interactive interface. Set static batch processing for the interactive interface, and merge multiple objects with the Static attribute enabled into one object during the rendering process to complete the rendering task in one go.

[0129] Optimize dynamic objects: turn off the 3D and 2D collision detection functions for dynamic objects in the scene to reduce the simulation of the physical properties of dynamic objects. Set dynamic batch processing for the interactive interface to combine multiple dynamic objects with the same material and fewer vertices than required into a single object for a single rendering call to reduce the number of rendering calls by the CPU.

[0130] Optimize resources: Optimize the resources of the feedback video frames of the virtual drone in the interactive interface, appropriately reduce the resolution of the video images fed back in the background to compress the data file size and reduce memory usage.

[0131] After optimizing the performance of the VR device's interactive interface through the above methods, the photocell sensor and the Unity Profiler tool can be used to evaluate the performance of the VR device's interactive interface again.

[0132] in, Figure 7 The waveform diagram of the spectral information of the performance evaluation of the interactive interface of the VR device using the photocell sensor. Figure 7 As shown in the figure, the inter-trial variance of the photoelectric signal waveform of the 12Hz flickering stimulation is 0.0912. This result shows that the performance optimization of the interactive interface of the VR device improves the inter-trial consistency of the flickering stimulation of the interactive interface of the VR device.

[0133] The results of another performance evaluation of the VR device's interactive interface using the Unity Profiler tool showed that the CPU running time was 5.73ms, the GPU running time was 1.74ms, and the total memory usage was 2.12GB. This result shows that the GPU and CPU usage of the VR device is too high or the memory usage is too large. This result shows that optimizing the performance of the VR device's interactive interface reduces the VR device's GPU, CPU usage and memory usage.

[0134] Based on the above content and the same concept, the present application provides a display device for a brain-computer interaction interface. Figure 8 As shown, the device includes a communication module 801 and a processing module 802 .

[0135] Processing module 802 is used to generate a brain-computer interaction interface based on the acquired interface parameters, and the interface parameters are determined according to the positional relationship between the observation point and the display screen and the size of the display screen. Communication module 801 is used to obtain the target brain wave signal, which is generated when the user looks at the brain-computer interaction interface; processing module 802 is also used to determine one or more control instructions corresponding to the target brain wave signal according to the correspondence between the brain wave signal and the control instruction, and send one or more control instructions to the controlled device; communication module 801 is also used to receive status information from the controlled device, and the status information is used to indicate the operating status of the controlled device; processing module 802 is used to generate a target image according to the status information, and display the target image in the brain-computer interaction interface.

[0136] In one possible design, the interface parameters include: the pitch angle between the observation point and the display screen, the yaw angle between the observation point and the display screen, the horizontal distance between the observation point and the display screen, the height of the display screen, the width of the display screen, the vertical field of view angle of the brain-computer exchange interface, the horizontal field of view angle of the brain-computer exchange interface, the layer depth of the brain-computer exchange interface, the width of the brain-computer exchange interface, and the height of the brain-computer exchange interface.

[0137] In one possible design, the interface parameters satisfy:

[0138]

[0139] Scenesh =Object h ;

[0140] Scenes w =Object w ;

[0141] Fov v =Pitch;

[0142] Fov h =Yaw;

[0143] Position z =Distance;

[0144] Among them, Pitch represents the pitch angle between the observation point and the display screen, Yaw represents the yaw angle between the observation point and the display screen, and Object h Indicates the height of the display screen, Object w Indicates the width of the display, Distance indicates the horizontal distance between the observation point and the display, and Scenes w Indicates the width of the brain-computer interface, Scenes h Indicates the height of the brain-computer interface, Fov v Indicates the vertical field of view of the brain-computer interface, Fov h Indicates the horizontal field of view of the brain-computer interface, Position z Depth of layers representing the brain-computer interface.

[0145] In one possible design, the optimization strategy corresponding to the brain-computer interaction device is determined according to the correspondence between the performance parameters of the brain-computer interaction device and the optimization strategy. The processing module 802 is specifically used to determine the optimization strategy corresponding to the brain-computer interaction device from the correspondence between the performance parameters of the brain-computer interaction device and the optimization strategy according to the usage of the graphics processing unit GPU, central processing unit CPU or memory of the brain-computer interaction device.

[0146] In one possible design, before displaying the target image in the brain-computer interaction interface, the processing module 802 is specifically used to: determine the optimization strategy corresponding to the interaction device based on the correspondence between the performance parameters of the brain-computer interaction device and the optimization strategy, and the brain-computer interaction device is used to present the brain-computer interaction interface; optimize and process the target image according to the optimization strategy corresponding to the brain-computer interaction device, and display the optimized target image on the brain-computer interaction interface.

[0147] Based on the same inventive concept, an embodiment of the present application provides an electronic device, which can realize the functions of the device discussed above. Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown.

[0148] The electronic device in the embodiment of the present application may include a processor 901. The processor 901 is the control center of the device, and various interfaces and lines can be used to connect various parts of the device, by running or executing instructions stored in the memory 903 and calling data stored in the memory 903. Optionally, the processor 901 may include one or more processing units, and the processor 901 may integrate an application processor and a modem processor, wherein the application processor mainly processes operating systems and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 901. In some embodiments, the processor 901 and the memory 903 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on independent chips.

[0149] The processor 901 may be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The method steps disclosed in the embodiments of the present application may be directly executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0150] In the embodiment of the present application, the memory 903 stores instructions that can be executed by at least one processor 901. The at least one processor 901 can be used to execute the method steps disclosed in the embodiment of the present application by executing the instructions stored in the memory 903.

[0151] The memory 903 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 903 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (RAM), a static random access memory (SRAM), a programmable read-only memory (PROM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 903 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 903 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0152] In the embodiment of the present application, the device may further include a communication interface 902 , through which the electronic device may transmit data.

[0153] Optional, can be Fig. 9 The processor 901 (or the processor 901 and the communication interface 902) implements Figure 8 The processing module 802 and / or the communication module 801 shown, that is, the actions of the processing module 802 and / or the communication module 801 can be executed by the processor 901 (or the processor 901 and the communication interface 902).

[0154] Based on the same inventive concept, the embodiment of the present application also provides a computer-readable storage medium, which may store instructions. When the instructions are executed on a computer, the computer executes the operation steps provided in the above method embodiment. The computer-readable storage medium may be Fig. 9 The memory 903 is shown.

[0155] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0156] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0157] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0158] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby executing the program on the computer or other programmable device.

Claims

1. A method for displaying a brain-computer interaction interface, characterized in that: The method comprises: Generate a brain-computer interaction interface based on the acquired interface parameters, wherein the interface parameters are determined according to a positional relationship between an observation point and a display screen and a size of the display screen; Acquire a target brain wave signal, where the target brain wave signal is generated when the user looks at the brain-computer interaction interface; Determine one or more control instructions corresponding to the target EEG signal according to the correspondence between the EEG signal and the control instruction, and send the one or more control instructions to the controlled device; receiving status information from the controlled device, wherein the status information is used to indicate an operating status of the controlled device; A target image is generated according to the state information, and the target image is displayed in the brain-computer interaction interface.

2. The method according to claim 1, characterized in that The interface parameters include: the pitch angle between the observation point and the display screen, the yaw angle between the observation point and the display screen, the horizontal distance between the observation point and the display screen, the height of the display screen, the width of the display screen, the vertical field angle of the brain-computer exchange interface, the horizontal field angle of the brain-computer exchange interface, the layer depth of the brain-computer exchange interface, the width of the brain-computer exchange interface, and the height of the brain-computer exchange interface.

3. The method according to claim 2, characterized in that The interface parameters satisfy: Scenes h =Object h ; Scenes w =Object w ; Fov v =Pitch; Fov h =Yaw; Position z =Distance; Wherein, the Pitch represents the pitch angle between the observation point and the display screen, the Yaw represents the yaw angle between the observation point and the display screen, and the Object h Indicates the height of the display screen. w represents the width of the display screen, the Distance represents the horizontal distance between the observation point and the display, and the Scenes w Indicates the width of the brain-computer exchange interface, the Scenes h represents the height of the brain-computer interface, the Fov v represents the vertical field of view of the brain-computer interface, the Fov h represents the horizontal field of view of the brain-computer interface, the Position z Represents the layer depth of the brain-computer exchange interface.

4. The method according to claim 1, characterized in that The target brain wave signal is generated when the user gazes at a stimulation block in the brain-computer interaction interface, and the positional relationship between the stimulation block and the brain-computer interaction interface is the same as the positional relationship between the stimulation block and the display screen.

5. The method according to claim 1, characterized in that Before displaying the target image in the brain-computer interaction interface, the method further includes: Determining the optimization strategy corresponding to the brain-computer interaction device according to the corresponding relationship between the performance parameters of the brain-computer interaction device and the optimization strategy, wherein the brain-computer interaction device is used to present the brain-computer interaction interface; The target image is optimized and processed according to the optimization strategy corresponding to the brain-computer interaction device, and the optimized target image is displayed on the brain-computer interaction interface.

6. The method according to claim 5, characterized in that The step of determining the optimization strategy corresponding to the brain-computer interaction device according to the corresponding relationship between the performance parameters of the brain-computer interaction device and the optimization strategy includes: According to the usage of the graphics processing unit GPU, central processing unit CPU or memory of the brain-computer interaction device, the optimization strategy corresponding to the brain-computer interaction device is determined from the correspondence between the performance parameters of the brain-computer interaction device and the optimization strategy.

7. A display device for a brain-computer interaction interface, characterized in that: The device comprises: A processing module, used to generate a brain-computer interaction interface based on the acquired interface parameters, wherein the interface parameters are determined according to the positional relationship between the observation point and the display screen and the size of the display screen; A communication module, used to obtain a target brain wave signal, wherein the target brain wave signal is generated when the user looks at the brain-computer interaction interface; The processing module is further used to determine one or more control instructions corresponding to the target EEG signal according to the correspondence between the EEG signal and the control instructions, and send the one or more control instructions to the controlled device; The communication module is further used to receive status information from the controlled device, where the status information is used to indicate the operating status of the controlled device; The processing module is used to generate a target image according to the state information and display the target image in the brain-computer interaction interface.

8. The device according to claim 7, characterized in that The interface parameters include: the pitch angle between the observation point and the display screen, the yaw angle between the observation point and the display screen, the horizontal distance between the observation point and the display screen, the height of the display screen, the width of the display screen, the vertical field angle of the brain-computer exchange interface, the horizontal field angle of the brain-computer exchange interface, the layer depth of the brain-computer exchange interface, the width of the brain-computer exchange interface, and the height of the brain-computer exchange interface.

9. The device according to claim 8, characterized in that The interface parameters satisfy: Scenes h =Object h ; Scenes w =Object w ; Fov v =Pitch; Fov h =Yaw; Position z =Distance; Wherein, the Pitch represents the pitch angle between the observation point and the display screen, the Yaw represents the yaw angle between the observation point and the display screen, and the Object h Indicates the height of the display screen. w represents the width of the display screen, the Distance represents the horizontal distance between the observation point and the display, and the Scenes w Indicates the width of the brain-computer exchange interface, the Scenes h represents the height of the brain-computer interface, the Fov v represents the vertical field of view of the brain-computer interface, the Fov h represents the horizontal field of view of the brain-computer interface, the Position z Indicates the layer depth of the brain-computer exchange interface.

10. An electronic device, characterized in that: The electronic device comprises a processor, and the processor is used to implement the steps of any one of the methods as claimed in claims 1-6 when executing a computer program stored in a memory.

Images