A display method and device of a brain-computer interaction interface

Abstract

The application discloses a display method and device of a brain-computer interaction interface, and relates to the technical field of human-computer interaction, to improve the display effect of the brain-computer interaction interface. The method comprises the following steps: a first device generates a brain-computer interaction interface based on acquired interface parameters. The interface parameters are determined according to the positional relationship between an observation point and a display screen and the size of the display screen. The first device acquires a target brain wave signal. The target brain wave signal is generated when a user gazes at the brain-computer interaction interface. The first device determines one or more control instructions corresponding to the target brain wave signal according to the corresponding relationship between the brain wave signal and the control instruction, and sends the one or more control instructions to a controlled device. The first device receives state information from the controlled device. The state information is used for indicating the running state of the controlled device. The first device generates a target image according to the state information, and displays the target image in the brain-computer interaction interface.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction technology, and in particular to a method and apparatus for displaying a brain-computer interface. Background Technology

[0002] Brain-computer interface (BCI) refers to establishing a direct pathway between the human brain and an external device to collect electroencephalogram (EEG) signals, thereby enabling information transmission and control, without relying on peripheral nerves and muscles. Virtual reality (VR) is a technology that generates virtual worlds through computer technology, 3D imaging technology, multimedia simulation technology, and display technology. VR devices based on brain-computer interfaces can provide users with a better user experience.

[0003] However, the display effect of the brain-computer interface for controlling VR devices based on brainwave signals in existing technical solutions is poor and needs to be improved. Summary of the Invention

[0004] This invention relates to the field of human-computer interaction technology and provides a method, system, and device for displaying a brain-computer interface, thereby improving the display effect of the brain-computer interface.

[0005] In a first aspect, embodiments of this application provide a method for displaying a brain-computer interface, comprising: generating a brain-computer interface based on acquired interface parameters. The interface parameters are determined according to the positional relationship between an observation point and a display screen, and the size of the display screen. Acquiring a target brainwave signal. The target brainwave signal is generated when a user gazes at the brain-computer interface. Determining one or more control commands corresponding to the target brainwave signal based on the correspondence between the brainwave signal and control commands, and sending one or more control commands to a controlled device. Receiving status information from the controlled device. The status information is used to indicate the operating status of the controlled device. Generating a target image based on the status information, and displaying the target image in the brain-computer interface.

[0006] This method determines interface parameters based on the positional relationship between the observation point and the display screen, as well as the screen size, and generates a brain-computer interface based on these parameters. A target image is acquired and displayed on the brain-computer interface to ensure that the size of the target image displayed on the interface is the same as the size displayed on the screen, thereby improving the display effect and enhancing the user's visual experience.

[0007] In one possible design, the interface parameters include: the pitch angle between the observation point and the display screen, the yaw angle between the observation point and the display screen, the horizontal distance between the observation point and the display screen, the height of the display screen, the width of the display screen, the vertical field of view of the brain-computer interface, the horizontal field of view of the brain-computer interface, the layer depth of the brain-computer interface, the width of the brain-computer interface, and the height of the brain-computer interface.

[0008] Based on this design, the interface parameters of the brain-computer interface can be determined according to the positional relationship between the observation point and the display screen, as well as the parameter information of the display screen. This can improve the display effect of the brain-computer interface and enhance the user's visual experience.

[0009] In one possible design, the interface parameters satisfy:

[0010] ;

[0011] ;

[0012] ;

[0013] ;

[0014] ;

[0015] ;

[0016] in, This indicates the pitch angle between the observation point and the display screen. This indicates the yaw angle between the observation point and the display screen. Indicates the height of the display screen. Indicates the width of the display screen. This indicates the horizontal distance between the observation point and the monitor. Indicates the width of the brain-computer interface. Indicates the height of the brain-computer interface. This represents the vertical field of view of the brain-computer interface. This indicates the horizontal field of view of the brain-computer interface. This indicates the layer depth of the brain-computer interface.

[0017] In one possible design, the target EEG signal is generated when the user gazes at a stimulus block in the brain-computer interface, and the positional relationship between the stimulus block and the brain-computer interface is the same as the positional relationship between the stimulus block and the display screen.

[0018] In one possible design, before displaying the target image in the brain-computer interface, the process further includes: determining the optimization strategy corresponding to the brain-computer interface based on the correspondence between the performance parameters of the brain-computer interface and the optimization strategy, wherein the brain-computer interface is used to present the brain-computer interface; optimizing and processing the target image according to the optimization strategy corresponding to the brain-computer interface, and displaying the optimized target image on the brain-computer interface.

[0019] This design optimizes the image of the target object based on the current performance of the imaging device, avoiding problems such as dropped frames and stuttering, thereby improving the stability of the imaging device.

[0020] In one possible design, the optimization strategy for the brain-computer interface device is determined based on the correspondence between the performance parameters of the brain-computer interface device and the optimization strategy. This includes determining the optimization strategy for the brain-computer interface device based on the usage of the graphics processing unit (GPU), central processing unit (CPU), or memory of the brain-computer interface device.

[0021] Secondly, embodiments of this application also provide a display device for a brain-computer interface, including a communication module and a processing module. Wherein:

[0022] The processing module generates a brain-computer interface based on acquired interface parameters, which are determined by the positional relationship between the observation point and the display screen, as well as the screen size. The communication module acquires target EEG signals generated when the user gazes at the brain-computer interface. The processing module also determines one or more control commands corresponding to the target EEG signal based on the correspondence between the EEG signal and control commands, and sends these commands to the controlled device. Furthermore, the communication module receives status information from the controlled device, indicating its operational status. Finally, the processing module generates a target image based on the status information and displays it on the brain-computer interface.

[0023] In one possible design, the interface parameters include: the pitch angle between the observation point and the display screen, the yaw angle between the observation point and the display screen, the horizontal distance between the observation point and the display screen, the height of the display screen, the width of the display screen, the vertical field of view of the brain-computer interface, the horizontal field of view of the brain-computer interface, the layer depth of the brain-computer interface, the width of the brain-computer interface, and the height of the brain-computer interface.

[0024] In one possible design, the interface parameters satisfy:

[0025] ;

[0026] ;

[0027] ;

[0028] ;

[0029] ;

[0030] ;

[0031] in, This indicates the pitch angle between the observation point and the display screen. This indicates the yaw angle between the observation point and the display screen. Indicates the height of the display screen. Indicates the width of the display screen. This indicates the horizontal distance between the observation point and the monitor. Indicates the width of the brain-computer interface. Indicates the height of the brain-computer interface. This represents the vertical field of view of the brain-computer interface. This indicates the horizontal field of view of the brain-computer interface. This indicates the layer depth of the brain-computer interface.

[0032] In one possible design, the optimization strategy corresponding to the brain-computer interface device is determined based on the correspondence between the performance parameters of the brain-computer interface device and the optimization strategy. Specifically, the processing module is used to determine the optimization strategy corresponding to the brain-computer interface device based on the usage of the graphics processing unit (GPU), central processing unit (CPU), or memory of the brain-computer interface device.

[0033] In one possible design, before displaying the target image in the brain-computer interface, the processing module is specifically used to: determine the optimization strategy corresponding to the brain-computer interface based on the correspondence between the performance parameters of the brain-computer interface and the optimization strategy, and the brain-computer interface is used to present the brain-computer interface; optimize and process the target image according to the optimization strategy corresponding to the brain-computer interface, and display the optimized target image on the brain-computer interface.

[0034] Thirdly, embodiments of this application also provide an electronic device, the electronic device including a processor, the processor being configured to implement the first aspect and any of its designs when executing a computer program stored in a memory.

[0035] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of the first aspect and any of its designs.

[0036] The technical effects of the second to fourth aspects and any one of their designs can be found in the technical effects of the corresponding designs in the first aspect, and will not be repeated here. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a schematic diagram of the structure of a prior art brain-computer interface system;

[0039] Figure 2 A flowchart illustrating a method for displaying a brain-computer interface provided in an embodiment of this application;

[0040] Figure 3 This is a schematic diagram of the structure of a brain-computer interface system provided in an embodiment of this application;

[0041] Figure 4 This application provides a schematic diagram of the structure of a communication system according to an embodiment of the present application.

[0042] Figure 5 A brain-computer interface provided in the embodiments of this application;

[0043] Figure 6 A waveform diagram of spectral information provided in an embodiment of this application;

[0044] Figure 7 A waveform diagram of another spectral information provided in an embodiment of this application;

[0045] Figure 8 A schematic diagram of the structure of a display device for a brain-computer interface provided in an embodiment of this application;

[0046] Figure 9 This is a schematic diagram of an electronic device structure provided in an embodiment of this application. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, a detailed description of the application will be provided below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0048] The following section introduces existing solutions that combine brain-computer interfaces and VR devices.

[0049] Figure 1 This is a schematic diagram of the structure of a prior art brain-computer interface system. Figure 1 As shown, Figure 1 The brain-computer interface system shown includes a signal acquisition module, a signal processing module, a peripheral control module, and an interactive interface module.

[0050] Signal processing typically includes signal preprocessing, feature extraction, and classification. The peripheral control module can generate control commands based on preset strategies and send these commands to the corresponding devices.

[0051] The interactive interface module can receive visual feedback from other devices and display it on the interface. Current technologies typically present images on the interactive interface by placing flashing stimuli within a rich virtual scene. Alternatively, in a VR scene, flashing stimuli can be overlaid on the background video of the virtual scene.

[0052] However, the display effect of the brain-computer interface of VR devices controlled by brainwave signals in existing technical solutions is poor and needs to be improved.

[0053] To address the aforementioned shortcomings, this application provides a method and apparatus for displaying a brain-computer interface, thereby improving the display effect of the brain-computer interface.

[0054] The method employed in this application includes: a first device generating a brain-computer interface based on acquired interface parameters. The interface parameters are determined according to the positional relationship between the observation point and the display screen, and the size of the display screen. The first device acquires a target brainwave signal. The target brainwave signal is generated when the user gazes at the brain-computer interface. The first device determines one or more control commands corresponding to the target brainwave signal based on the correspondence between the brainwave signal and control commands, and sends one or more control commands to the controlled device. The first device receives status information from the controlled device. This status information indicates the operating status of the controlled device. The first device generates a target image based on the status information and displays the target image in the brain-computer interface.

[0055] This method determines interface parameters based on the positional relationship between the observation point and the display screen, as well as the screen size, and generates a brain-computer interface based on these parameters. A target image is acquired and displayed on the brain-computer interface to ensure that the size of the target image displayed on the interface is the same as the size displayed on the screen, thereby improving the display effect and enhancing the user's visual experience.

[0056] Furthermore, the first device may be included in a computer system for performing the method shown in this application, or it may be a processing device in a computer system for performing the method shown in this application, such as a processor or processing module, etc., which is not specifically limited in this application.

[0057] Figure 2 This is a flowchart illustrating a method for displaying a brain-computer interface according to an embodiment of this application. Taking a first device as the executing entity as an example, the process may include the following steps:

[0058] S201, the first device generates a brain-computer interface based on the acquired interface parameters. These interface parameters are determined according to the positional relationship between the observation point and the display screen, as well as the size of the display screen.

[0059] In one or more embodiments, the interface parameters include at least one of the following: the pitch angle between the observation point and the display screen, the yaw angle between the observation point and the display screen, the horizontal distance between the observation point and the display screen, the height of the display screen, the width of the display screen, the vertical field of view of the brain-computer interface, the horizontal field of view of the brain-computer interface, the layer depth of the brain-computer interface, the width of the brain-computer interface, and the height of the brain-computer interface. Optionally, the interface parameters also include the resolution of the display.

[0060] It is understood that an observation point can be the position where the user observes the display screen during testing. Alternatively, an observation point can be a reference point generated based on business requirements. The reference point can be a virtual location or a real location. This application does not impose specific limitations.

[0061] In one or more embodiments, the interface parameters satisfy:

[0062] ;

[0063] ;

[0064] ;

[0065] ;

[0066] ;

[0067] ;

[0068] in, This indicates the pitch angle between the observation point and the display screen. This indicates the yaw angle between the observation point and the display screen. Indicates the height of the display screen. Indicates the width of the display screen. This indicates the horizontal distance between the observation point and the monitor. Indicates the width of the brain-computer interface. Indicates the height of the brain-computer interface. This represents the vertical field of view of the brain-computer interface. This indicates the horizontal field of view of the brain-computer interface. This indicates the layer depth of the brain-computer interface.

[0069] Understandably, based on this embodiment, determining the interface parameters of the brain-computer interface according to the positional relationship between the observation point and the display screen, as well as the parameter information of the display screen, can improve the display effect of the brain-computer interface and enhance the user's visual experience.

[0070] S202, the first device acquires the target brainwave signal. The target brainwave signal is generated when the user gazes at the brain-computer interface.

[0071] As an example, the first device may include a module or apparatus with the function of acquiring brainwave signals, and the EEG device can acquire brainwave signals through its own module or apparatus. The first device can acquire brainwave signals generated by the user in real time. Furthermore, after acquiring brainwave signals through its own module or apparatus, the first device can also perform preprocessing such as filtering on the acquired brainwave signals to reduce noise signals in the brainwave signals.

[0072] As another example, the device used to collect brainwave signals can be operated by other devices. For example, Figure 3 This is a schematic diagram of the structure of a brain-computer interface system provided in an embodiment of this application, such as... Figure 3 As shown, the signal acquisition device is used to acquire brainwave signals. The signal acquisition device can be a portable device or a fixed device; this application does not specifically limit its use. The signal acquisition device can acquire brainwave signals generated by the user in real time and feed back the acquired brainwave signals to the first device. In this case, the signal acquisition device can be considered as a device independent of the first device, or a component within an independent device. The first device and the signal acquisition device can be connected via a wired interface and / or a wireless interface. Correspondingly, the signal acquisition device can send brainwave signals to the first device via the wired interface and / or wireless interface. The first device can receive brainwave signals from the signal acquisition device via the wired interface and / or wireless interface.

[0073] In one or more embodiments, the target EEG signal is generated when the user gazes at a stimulus block in a brain-computer interface (BCI), and the positional relationship between the stimulus block and the BCI is the same as the positional relationship between the stimulus block and the display screen. The stimulus block is a module that induces the user to generate corresponding EEG signals according to preset rules. The stimulus block can be a virtual module, such as a virtual light module displayed on the display screen or BCI. The stimulus block can also be a real device or module, such as a device or module independent of the display screen or BCI. For example, as... Figure 5 As shown, stimulation blocks 1, 2, 3, 4, 5, and 6 are distributed on the brain-computer interface. These stimulation blocks can be used to induce the user to generate corresponding brainwave signals. Specifically, at least one of stimulation blocks 1, 2, 3, 4, 5, and 6 can be used to induce the user to generate brainwave signals.

[0074] S203, the first device determines one or more control commands corresponding to the target EEG signal based on the correspondence between EEG signals and control commands, and sends one or more control commands to the controlled device. Correspondingly, the controlled device receives one or more control commands from the first device.

[0075] In one or more embodiments, the correspondence between brainwave signals and control commands can be pre-defined according to business requirements. Optionally, the correspondence between brainwave signals and control commands can be stored in a table. One brainwave signal can correspond to multiple control commands, or one brainwave signal can correspond to one control command. Furthermore, when the correspondence between brainwave signals and control commands does not include a target brainwave signal, the target brainwave signal can be added to the correspondence according to business requirements, and a correspondence between the target brainwave signal and one or more control commands can be constructed.

[0076] As an example, the correspondence between brainwave signals and control commands can also refer to the correspondence between the signal type of the brainwave signal and the control command. In other words, the correspondence between brainwave signals and control commands is determined based on the correspondence between the signal type of the brainwave signal and the control command. Accordingly, after acquiring the target brainwave signal, the first device can determine the signal type of the target brainwave signal based on the pre-stored correspondence between brainwave signals and signal types, and determine one or more control commands corresponding to the target brainwave signal based on the correspondence between signal type and control commands.

[0077] For example, such as Figure 3As shown, the EEG decoding module can decode brainwave information to obtain the signal type corresponding to the brainwave signal. The EEG decoding module can send the signal type corresponding to the brainwave information to the control plug-in via the transmission control protocol (TCP).

[0078] Optionally, the first device and the EEG decoding module can be integrated or exist as a single entity, or the first device may include the EEG decoding module. In this case, the first device and the EEG decoding module can be connected via an internal bus. Correspondingly, the first device can directly call the EEG decoding module to decode the brainwave signal, thereby obtaining the signal type corresponding to the brainwave signal. Further, the first device can determine one or more control commands corresponding to the brainwave signal from a pre-stored correspondence between signal types and control commands based on the signal type corresponding to the brainwave signal. (Continuing with...) Figure 3 For example, the control plugin can be represented as the first device, the decoding instruction can be represented as the signal type corresponding to the EEG signal, and the peripheral control instruction can be represented as the control instruction. The control plugin can determine the peripheral control instruction corresponding to the signal type based on the decoding instruction mapping relationship.

[0079] Table 1 is a correspondence table between signal types and control commands provided in an embodiment of this application. Specifically, Table 1 shows the correspondence between the signal types of brainwave signals and the control commands of the UAV. As shown in Table 1, when the signal type identifier of the target brainwave signal is 1, the corresponding control command is takeoff. When the signal type identifier of the target brainwave signal is 2, the corresponding control command is landing.

[0080]

[0081] Table 1

[0082] In one or more embodiments, the first device determines the instruction buffer corresponding to multiple control instructions based on the correspondence between control instructions and instruction buffers, and stores the multiple control instructions in the instruction buffer.

[0083] Understandably, the types of control instructions that the instruction buffer in this application can cache can correspond to the operating state of the controlled device. That is, when the operating state of the controlled device is different, the instruction buffer can cache different types of control instructions.

[0084] Optionally, the first device sends one or more control commands to the controlled device.

[0085] In one or more embodiments, after obtaining the target control command, the first device can send the target command to the controlled device. Correspondingly, the controlled device receives the target control command from the first device. The controlled device can then perform corresponding actions according to the target control command. Furthermore, after receiving the target control command, the controlled device can also send an acknowledgment message to the first device. For example, after receiving the target control command, the controlled device can send an acknowledgment character (ACK) to the first device. Figure 3 As shown, the flight equipment can be represented as the controlled equipment. After receiving the peripheral control command, the control plug-in can send the peripheral control command to the flight equipment.

[0086] In one or more embodiments, the first device and the interactive interface can be connected via named pipes to enable data communication between them. For example... Figure 3 As shown, the control plugin can send data flow feedback to the interactive interface module via named pipes. Similarly, the interactive interface module can send status data to the control plugin via named pipes.

[0087] Furthermore, before sending the data stream, the first device and the interactive interface can preprocess the data stream using data processing technology. Preprocessing operations can include compression, trimming, and encapsulation. For example, before sending the data stream to the interactive interface, the first device can compress the data stream to reduce its size. Similarly, before sending its runtime status data stream to the first device, the interactive interface can encapsulate the runtime status data stream to ensure data transmission integrity.

[0088] Furthermore, corresponding data buffers can be set in the named pipe for different types of data streams according to business needs. For example, when the interactive interface sends a running status data stream to the first device through the named pipe, the controlled device can store the running status data in the running status data buffer of the named pipe in FIFO form. Correspondingly, the first device can obtain the running status data stream of the interactive interface from the running status data buffer.

[0089] Understandably, the first device and the interactive interface can use the same programming language when communicating via named pipes. Alternatively, they can use two different programming languages. That is, when the first device sends a data stream to the interactive interface, the processes of data acquisition, processing, and writing can be encapsulated in one programming language. Correspondingly, the process of the interactive interface acquiring and reading the data stream can be decapsulated in another programming language. This application uses this communication method, which enables cross-language communication even if the first device and the interactive interface support different programming languages, without requiring format conversion between the two languages, thus improving communication efficiency between the two devices.

[0090] S204, the first device receives status information from the controlled device. Correspondingly, the controlled device sends its own status information to the first device. This status information indicates the operating status of the controlled device.

[0091] In one or more embodiments, after receiving one or more control commands from the first device, the controlled device can execute the corresponding control commands based on the one or more control commands. The specific control commands executed by the controlled device can be found in step S203, and will not be repeated here. After executing the control commands, the controlled device can send its own device status information to the first device. Alternatively, the controlled device can send its own device status information to the first device in real time during the execution of the control commands. Alternatively, the controlled device can send its own device status information to the first device according to preset rules. For example, the preset rules could be to send status information to the controlled device periodically. The controlled device can also send status information to the first device according to indication information used to indicate sending status information to the first device.

[0092] In one or more embodiments, the specific implementation method of the controlled device sending status information to the first device can be found in the specific implementation method of the first device sending control commands to the controlled device, which will not be repeated here.

[0093] S205, the first device generates a target image based on the status information and displays the target image in the brain-computer interface.

[0094] In one or more embodiments, after receiving status information from the controlled device, the first device can generate a corresponding target image based on that status information. For example, taking a flying device as an example, the flying device can report its current operating status as "flying" to the first device. Accordingly, the first device receives the status information from the flying device indicating that its operating status is "flying," and can generate an image showing that the flying status is "flying." As another example, if the flying device has a shooting function, it can report the images it captures to the first device in real time. Accordingly, the first device receives the images captured by the flying device and processes them based on parameter information from the brain-computer interface to obtain the target image corresponding to the captured image.

[0095] In one or more embodiments, after generating the corresponding target image, the first device can display the target image in a brain-computer interface.

[0096] In this application, in order to improve the user experience, the brain-computer interface can be optimized according to a pre-set strategy before displaying the target image in the brain-computer interface.

[0097] The following describes the optimization process for the brain-computer interface provided in this application:

[0098] In one or more embodiments, the first device can determine the optimization strategy corresponding to the brain-computer interface (BCI) based on the correspondence between the performance parameters of the BCI and the optimization strategy. The BCI is used to present a brain-computer interface. The first device can optimize the target image according to the optimization strategy corresponding to the BCI and display the optimized target image on the brain-computer interface.

[0099] In one or more embodiments, the first device determines the optimization strategy corresponding to the brain-computer interaction device based on the correspondence between the performance parameters of the brain-computer interaction device and the optimization strategy, including: determining the optimization strategy corresponding to the brain-computer interaction device based on the usage of the graphics processing unit (GPU), central processing unit (CPU), or memory of the brain-computer interaction device from the correspondence between the performance parameters of the brain-computer interaction device and the optimization strategy.

[0100] Optionally, the current performance parameters of the brain-computer interface device can be obtained by evaluating the device's performance using auxiliary means. These auxiliary means may include task managers, development engines, performance testing tools, and sensors. Optionally, the performance parameters of the imaging device may include graphics processing unit (GPU), central processing unit (CPU), and memory.

[0101] For example, frame rate and latency tests are performed on flickering stimuli and background using performance testing tools to observe the stable frame rate, minimum frame rate, frame rate drop, and operation latency of the imaging device during operation.

[0102] For example, based on the resource usage data in the Task Manager, it can be determined that the current performance bottleneck may be due to excessive GPU computing load, limited CPU processing power, or memory overload.

[0103] In one or more embodiments, the brain-computer interface device can determine the optimization strategy of the imaging device based on the correspondence between the performance parameters and optimization strategies of the brain-computer interface device, according to the usage of the brain-computer interface device's GPU, CPU, and memory. The usage of the brain-computer interface device's GPU, CPU, and memory can be, for example, the total amount and current usage of the imaging device's GPU, CPU, and memory, or the utilization rate of the imaging device's GPU, CPU, and memory.

[0104] As an example, the optimization strategy for brain-computer interface devices may include at least one of static object optimization strategy, dynamic object optimization strategy, and resource optimization strategy.

[0105] The static object optimization strategies include: setting low-level anti-aliasing, selecting low-dynamic-range rendering, simplifying shadow quality, and setting multi-level of detail (LOD) rendering to reduce GPU computational load and avoid flickering and frame drops. Additionally, during static processing, static batching can be used to merge multiple static objects into a single object, completing the rendering task in one go. This reduces the number of rendering passes, improves GPU rendering efficiency, further avoids flickering and frame drops, and enhances the reliability of the user interface.

[0106] Dynamic object optimization strategies include disabling dynamic collision detection, enabling offline collision detection mode, and reducing the simulation of physical properties for dynamic objects. These measures aim to reduce CPU workload, lower data stream latency, and prevent stuttering, thereby improving scene smoothness. Additionally, during dynamic processing, dynamic batching can be configured to combine multiple dynamic objects with the same material and fewer vertices than required into a single object for a single rendering call. This reduces the number of rendering calls, improves CPU utilization, and consequently reduces the frequency of flickering and frame drops.

[0107] Resource optimization strategies include reducing the number of objects in the frame, simplifying object models, simplifying object textures, and appropriately reducing the number of light sources to reduce memory usage and increase the space occupied by device feedback data streams and flickering stimuli, thus avoiding high latency in device feedback data streams and frame drops due to flickering stimuli. Additionally, when applied to scenarios with only device status feedback and a control interface, resource optimization can be performed on device status. For example, reducing the frequency of status feedback, lowering background image resolution, and converting image formats can compress data file sizes to balance visual effects and performance requirements, further reducing memory usage and avoiding high latency in device feedback data streams and frame drops due to flickering stimuli.

[0108] The method provided in this application is described below using Example 1 as an example. Example 1 uses a steady-state visual evoked potential-braincomputer interface (SSVEP-BCI) in a VR environment to control an AirSim virtual drone to take pictures and display the drone's pictures on an imaging device.

[0109] In this embodiment, the control plugin, EEG decoding module, VR device, and AirSim virtual drone are used as examples.

[0110] The control plugin can be hosted as a Dynamic Link Library (DLL) file. The DLL file can be created using C++. The control plugin and the EEG decoding module can communicate via a transmission control protocol. The communication link between the control plugin and the VR device can be a local direct communication link created using named pipes.

[0111] In this embodiment, the control plugin can call the C++ version image acquisition interface provided by AirSim to obtain the first-person view image of the AirSim virtual drone, and send the acquired frame image to the VR device through a named pipe.

[0112] The interactive interface program for VR devices can use C# to perform real-time null checks on named pipes. When the named pipe is not null, the image data in the named pipe is read and displayed as the interface background.

[0113] Understandably, the communication between the control plugin and the VR device runs independently in a multi-threaded loop to present images observed by the AirSim virtual drone in real time.

[0114] In this embodiment, the EEG decoding module can decode brainwave signals to obtain decoding instructions. After obtaining the decoding instructions, the EEG decoding module can send the decoding instructions to the control plugin via the TCP protocol. The control plugin can determine the control action corresponding to the decoding instructions based on the preset correspondence between decoding instructions and control actions.

[0115] After receiving a control action, the control plugin can store the action in a buffer based on the AirSim virtual drone's operating status. For example, when the "fly" flag, which indicates the AirSim virtual drone's flight status, is false, the control action can be stored in buffer A. When the "fly" flag, which indicates the AirSim virtual drone's flight status, is true, the control action can be stored in buffer B.

[0116] When the number of "takeoff" control actions in buffer A exceeds 3, the control plugin sends the "takeoff" control action to the AirSim virtual drone and sets the "fly" flag to true. Similarly, when the number of "landing" control actions in buffer A exceeds 3, the control plugin sends the "landing" control action to the AirSim virtual drone and sets the "fly" flag to false.

[0117] In this embodiment, buffer B is 4. A 5-dimensional matrix. The first dimension represents the velocity of the flight device in four degrees of freedom: forward / backward, left / right, up / down, and clockwise / counterclockwise rotation. The second dimension buffers five control commands. The control plugin sums all control actions in buffer B to obtain the target control action and sends it to the AirSim virtual drone.

[0118] Understandably, the control functions implemented by the control plugin can be encapsulated as a DDL control plugin, so that the VR device's interactive interface program can call the DDL control plugin in a multi-threaded manner. Figure 4 This is a schematic diagram of the structure of a communication system provided in an embodiment of this application. Figure 4 As shown, the functions of acquiring, processing, and writing data streams are encapsulated into a control plugin, and the data stream is sent to the named pipe in the form of a FIFO file. Correspondingly, the interactive interface can read the data stream from the named pipe and display its status.

[0119] In addition, this application can also utilize cross-platform visual size standardization technology to adjust the interactive interface of VR devices.

[0120] Figure 5 A brain-computer interface provided in the embodiments of this application, such as Figure 5 As shown, Figure 5 Taking AirSim virtual drones as the controlled device as an example, Figure 5 This is the target image generated by the first device based on real-time feedback from the drone. The height of the display screen is also considered. The width of the display screen is 33.57cm. The distance between the observation point and the display screen is 59.67 cm. The height is 70cm, and the screen resolution is 2560. 1440. The pitch angle of the VR imaging device. and yaw angle This can be determined based on the actual height of the display screen, the actual width of the display screen, and the distance between the observation point and the display screen. Specifically, the actual height of the display screen, the actual width of the display screen, the distance between the observation point and the display screen, and the pitch and yaw angles of the VR imaging device must satisfy the following:

[0121] ;

[0122] Furthermore, this embodiment utilizes cross-platform visual size standardization technology to ensure that the visual size of the interactive interface in the VR scene is consistent with that of the display screen. In other words, the parameters of the VR interactive interface can be adjusted according to the parameters of the display screen, thereby achieving the same resolution for the VR screen and the display screen (i.e., 2560). 1440), displaying layer width. Same width as the display screen (i.e.) Display layer height Same height as the display screen (i.e.) Vertical view of VR camera The same as the pitch angle of the observation point (i.e.) ), horizontal field of view of the VR camera The same yaw angle as the observation point (i.e.) ), Layer depth of the interactive interface The distance between the observation point and the display screen is the same (i.e.) ).

[0123] The interactive interface includes stimulus block 1, stimulus block 2, stimulus block 3, stimulus block 4, stimulus block 5, and stimulus block 6. The positional relationship between the stimulus blocks and the intersection interface is determined based on the positional relationship between the stimulus blocks and the display screen.

[0124] In this embodiment, a photocell sensor and the Unity Profiler tool can be used to evaluate the performance of the VR device's interactive interface.

[0125] Taking the performance evaluation of the VR device's interactive interface using a photocell sensor as an example, the photocell sensor is used to collect the spectral information of a specified stimulus block in the VR device's interactive interface, and the spectral information is plotted as a waveform. Figure 6 A waveform diagram of spectral information provided in an embodiment of this application. For example... Figure 6 As shown, the trial variance of the photoelectric signal waveform of the 12Hz flicker stimulus is 120.6074. Figure 6 The content shown indicates that the consistency of flickering stimuli between trials of the VR device's interactive interface is poor, and there is a problem of flickering and dropped frames.

[0126] Taking the performance evaluation of the VR device's interactive interface using the Unity Profiler tool as an example, the Unity Profiler tool was used to analyze the resource usage of the running VR device. The results showed that the VR device's CPU runtime was 12.11ms, GPU runtime was 2.43ms, and total memory usage was 8.56GB. These results indicate that the VR device's GPU and CPU utilization is too high, and its memory usage is excessive.

[0127] Based on the above performance evaluation results, the performance of the VR device's user interface can be optimized. Performance optimization strategies include:

[0128] Optimize static objects: Adjust the anti-aliasing level of the VR device's interactive interface to "4xMulti Sampling", set a multi-level-of-detail rendering mode, and enable the "Static" property for all static objects in the VR device's interactive interface. Set up static batch processing for the interactive interface to merge multiple objects with the Static property enabled into one object during the rendering process, so as to complete the rendering task in one go.

[0129] Optimize dynamic objects: Disable 3D and 2D collision detection for dynamic objects in the scene to reduce the simulation of their physical properties. Set up dynamic batch processing for the interactive interface, combining multiple dynamic objects with the same material and fewer than the required number of vertices into a single object for single rendering calls, thereby reducing the number of CPU rendering calls.

[0130] Resource optimization: Optimize the resources for the feedback video frames of the virtual drone in the interactive interface, and appropriately reduce the resolution of the feedback video images in the background to compress the data file size and reduce memory usage.

[0131] After optimizing the VR device's interface performance using the methods described above, the performance of the VR device's interface can be re-evaluated using a photocell sensor and the Unity Profiler tool.

[0132] in, Figure 7 This is a waveform diagram of the spectral information used to re-evaluate the performance of the VR device's interactive interface using a photocell sensor. (Example:) Figure 7 As shown, the trial-to-trial variance of the photoelectric signal waveform for a 12Hz flickering stimulus is 0.0912. This result indicates that performance optimization of the VR device's interface improves the trial-to-trial consistency of the flickering stimulus.

[0133] The performance evaluation of the VR device's interface using Unity Profiler showed a CPU runtime of 5.73ms, a GPU runtime of 1.74ms, and a total memory usage of 2.12GB. This indicates that the VR device's GPU and CPU utilization are too high, or its memory usage is excessive. Therefore, optimizing the VR device's interface performance reduces GPU, CPU utilization, and memory usage.

[0134] Based on the above content and the same concept, this application provides a display device for a brain-computer interface. For example... Figure 8 As shown, the device includes a communication module 801 and a processing module 802.

[0135] Processing module 802 is used to generate a brain-computer interface based on acquired interface parameters, which are determined according to the positional relationship between the observation point and the display screen, as well as the size of the display screen. Communication module 801 is used to acquire target EEG signals, which are generated when the user gazes at the brain-computer interface. Processing module 802 is also used to determine one or more control commands corresponding to the target EEG signal based on the correspondence between the EEG signal and control commands, and to send one or more control commands to the controlled device. Communication module 801 is also used to receive status information from the controlled device, which indicates the operating status of the controlled device. Processing module 802 is used to generate a target image based on the status information and display the target image in the brain-computer interface.

[0136] In one possible design, the interface parameters include: the pitch angle between the observation point and the display screen, the yaw angle between the observation point and the display screen, the horizontal distance between the observation point and the display screen, the height of the display screen, the width of the display screen, the vertical field of view of the brain-computer interface, the horizontal field of view of the brain-computer interface, the layer depth of the brain-computer interface, the width of the brain-computer interface, and the height of the brain-computer interface.

[0137] In one possible design, the interface parameters satisfy:

[0138] ;

[0139] ;

[0140] ;

[0141] ;

[0142] ;

[0143] ;

[0144] in, This indicates the pitch angle between the observation point and the display screen. This indicates the yaw angle between the observation point and the display screen. Indicates the height of the display screen. Indicates the width of the display screen. This indicates the horizontal distance between the observation point and the monitor. Indicates the width of the brain-computer interface. Indicates the height of the brain-computer interface. This represents the vertical field of view of the brain-computer interface. This indicates the horizontal field of view of the brain-computer interface. This indicates the layer depth of the brain-computer interface.

[0145] In one possible design, the optimization strategy corresponding to the brain-computer interface device is determined based on the correspondence between the performance parameters of the brain-computer interface device and the optimization strategy. Specifically, the processing module 802 is used to: determine the optimization strategy corresponding to the brain-computer interface device based on the usage of the graphics processing unit (GPU), central processing unit (CPU), or memory of the brain-computer interface device from the correspondence between the performance parameters of the brain-computer interface device and the optimization strategy.

[0146] In one possible design, before displaying the target image in the brain-computer interface, the processing module 802 is specifically used to: determine the optimization strategy corresponding to the brain-computer interface based on the correspondence between the performance parameters of the brain-computer interface and the optimization strategy, the brain-computer interface being used to present the brain-computer interface; optimize and process the target image according to the optimization strategy corresponding to the brain-computer interface, and display the optimized target image on the brain-computer interface.

[0147] Based on the same inventive concept, this application provides an electronic device that can realize the functions of the device described above. Figure 9 A schematic diagram of an electronic device structure provided in an embodiment of this application is shown.

[0148] The electronic device in this embodiment may include a processor 901. The processor 901 is the control center of the device, and can connect to various parts of the device via various interfaces and lines, executing instructions stored in a memory 903 and accessing data stored in the memory 903. Optionally, the processor 901 may include one or more processing units. The processor 901 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 901. In some embodiments, the processor 901 and the memory 903 may be implemented on the same chip; in some embodiments, they may be implemented separately on independent chips.

[0149] The processor 901 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The methods and steps disclosed in the embodiments of this application can be directly executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0150] In this embodiment of the application, the memory 903 stores instructions that can be executed by at least one processor 901. By executing the instructions stored in the memory 903, at least one processor 901 can perform the method steps disclosed in this embodiment of the application.

[0151] Memory 903, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 903 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 903 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In the embodiments of this application, memory 903 can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0152] In this embodiment of the application, the device may further include a communication interface 902, through which the electronic device can transmit data.

[0153] Optional, can be made by Figure 9 The processor 901 (or processor 901 and communication interface 902) shown implements... Figure 8 The processing module 802 and / or communication module 801 shown mean that the actions of the processing module 802 and / or communication module 801 can be executed by the processor 901 (or the processor 901 and the communication interface 902).

[0154] Based on the same inventive concept, embodiments of this application also provide a computer-readable storage medium that can store instructions, which, when executed on a computer, cause the computer to perform the operation steps provided in the above-described method embodiments. This computer-readable storage medium may be... Figure 9 The memory 903 shown.

[0155] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0156] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0157] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0158] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby in the computer or other programmable apparatus.

Claims

1. A method for displaying a brain-computer interface, characterized in that, The method includes: A brain-computer interface is generated based on the acquired interface parameters, which are determined according to the positional relationship between the observation point and the display screen and the size of the display screen. Acquire a target brainwave signal, wherein the target brainwave signal is generated when the user gazes at the brain-computer interface; Based on the correspondence between EEG signals and control commands, determine one or more control commands corresponding to the target EEG signal, and send the one or more control commands to the controlled device; Receive status information from the controlled device, the status information being used to indicate the operating status of the controlled device; A target image is generated based on the state information, and the target image is displayed on the brain-computer interface; The interface parameters include: the pitch angle between the observation point and the display screen, the yaw angle between the observation point and the display screen, the horizontal distance between the observation point and the display screen, the height of the display screen, the width of the display screen, the vertical field of view of the brain-computer interface, the horizontal field of view of the brain-computer interface, the layer depth of the brain-computer interface, the width of the brain-computer interface, and the height of the brain-computer interface. The interface parameters satisfy: ； ； ； ； ； ； Among them, the This indicates the pitch angle between the observation point and the display screen. This indicates the yaw angle between the observation point and the display screen. Indicates the height of the display screen, the This indicates the width of the display screen. This indicates the horizontal distance between the observation point and the display. This indicates the width of the brain-computer interface. The height of the brain-computer interface is indicated by the height of the interface. This represents the vertical field of view of the brain-computer interface. This represents the horizontal field of view of the brain-computer interface. This indicates the layer depth of the brain-computer interface.

2. The method as described in claim 1, characterized in that, The target EEG signal is generated when the user looks at the stimulation block in the brain-computer interface, and the positional relationship between the stimulation block and the brain-computer interface is the same as the positional relationship between the stimulation block and the display screen.

3. The method as described in claim 1, characterized in that, Before displaying the target image in the brain-computer interface, the method further includes: The optimization strategy corresponding to the brain-computer interface is determined based on the correspondence between the performance parameters of the brain-computer interface and the optimization strategy. The brain-computer interface is used to present the brain-computer interface. The target image is optimized and processed according to the optimization strategy corresponding to the brain-computer interface, and the optimized target image is displayed on the brain-computer interface.

4. The method as described in claim 3, characterized in that, The step of determining the optimization strategy corresponding to the brain-computer interface device based on the correspondence between the performance parameters of the brain-computer interface device and the optimization strategy includes: Based on the usage of the graphics processing unit (GPU), central processing unit (CPU), or memory of the brain-computer interface device, the corresponding optimization strategy for the brain-computer interface device is determined from the correspondence between the performance parameters of the brain-computer interface device and the optimization strategy.

5. A display device for a brain-computer interface, characterized in that, The device includes: The processing module is used to generate a brain-computer interface based on the acquired interface parameters, which are determined according to the positional relationship between the observation point and the display screen and the size of the display screen. The communication module is used to acquire target brainwave signals, which are generated when the user gazes at the brain-computer interface; The processing module is further configured to determine one or more control commands corresponding to the target EEG signal based on the correspondence between EEG signals and control commands, and send the one or more control commands to the controlled device. The communication module is also used to receive status information from the controlled device, the status information being used to indicate the operating status of the controlled device; The processing module is used to generate a target image based on the state information and display the target image on the brain-computer interface; The interface parameters include: the pitch angle between the observation point and the display screen, the yaw angle between the observation point and the display screen, the horizontal distance between the observation point and the display screen, the height of the display screen, the width of the display screen, the vertical field of view of the brain-computer interface, the horizontal field of view of the brain-computer interface, the layer depth of the brain-computer interface, the width of the brain-computer interface, and the height of the brain-computer interface. The interface parameters satisfy: ； ； ； ； ； ； Among them, the This indicates the pitch angle between the observation point and the display screen. This indicates the yaw angle between the observation point and the display screen. Indicates the height of the display screen, the This indicates the width of the display screen. This indicates the horizontal distance between the observation point and the display. This indicates the width of the brain-computer interface. The height of the brain-computer interface is indicated by the height of the interface. This represents the vertical field of view of the brain-computer interface. This represents the horizontal field of view of the brain-computer interface. This indicates the layer depth of the brain-computer interface.

6. The apparatus as claimed in claim 5, characterized in that, The target EEG signal is generated when the user looks at the stimulation block in the brain-computer interface, and the positional relationship between the stimulation block and the brain-computer interface is the same as the positional relationship between the stimulation block and the display screen.

7. The apparatus as claimed in claim 5, characterized in that, Before displaying the target image in the brain-computer interface, the processing module is further configured to: The optimization strategy corresponding to the brain-computer interface is determined based on the correspondence between the performance parameters of the brain-computer interface and the optimization strategy. The brain-computer interface is used to present the brain-computer interface. The target image is optimized and processed according to the optimization strategy corresponding to the brain-computer interface, and the optimized target image is displayed on the brain-computer interface.

8. The apparatus as claimed in claim 7, characterized in that, When determining the optimization strategy corresponding to the brain-computer interface device based on the correspondence between the performance parameters of the brain-computer interface device and the optimization strategy, the processing module is specifically used for: Based on the usage of the graphics processing unit (GPU), central processing unit (CPU), or memory of the brain-computer interface device, the corresponding optimization strategy for the brain-computer interface device is determined from the correspondence between the performance parameters of the brain-computer interface device and the optimization strategy.

9. An electronic device, characterized in that, The electronic device includes a processor that executes a computer program stored in a memory to implement the steps of the method as described in any one of claims 1-4.

Images