XR Control Method, Device, Medium and Electronic Device Based on Brain-Computer Interface

Abstract

The present application provides an XR control method, device, medium and electronic device based on a brain-computer interface. The method includes: displaying a control stimulus interface of a device to be controlled; collecting electroencephalogram signals of a user when watching the control stimulus interface; encoding the electroencephalogram signals within a preset time length to generate an encoding probability vector of the electroencephalogram signals; decoding the encoding probability vector to generate a steady-state command response intensity vector; performing hierarchical threshold processing on the steady-state command response intensity vector to obtain a characteristic-state command response intensity vector; performing filtering processing on the characteristic-state command response intensity vector or the steady-state command response intensity vector to obtain a command output probability vector, determining a preset command and the number of commands to be output; and transmitting the output preset command to the device to be controlled so that the device to be controlled executes the preset command. The present application can stably obtain the true control intention transmitted by the user through electroencephalogram signals, and generate a corresponding preset command to be transmitted to the device to be controlled to execute the preset command.

Description

Technical Field

[0001] This application belongs to the technical field of brain-computer interfaces, and relates to an XR control method, device, medium and electronic device based on a brain-computer interface. Background Art

[0002] A brain-computer interface (BCI) is a technology that establishes direct communication between the human brain and external devices. It realizes communication between the human brain and external devices by directly monitoring and analyzing signals generated by brain activities. Due to the complexity and dynamic changes of brain signals, traditional decoders are difficult to adapt to their real-time fluctuations, resulting in performance degradation. In addition, how to accurately capture effective signals from electroencephalogram (EEG) signals and identify and output the true control intention of users based on the effective signals is an urgent problem to be solved for realizing the function of controlling external devices through the human brain. Summary of the Invention

[0003] This application provides an XR control method, device, medium and electronic device based on a brain-computer interface, which is used to solve the problem of how to control external physical devices based on EEG signals.

[0004] In a first aspect, this application provides an XR control method based on a brain-computer interface, including: displaying a control stimulus interface of a device to be controlled; collecting EEG signals of a user when watching the control stimulus interface; encoding the EEG signals within a preset time length to generate an encoded probability vector of the EEG signals; decoding the encoded probability vector to generate a steady-state command response intensity vector; wherein each element of the steady-state command response intensity vector represents the response intensity of a preset command; the steady-state command response intensity vector is a set of response intensities of EEG signals at a certain refresh point corresponding to each preset command; performing hierarchical threshold processing on the steady-state command response intensity vector to obtain a characteristic-state command response intensity vector; performing filtering processing on the characteristic-state command response intensity vector or the steady-state command response intensity vector to obtain a command output probability vector, and determining the output preset command and the number of commands based on the command output probability vector; transmitting the output preset command to the device to be controlled, so that the device to be controlled executes the preset command.

[0005] In an implementation manner of the first aspect, the step of encoding the EEG signals within a preset time length to generate an encoded probability vector of the EEG signals includes: encoding the EEG signals within a preset time length to obtain an encoded vector; performing normalization processing on the encoded vector to obtain the encoded probability vector.

[0006] In an implementation of the first aspect, the step of decoding the encoded probability vector to generate a steady-state instruction response intensity vector includes: assigning the element value σj of the instruction j corresponding to the element with the largest value in the encoded probability vector as the steady-state threshold, and assigning the element values of the instructions corresponding to the elements with non-maximum values in the encoded probability vector as 0 to obtain the steady-state instruction response intensity vector; wherein, the maximum value in the steady-state threshold is less than or equal to 1 / 2 of the maximum value in the hierarchical thresholds.

[0007] In an implementation of the first aspect, the step of performing hierarchical threshold processing on the steady-state instruction response intensity vector to obtain a characteristic-state instruction response intensity vector includes: determining whether the steady-state threshold of the steady-state instruction response intensity vector is less than the minimum hierarchical threshold in the hierarchical thresholds. If so, the steady-state instruction response intensity vector does not belong to the characteristic-state instruction response intensity vector; if not, determining whether the steady-state threshold is greater than or equal to any intermediate hierarchical threshold in the hierarchical thresholds. If so, assign the steady-state threshold as the corresponding intermediate hierarchical threshold to obtain the characteristic-state instruction response intensity vector corresponding to the intermediate level, otherwise assign the steady-state threshold as the highest hierarchical threshold to obtain the characteristic-state instruction response intensity vector corresponding to the highest level.

[0008] In an implementation of the first aspect, the step of performing filtering processing on the characteristic-state instruction response intensity vector or the steady-state instruction response intensity to obtain a preset instruction output probability vector includes: if the steady-state instruction response intensity vector does not belong to the characteristic-state instruction response intensity vector, performing filtering processing on the steady-state instruction response intensity to obtain a preset instruction output probability vector; if the steady-state instruction response intensity vector belongs to the characteristic-state instruction response intensity vector, performing filtering processing on the characteristic-state instruction response intensity obtained after hierarchical threshold processing to obtain a preset instruction output probability vector.

[0009] In an implementation of the first aspect, the process of the normalization processing includes: , represents the encoded probability vector at time t, represents the encoded vector of the EEG signal at time t, represents the instruction response intensity value, i.e., the element value, of the i-th element in the encoded vector; n represents the number of columns of the encoded vector, i.e., the total number of instructions represented by the encoded vector.

[0010] In an implementation of the first aspect, the process of the normalization processing includes: , represents the encoded probability vector at time t, represents the encoded vector of the EEG signal at time t, The instruction response intensity value representing the i-th element in the encoding vector; n represents the number of columns of the encoding vector, that is, the total number of instructions represented by the encoding vector.

[0011] In one implementation of the first aspect, it further includes: calculating and obtaining the hierarchical threshold in the following manner ; , where and respectively represent the mean value and the standard deviation, * represents the set of normalized SSVEP response intensities represented by different instructions under multiple training sets, that is, the set of encoding probability vectors corresponding to the EEG signals respectively within multiple preset time lengths; m is the maximum value that the elements in the feature state instruction response intensity vector can take, m is an even number, the maximum value that the elements in the stable state instruction response intensity vector can take is m / 2, and the number of thresholds in the hierarchical threshold is m / 2; the hierarchical threshold is a set of; is the -th threshold in the hierarchical threshold, .

[0012] In one implementation of the first aspect, the steps of filtering the feature state instruction response intensity vector or the stable state instruction response intensity vector to obtain an instruction output probability vector and determining the preset instruction and the number of instructions to be output based on the instruction output probability vector include: setting the slope control parameter of each preset instruction; based on the slope control parameter of each preset instruction, iteratively calculating the encoding probability vector generated at the current generation refresh point and the feature state instruction response intensity vector or the stable state instruction response intensity vector generated at the previous generation refresh point to obtain the instruction output probability vector at the current generation refresh point; determining the preset instruction and the number of instructions to be output at the current generation refresh point based on the instruction output probability vector at the current generation refresh point.

[0013] In one implementation of the first aspect, it further includes: setting the output threshold of each preset instruction; determining the preset instruction and the number of instructions to be output at the current generation refresh point based on the output threshold of the preset instruction and the instruction output probability vector at the current generation refresh point.

[0014] In an implementation of the first aspect, it further includes: the slope control parameter includes an ascending control parameter and a descending control parameter; the ascending control parameter and the descending control parameter of each preset instruction are set, where the ascending control parameter is used to control the delay duration of the execution of the corresponding preset instruction, and the descending control parameter is used to control the exit duration of the execution of the corresponding preset instruction; during the iterative calculation process, the increase amplitude of the execution probability of the corresponding preset instruction is adjusted based on the ascending control parameter, and the decrease amplitude of the execution probability of the corresponding preset instruction is adjusted based on the descending control parameter.

[0015] In an implementation of the first aspect, it further includes: when multiple preset instructions are output, the multiple preset instructions are combined and output simultaneously.

[0016] In an implementation of the first aspect, it further includes: obtaining a user action sensing signal and an environmental device sensing signal; analyzing and processing the user action sensing signal and the environmental device sensing signal to identify the device selected by the user action as the device to be controlled; triggering the display of the control stimulation interface of the device to be controlled.

[0017] In a second aspect, an XR control device based on a brain-computer interface includes: a display module for displaying a control stimulation interface of a device to be controlled; an electroencephalogram acquisition module for acquiring electroencephalogram signals of a user when watching the control stimulation interface; a processing module communicatively connected to the display module and the electroencephalogram acquisition module, encoding the electroencephalogram signals within a preset time length to generate an encoded probability vector of the electroencephalogram signals; encoding the electroencephalogram signals within a preset time length to generate an encoded probability vector of the electroencephalogram signals; decoding the encoded probability vector to generate a steady-state instruction response intensity vector; where each element of the steady-state instruction response intensity vector represents the response intensity of a preset instruction; the steady-state instruction response intensity vector is a set of the response intensities of the electroencephalogram signals at a certain refresh point corresponding to each preset instruction; performing hierarchical threshold processing on the steady-state instruction response intensity vector to obtain a characteristic-state instruction response intensity vector; performing filtering processing on the characteristic-state instruction response intensity vector or the steady-state instruction response intensity vector to obtain an instruction output probability vector, and determining the output preset instruction and the number of instructions based on the instruction output probability vector; a communication module communicatively connected to the processing module for transmitting the output preset instruction to the device to be controlled, so that the device to be controlled executes the preset instruction.

[0018] In a third aspect, the present application provides an electronic device, including: one or more processors; and one or more memories, where computer-readable code is stored in the memories, and when the computer-readable code is run by the one or more processors, it implements the brain-computer interface-based XR control method as described in any one of the above.

[0019] Fourthly, the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the XR control method based on a brain-computer interface as described in any one of the above is implemented.

[0020] As described above, the XR control method, device, medium and electronic device based on a brain-computer interface of the present application have the following beneficial effects: The ITR (Information Transfer Rate) of the XR control method based on a brain-computer interface of the present application can reach 290.3 bit / s, which has a significant improvement compared with about 260 bit / s of other traditional methods.

[0021] In the present application, through the recognition and analysis of electroencephalogram signals, the true control intention transmitted by the user through electroencephalogram signals can be stably obtained, and the control intention is generated into corresponding control instructions and transmitted to the corresponding device to be controlled, so that the device to be controlled executes the control instructions of the user.

[0022] The present application only performs recognition and control based on electroencephalogram signals, without the participation of eye movement processing operations. This not only saves the computing power of the device, but also ensures the accurate and stable output of preset instructions through the encoding and decoding processing of electroencephalogram signals.

[0023] The present application is very suitable for patients who cannot independently control external objects through limb behaviors, facilitating their interaction with the outside world and improving their quality of life and possibilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It shows a schematic structural diagram of an implementation of the XR control device based on a brain-computer interface according to an embodiment of the present application.

[0025] Figure 2 It shows a schematic processing flow diagram of a processing component of the XR control device based on a brain-computer interface according to an embodiment of the present application.

[0026] Figure 3 It shows a second schematic processing flow diagram of a processing component of the XR control device based on a brain-computer interface according to an embodiment of the present application.

[0027] Figure 4 It shows a third schematic processing flow diagram of a processing component of the XR control device based on a brain-computer interface according to an embodiment of the present application.

[0028] Figure 5 It shows a fourth schematic processing flow diagram of a processing component of the XR control device based on a brain-computer interface according to an embodiment of the present application.

[0029] Figure 6Shown is another schematic diagram of the implementation structure of the XR control device based on the brain-computer interface according to the embodiments of the present application.

[0030] Figure 7 Shown is a schematic diagram of an implementation process of the brain-computer interface-based XR control method according to the embodiments of the present application.

[0031] Figure 8 Shown is a schematic diagram of an implementation structure of the electronic device according to the embodiments of the present application.

[0032] Figure 9 Shown is a schematic diagram of the ITR that can be achieved by the present application under different frequency band parameters.

[0033] Figure 10A Shown is a schematic diagram of the coding probability vectors of electroencephalogram signals of different acquisition channels within the same preset time length.

[0034] Figure 10B Shown as Figure 10A the schematic diagram of the preset instructions corresponding to the output of the coding probability vectors of the electroencephalogram signals in

[0035] Description of component labels

[0036] 100 XR Control Device Based on Brain-Computer Interface 110 Display Module 120 Brain Electrical Activity Acquisition Module 130 Processing Module 140 Communication Module 150 Sensor Module 800 Electronic Device 810 Processor 820 Memory 830 I / O Interface 840 Communication Module S201~S206 Step S301~S303 Step S401~S402 Step S501~S503 Step S710~S770 Step Detailed implementation manners

[0037] The following uses specific specific examples to illustrate the implementation manners of the present application. Those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in this specification. The present application can also be implemented or applied through other different specific implementation manners. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0038] It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present application in a schematic manner. Therefore, only the components related to the present application are shown in the diagrams, rather than being drawn according to the number, shape, and size of the components in actual implementation. The type, quantity, and ratio of each component in actual implementation can be arbitrarily changed, and the component layout type may also be more complex.

[0039] The core of Brain-Computer Interface (BCI) technology lies in the real-time monitoring and analysis of electroencephalogram (EEG) activity. EEG activity is the electrical signal generated by the activities of brain neurons, which can reflect an individual's thinking, emotions, and intentions. The principle of BCI technology is mainly based on the acquisition, processing, and decoding of EEG signals. First, through a non-invasive electrode array placed on the scalp, the electrical activities of the cerebral cortex can be captured. These signals are then amplified, filtered, and digitized for further analysis. Using signal processing techniques such as Fourier transform and machine learning algorithms, useful information can be extracted from the complex EEG background noise, and the signal features related to specific thinking patterns can be identified. Once these features are identified, they can be converted into control commands to operate external devices.

[0040] With the continuous development of Internet of Things (IoT) technology, the interconnection of devices in the IoT is a core component of the interconnection of all things. It refers to connecting physical devices to the Internet through various network technologies to achieve communication and data exchange between devices. In the IoT, physical devices can be sensors, controllers, smart home appliances, industrial devices, medical devices, etc. They collect data through embedded systems and sensors; physical devices are connected to the Internet through wired or wireless network technologies to achieve data transmission and communication; physical devices can send and receive data from each other to achieve collaborative work and intelligent control; physical device data is usually uploaded to the cloud for storage and processing, and users can remotely monitor and control devices through the cloud platform.

[0041] Based on BCI technology and IoT technology, this application proposes an implementation solution for an XR (Extended Reality) control method, device, medium, and electronic device based on a BCI. It can directly output data or instructions to any physical device in the IoT through EEG activity, realizing direct control of physical devices by the human brain. This control can be either close-range control or remote control. The control stimulation interface of the physical device is displayed in front of the user through a certain display device. The user generates EEG activity by viewing this control stimulation interface. At this time, the user's EEG signals are collected and analyzed to obtain the user's true control intention, and then the control instruction corresponding to the true control intention is obtained. This control instruction is output to the corresponding physical device to make it respond and execute the control instruction.

[0042] Next, the technical solutions in the embodiments of this application will be described in detail with reference to the accompanying drawings in the embodiments of this application.

[0043] As Figure 1As shown, it shows a schematic diagram of an implementation structure of an XR (Extended Reality) control device based on a brain-computer interface of the present application. The XR control device based on a brain-computer interface 100 includes: a display module 110, an EEG acquisition module 120, a processing module 130, and a communication module 140.

[0044] The display module 110 is configured to display a control stimulation interface of the device to be controlled.

[0045] In an implementation of the present application, the display module is a display component of the XR control device, which may include software and hardware. The control stimulation interface is a virtual display interface.

[0046] In another implementation of the present application, the display module may also be a display component external to the XR control device, such as an independent display that can be communicatively connected to the XR control device. The control stimulation interface is a GUI interface displayed on an independent display.

[0047] The EEG acquisition module 120 is configured to acquire EEG signals when the user views the control stimulation interface.

[0048] In one implementation of the present application, the EEG acquisition module is an EEG acquisition component that comes with the XR control device, and the other end of the acquisition component is connected to the user's brain to collect EEG signals.

[0049] In one implementation of the present application, the EEG acquisition module is an acquisition component external to the XR control device, which is connected to the XR control device through an interface, and transmits the collected EEG signals to the XR control device through the interface. Because EEG signals are easily affected by environmental interference, it is preferred to use a wired transmission interface to communicate with the XR control device. If the purpose of the present application can be better achieved when the EEG signal is connected to the XR control device through wireless communication, the use of a wireless communication interface can also be considered.

[0050] The processing module 130 is communicatively connected to the display module 110 and the electroencephalogram acquisition module 120, and is configured to encode the electroencephalogram signals within a preset time length to generate an encoded probability vector of the electroencephalogram signals; encode the electroencephalogram signals within a preset time length to generate an encoded probability vector of the electroencephalogram signals; decode the encoded probability vector to generate a steady-state command response intensity vector; wherein, each element of the steady-state command response intensity vector represents the response intensity of a preset command; the steady-state command response intensity vector is a set of response intensities of the electroencephalogram signals at a certain refresh point corresponding to each preset command; perform hierarchical threshold processing on the steady-state command response intensity vector to obtain a characteristic-state command response intensity vector; perform filtering processing on the characteristic-state command response intensity vector or the steady-state command response intensity vector to obtain a command output probability vector, and determine the output preset command and the number of commands based on the command output probability vector; the communication module 140 is communicatively connected to the processing module 130, and is configured to transmit the output preset command to the device to be controlled, so that the device to be controlled executes the preset command.

[0051] In this application, the XR (Extended Reality) control device based on the brain-computer interface refers to a real-virtual combined and human-computer interactive environment generated through brain-computer interface technology, computer technology, and wearable devices. XR (Extended Reality, extended reality) is a general term for various technologies such as AR (Augmented Reality, augmented reality), VR (Virtual Reality, virtual reality), and MR (Mixed Reality, mixed reality).

[0052] In an embodiment of this application, the XR (Extended Reality) control device based on the brain-computer interface is a VR (Virtual Reality, virtual reality) control device based on the brain-computer interface, which simulates a virtual world through an opaque head-mounted device, provides simulations of the user's vision, hearing and other senses, and creates a strong "immersive feeling" and "sense of presence".

[0053] The XR (Extended Reality) control device based on the brain-computer interface is an AR (Augmented Reality, augmented reality) control device based on the brain-computer interface, which superimposes the real environment and virtual information onto the same picture in real time, projects the virtual content into the real world, and is perceived by the human senses, so as to achieve a sensory experience beyond reality.

[0054] The XR (Extended Reality) control device based on a brain-computer interface is an MR (Mixed Reality) control device based on a brain-computer interface, which mixes the real world and the virtual world to generate a new visual environment that simultaneously includes physical entities and virtual information and must be "real-time".

[0055] In an implementation manner of the present application, the communication module can transmit the preset instruction to the device to be controlled through the Internet of Things, so that the device to be controlled executes the preset instruction.

[0056] In an implementation manner of the present application, the hardware structure of the communication component can be selected and designed according to different communication requirements. The communication component can be a wired communication component, which is suitable for stable and high-speed data transmission; the communication component can also be a wireless communication component, which provides flexibility and portability and is suitable for remote control or interaction.

[0057] Through the recognition and analysis of electroencephalogram signals, the present application can stably obtain the real control intention transmitted by the user through electroencephalogram signals, generate corresponding control instructions according to the control intention, and transmit them to the corresponding device to be controlled, so that the device to be controlled executes the user's control instructions.

[0058] The present application only performs recognition control based on electroencephalogram signals without the participation of eye movement processing operations, which not only saves the computing power of the device, but also ensures the accurate and stable output of preset instructions through the encoding and decoding processing of electroencephalogram signals.

[0059] The present application is very suitable for patients who cannot independently control external objects through limb behaviors, facilitating their interaction with the outside world and improving their quality of life and possibilities.

[0060] In an implementation manner of the present application, the step of the processing module encoding the electroencephalogram signals within a preset time length to generate an encoding probability vector of the electroencephalogram signals includes: the processing module encodes the electroencephalogram signals within a preset time length to obtain an encoding vector; the processing module normalizes the encoding vector to obtain the encoding probability vector.

[0061] In an implementation manner of the present application, the step of the processing module decoding the encoding probability vector to generate a steady-state instruction response intensity vector includes: the processing module assigns the element value σj of the instruction j corresponding to the element with the largest value in the encoding probability vector as the steady-state threshold, and assigns the element values of the instructions corresponding to the elements with non-maximum values in the encoding probability vector as 0 to obtain the steady-state instruction response intensity vector; wherein, the maximum value in the steady-state threshold is less than or equal to 1 / 2 of the maximum value in the hierarchical threshold.

[0062] In an implementation manner of the present application, as Figure 2 shown, the steps for the processing module to perform hierarchical threshold processing on the steady-state instruction response intensity vector to obtain the characteristic-state instruction response intensity vector include steps S201 to S205. Step S206 is the subsequent processing of steps S202, S204, and S205.

[0063] S201, the processing module determines whether the steady-state threshold of the steady-state instruction response intensity vector is less than the minimum level threshold in the hierarchical threshold;

[0064] S202, if so, then the steady-state instruction response intensity vector does not belong to the characteristic-state instruction response intensity vector;

[0065] S203, if not, then determine whether the steady-state threshold is greater than or equal to any intermediate level threshold in the hierarchical threshold;

[0066] S204, if so, then assign the steady-state threshold to the corresponding intermediate level threshold to obtain the characteristic-state instruction response intensity vector corresponding to the intermediate level;

[0067] S205, if not, then assign the steady-state threshold to the highest level threshold to obtain the characteristic-state instruction response intensity vector corresponding to the highest level.

[0068] S206, perform filtering processing on the characteristic-state instruction response intensity vector or the steady-state instruction response intensity vector to obtain an instruction output probability vector, and determine the preset instruction and the number of instructions to be output based on the instruction output probability vector.

[0069] In an implementation manner of the present application, the steps for the processing module to obtain the preset instruction output probability vector include: if the steady-state instruction response intensity vector does not belong to the characteristic-state instruction response intensity vector, then perform filtering processing on the steady-state instruction response intensity to obtain the preset instruction output probability vector; if the steady-state instruction response intensity vector belongs to the characteristic-state instruction response intensity vector, then perform filtering processing on the characteristic-state instruction response intensity obtained after hierarchical threshold processing to obtain the preset instruction output probability vector.

[0070] In an implementation manner of the present application, the hierarchical threshold is calculated in the following manner ;

[0071]

[0072] where and represent the mean and standard deviation respectively, *It represents the set of normalized SSVEP response intensities represented by different instructions under multiple training sets, that is, the set of coding probability vectors corresponding to the EEG signals within multiple preset time lengths; m is the maximum value that the elements in the eigenstate instruction response intensity vector can take, m is an even number, the maximum value that the elements in the steady-state instruction response intensity vector can take is m / 2, and the number of thresholds in the hierarchical threshold is m / 2; the hierarchical threshold is the set of; is the th threshold in the hierarchical threshold, .

[0073] For example: If the maximum value of the eigenstate instruction response intensity vector is 6, then there will be a total of 6 / 2 = 3 thresholds, and these three thresholds are the cases where i = 1, 2, and 3 respectively.

[0074] For example: Suppose the hierarchical threshold is [2, 4, 6, 8, 10], the minimum hierarchical threshold is 2, in addition, 2, 4, 6, and 8 can all be called intermediate hierarchical thresholds, and 10 is the highest hierarchical threshold.

[0075] Case 1: Assign the element value σj of the instruction j corresponding to the element with the largest value in the coding probability vector as 1, and assign the element values of the instructions corresponding to the elements with non-maximum values in the coding probability vector as 0, that is, obtain the steady-state instruction response intensity vector; since the element value σj is 1, which is less than the minimum hierarchical threshold 2 in the hierarchical threshold, the steady-state instruction response intensity vector does not belong to the eigenstate instruction response intensity vector.

[0076] Case 2: Assign the element value σj of the instruction j corresponding to the element with the largest value in the coding probability vector as 2, and assign the element values of the instructions corresponding to the elements with non-maximum values in the coding probability vector as 0, that is, obtain the steady-state instruction response intensity vector; since the element value σj is 2, which is greater than or equal to the minimum hierarchical threshold 2 in the intermediate hierarchical threshold of the hierarchical threshold, assign the element value σj of the instruction j corresponding to the element with the largest value in the steady-state instruction response intensity vector as the minimum hierarchical threshold 2, and obtain the eigenstate instruction response intensity vector.

[0077] Case 3: Assign the element value σj of the instruction j corresponding to the element with the largest value in the coding probability vector as 3, and assign the element values of the instructions corresponding to the elements with non-maximum values in the coding probability vector as 0, that is, obtain the steady-state instruction response intensity vector; since the element value σj is 3, which is greater than the minimum hierarchical threshold 2 in the intermediate hierarchical threshold of the hierarchical threshold, assign the element value σj of the instruction j corresponding to the element with the largest value in the steady-state instruction response intensity vector as the upper-level threshold 4, and obtain the eigenstate instruction response intensity vector.

[0078] In an implementation manner of the present application, the normalization process is as follows:

[0079]

[0080] represents the encoding probability vector at time t, represents the encoding vector of the EEG signal at time t, represents the instruction response intensity value of the i-th element in the encoding vector, that is, the element value; n represents the number of columns of the encoding vector, that is, the total number of instructions represented by the encoding vector.

[0081] In an implementation manner of the present application, the process of the other normalization is as follows:

[0082]

[0083] represents the encoding probability vector at time t, represents the encoding vector of the EEG signal at time t, represents the instruction response intensity value of the i-th element in the encoding vector, that is, the element value; n represents the number of columns of the encoding vector, that is, the total number of instructions represented by the encoding vector.

[0084] In an implementation manner of the present application, the processor module obtains the hierarchical threshold through the following method ;

[0085]

[0086] Among them, and respectively represent the mean value and the standard deviation, * represents the set of normalized SSVEP response intensities represented by different instructions under multiple training sets, that is, the set of encoding probability vectors corresponding to the EEG signals within multiple preset time lengths; m is the maximum value that the elements in the feature state instruction response intensity vector can take, m is an even number, the maximum value that the elements in the stable state instruction response intensity vector can take is m / 2, and the number of thresholds in the hierarchical threshold is m / 2; the hierarchical threshold is a set of; is the -th threshold in the hierarchical threshold, . In this implementation manner, * is different from ** in the normalization process, but it is also clear.

[0087] In one implementation of the present application, as Figure 3 shown, the processing module controls the output of the preset instructions in the following manner, including steps S301 to S303.

[0088] S301, the processing module sets the slope control parameters of each of the preset instructions;

[0089] S302, the processing module performs iterative calculations on the coding probability vector generated at the current generation refresh point and the eigenstate instruction response intensity vector or the steady-state instruction response intensity vector generated at the previous generation refresh point based on the slope control parameters of each of the preset instructions, to obtain the instruction output probability vector at the current generation refresh point;

[0090] S303, the processing module determines the preset instructions and the number of instructions output at the current generation refresh point based on the instruction output probability vector at the current generation refresh point.

[0091] In one implementation of the present application, as Figure 4 shown, the processing module controls the output of the preset instructions in the following manner, including steps S401 to S402.

[0092] S401, the processing module sets the output thresholds of each of the preset instructions;

[0093] S402, the processing module determines the preset instructions and the number of instructions output at the current generation refresh point based on the output thresholds of the preset instructions and the instruction output probability vector at the current generation refresh point.

[0094] In one implementation of the present application, as Figure 5 shown, the processing module controls the output of the preset instructions in the following manner, including steps S501 to S503.

[0095] S501, the slope control parameters set by the processing module include an ascending control parameter and a descending control parameter;

[0096] S502, the processing module sets the ascending control parameter and the descending control parameter of each of the preset instructions, where the ascending control parameter is used to control the delay duration of the execution of the corresponding preset instruction, and the descending control parameter is used to control the exit duration of the execution of the corresponding preset instruction;

[0097] S503, during the iterative calculation process, the processing module adjusts the increasing amplitude of the execution probability of the corresponding preset instruction based on the ascending control parameter, and adjusts the decreasing amplitude of the execution probability of the corresponding preset instruction based on the descending control parameter.

[0098] In an implementation of the present application, when there are multiple preset instructions to be output, the processing module combines and outputs the multiple preset instructions simultaneously.

[0099] In an implementation of the present application, as Figure 6 shown, the brain-computer interface-based XR control device 100 further includes a sensor module 150. The sensor module 150 is configured to sense the user's actions to obtain action sensing signals and sense objects in the environment to obtain object sensing signals. The processing module 130 analyzes and processes the action sensing signals and object sensing signals, identifies the object selected by the user as the device to be controlled, and triggers the display module 110 to display the control stimulation interface of the device to be controlled.

[0100] The brain-computer interface-based XR (extended reality) control device provided by the embodiments of the present application can implement the brain-computer interface-based XR (extended reality) control method described in the present application. However, the implementation device of the brain-computer interface-based XR (extended reality) control method described in the present application includes, but is not limited to, the structure of the brain-computer interface-based XR (extended reality) control device listed in this embodiment. Any structural deformation and replacement of the prior art made according to the principles of the present application are included in the protection scope of the present application.

[0101] As Figure 7 shown, it shows a schematic diagram of an implementation process of the brain-computer interface-based XR (extended reality) control method of the present application, including steps S710 to S770.

[0102] S710, display the control stimulation interface of the device to be controlled.

[0103] S720, collect the electroencephalogram signals of the user when watching the control stimulation interface.

[0104] S730, encode the electroencephalogram signals within a preset time length to generate an encoded probability vector of the electroencephalogram signals.

[0105] S740, decode the encoded probability vector to generate a steady-state instruction response intensity vector; wherein, each element of the steady-state instruction response intensity vector represents the response intensity of a preset instruction; the steady-state instruction response intensity vector is a set of response intensities of the electroencephalogram signals at a certain refresh point corresponding to each preset instruction.

[0106] S750, perform hierarchical threshold processing on the steady-state instruction response intensity vector to obtain a characteristic-state instruction response intensity vector.

[0107] In an implementation of the present application, the step of the processing module encoding the EEG signals within a preset time length to generate an encoding probability vector of the EEG signals includes: the processing module encoding the EEG signals within a preset time length to obtain an encoding vector; the processing module performing normalization processing on the encoding vector to obtain the encoding probability vector.

[0108] In an implementation of the present application, the step of the processing module decoding the encoding probability vector to generate a steady-state instruction response intensity vector includes: the processing module assigning the element value σj of the instruction j corresponding to the element with the largest value in the encoding probability vector as the steady-state threshold, and assigning the element values of the instructions corresponding to the elements with non-maximum values in the encoding probability vector as 0 to obtain the steady-state instruction response intensity vector; wherein, the maximum value in the steady-state threshold is less than or equal to 1 / 2 of the maximum value in the hierarchical threshold.

[0109] In an implementation of the present application, as Figure 2 shown, the step of the processing module performing hierarchical threshold processing on the steady-state instruction response intensity vector to obtain a characteristic-state instruction response intensity vector includes steps S201 to S205. Step S206 is the subsequent processing of steps S202, S204, and S205.

[0110] S201, the processing module determines whether the steady-state threshold of the steady-state instruction response intensity vector is less than the minimum hierarchical threshold in the hierarchical threshold;

[0111] S202, if so, then the steady-state instruction response intensity vector does not belong to the characteristic-state instruction response intensity vector;

[0112] S203, if not, then determine whether the steady-state threshold is greater than or equal to any intermediate hierarchical threshold in the hierarchical threshold;

[0113] S204, if so, then assign the steady-state threshold as the corresponding intermediate hierarchical threshold to obtain a characteristic-state instruction response intensity vector corresponding to the intermediate level;

[0114] S205, if not, then assign the steady-state threshold as the highest hierarchical threshold to obtain a characteristic-state instruction response intensity vector corresponding to the highest level.

[0115] S206, perform filtering processing on the characteristic-state instruction response intensity vector or the steady-state instruction response intensity vector to obtain an instruction output probability vector, and determine the preset instructions and the number of instructions to be output based on the instruction output probability vector.

[0116] For example, assign the element value σj of the instruction j corresponding to the element with the largest value in the encoding probability vector to 3, and assign the element values of the instructions corresponding to the elements with non-largest values in the encoding probability vector to 0, that is, obtain the steady-state instruction response intensity vector [3, 0, 0, 0, 0, 0, 0, 0, 0, 0].

[0117] When the element value σj is greater than or equal to the minimum level threshold 2 and the element value σj is between 2 and 4 in the hierarchical threshold [2, 4, 6], the element value σj can be assigned to 4, and the steady-state instruction response intensity vector [3, 0, 0, 0, 0, 0, 0, 0, 0, 0] can be classified as the characteristic-state instruction response intensity vector [4, 0, 0, 0, 0, 0, 0, 0, 0, 0]. Among them, the maximum value in the steady-state threshold is less than or equal to 1 / 2 of the maximum value in the hierarchical threshold. The hierarchical threshold can be obtained by the following calculation method ; ;

[0118] Among them, and represent the mean and standard deviation respectively, * represents the set of normalized SSVEP response intensities represented by different instructions under multiple training sets, that is, the set of encoding probability vectors corresponding to the electroencephalogram signals within multiple preset time lengths; m is the maximum value that the elements in the characteristic-state instruction response intensity vector can take, m is an even number, the maximum value that the elements in the steady-state instruction response intensity vector can take is m / 2, and the number of thresholds in the hierarchical threshold is m / 2; the hierarchical threshold is the set of; is the th threshold in the hierarchical threshold, .

[0119] S760, perform filtering processing on the characteristic-state instruction response intensity vector or the steady-state instruction response intensity vector to obtain an instruction output probability vector, and determine the preset instruction and the number of instructions to be output based on the instruction output probability vector.

[0120] S770, transmit the preset instruction output to the device to be controlled, so that the device to be controlled executes the preset instruction.

[0121] In an embodiment of the present application, the XR (extended reality) control method based on a brain-computer interface described in the embodiments of the present application further includes steps such as Figure 3 , Figure 4 and Figure 5 described above, which will not be repeated here.

[0122] The protection scope of the XR control method based on the brain-computer interface described in the embodiments of this application is not limited to the execution order of the steps listed in this embodiment. Any solution achieved by adding or reducing steps of the prior art or replacing steps according to the principles of this application is included in the protection scope of this application.

[0123] In one implementation manner of this application, the embodiments of this application provide an electronic device, such as Figure 8 shown, the electronic device 800 includes one or more processors 810 and one or more memories 820; computer-readable code is stored in the memory, and when the computer-readable code is run by the one or more processors, the XR control method based on the brain-computer interface described above is implemented. In addition, the electronic device may further include conventional electronic devices such as an I / O interface 830, a communication module 840, etc., which will not be elaborated here.

[0124] See Figure 9 shown, which shows a schematic diagram of the ITR (Information Transfer Rate) that this application can achieve under different frequency band parameters. Among them, Figure 9 the abscissa represents the value of the slope control parameter of the filter, the ordinate represents the value of the output threshold of the preset instruction, and the numbers in each color block represent the ITR under different abscissa and ordinate parameters. From Figure 9 it can be seen that the ITR of the XR control method based on the brain-computer interface described in this application can reach 290.3 bit / s, which has a significant effect improvement compared with about 260 bit / s of other traditional methods.

[0125] See Figure 10A and Figure 10B shown, which shows the continuous control effect diagram implemented by this application.

[0126] Figure 10A Among them, the curves of different colors represent the coding probability vectors of the electroencephalogram signals under different frequency band parameters. The abscissa represents time, and the ordinate represents the value of the coding vector. Figure 10A Among (a), (b), and (c) in represent the coding probability vectors of the electroencephalogram signals of different acquisition channels within the same preset time length.

[0127] Figure 10B Among them, the curves of different colors represent the preset instructions corresponding to the output of the electroencephalogram signals under different frequency band parameters. The abscissa represents time, and the ordinate represents whether to output the preset instruction. Figure 10B Among (a), (b), and (c) in represent the preset instructions corresponding to the output of the electroencephalogram signals of different acquisition channels within the same preset time length, and Figure 10AThe (a), (b), and (c) in it correspond one by one. Each color represents a preset instruction, and different colors represent different preset instructions. When the preset instruction is not 0, it is output at the current point.

[0128] Figure 10B The meanings represented essentially have only two values: "output" and "not output". For the convenience of this application, different heights are set for different instructions. When the value is 0, it is "not output", and when the value is not 0, it is "output".

[0129] Figure 10A and Figure 10B The topmost red and black solid lines of represent the time when the stimulus starts and the time when the stimulus ends respectively, and the dashed line represents the time at the start / end plus the trigger time required for the SSVEP response.

[0130] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the XR control method based on a brain-computer interface as described above in the present application is implemented.

[0131] Those of ordinary skill in the art can understand that all or part of the steps in the method of the above embodiments can be completed by instructing a processor through a program. The program can be stored in a computer-readable storage medium, and the storage medium is a non-transitory medium, such as a random access memory, a read-only memory, a flash memory, a hard disk, a solid-state drive, a magnetic tape, a floppy disk, an optical disc, and any combination thereof. The above storage medium can be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a digital video disc (DVD)), or a semiconductor medium (such as a solid-state disk (SSD)), etc.

[0132] The descriptions of the processes or structures corresponding to the above respective drawings have their own emphases. For parts not detailed in a certain process or structure, reference can be made to the relevant descriptions of other processes or structures.

[0133] In an implementation of the present application, embodiments of the present application may further provide a computer program product, which includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website, computer, or data center to another website, computer, or data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or a wireless manner (such as infrared, wireless, microwave, etc.). When the computer program product is executed by a computer, the computer executes the method described in the foregoing method embodiments. The computer program product may be a software installation package. In the case where the foregoing method is required, the computer program product may be downloaded and executed on the computer.

[0134] In several embodiments provided by the present application, it should be understood that the disclosed system, device, or method may be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules / units is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple modules or units may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections between each other may be through some interfaces, indirect couplings or communication connections of devices or modules or units, and may be in electrical, mechanical, or other forms.

[0135] Those of ordinary skill in the art should also further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described according to their functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. A person skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0136] The above embodiments merely illustrate the principles and effects of the present application, rather than limiting the present application. Any person familiar with this technology may modify or change the above embodiments without departing from the spirit and scope of the present application. Therefore, all equivalent modifications or changes completed by those with ordinary knowledge in the technical field to which the present application pertains without departing from the spirit and technical ideas disclosed by the present application should still be covered by the claims of the present application.

Claims

1. An XR control method based on a brain-computer interface, characterized in that, including: displaying a control stimulus interface of a device to be controlled; collecting electroencephalogram (EEG) signals of a user while the user is viewing the control stimulus interface; encoding the EEG signals within a preset time length to generate an encoded probability vector of the EEG signals; decoding the encoded probability vector to generate a steady-state command response intensity vector; wherein each element of the steady-state command response intensity vector represents the response intensity of a preset command; the steady-state command response intensity vector is a set of response intensities of the EEG signals at a certain refresh point corresponding to each preset command; performing hierarchical threshold processing on the steady-state command response intensity vector to obtain a characteristic-state command response intensity vector; performing filtering processing on the characteristic-state command response intensity vector or the steady-state command response intensity vector to obtain a command output probability vector, and determining the preset command and the number of commands to be output based on the command output probability vector; transmitting the output preset command to the device to be controlled, so that the device to be controlled executes the preset command.

2. The XR control method based on a brain-computer interface according to claim 1, wherein The step of encoding the EEG signals within a preset time length to generate an encoded probability vector of the EEG signals includes: encoding the EEG signals within a preset time length to obtain an encoded vector; performing normalization processing on the encoded vector to obtain the encoded probability vector.

3. The XR control method based on a brain-computer interface according to claim 1, wherein The step of decoding the encoded probability vector to generate a steady-state command response intensity vector includes: assigning the element value σj of the command j corresponding to the element with the largest value in the encoded probability vector as the steady-state threshold, and assigning the element values of the commands corresponding to the elements with non-maximum values in the encoded probability vector as 0, to obtain the steady-state command response intensity vector; wherein the maximum value in the steady-state threshold is less than or equal to 1 / 2 of the maximum value in the hierarchical threshold.

4. The XR control method based on a brain-computer interface according to claim 3, wherein The step of performing hierarchical threshold processing on the steady-state command response intensity vector to obtain a characteristic-state command response intensity vector includes: judging whether the steady-state threshold of the steady-state command response intensity vector is less than the minimum hierarchical threshold in the hierarchical threshold, if so, then the steady-state command response intensity vector does not belong to the characteristic-state command response intensity vector; if not, then judging whether the steady-state threshold is greater than or equal to any intermediate hierarchical threshold in the hierarchical threshold, if so, then assigning the steady-state threshold as the corresponding intermediate hierarchical threshold to obtain the characteristic-state command response intensity vector of the corresponding intermediate level, otherwise assigning the steady-state threshold as the highest hierarchical threshold to obtain the characteristic-state command response intensity vector of the corresponding highest level.

5. The XR control method based on a brain-computer interface according to claim 3, wherein The step of performing filtering processing on the characteristic-state command response intensity vector or the steady-state command response intensity to obtain a preset command output probability vector includes: if the steady-state command response intensity vector does not belong to the characteristic-state command response intensity vector, then performing filtering processing on the steady-state command response intensity to obtain a preset command output probability vector; if the steady-state command response intensity vector belongs to the characteristic-state command response intensity vector, then performing filtering processing on the characteristic-state command response intensity obtained after hierarchical threshold processing to obtain a preset command output probability vector.

6. The XR control method based on a brain-computer interface according to claim 2, wherein The normalization process includes: , represents the encoding probability vector at time t, represents the encoding vector of the EEG signal at time t, represents the instruction response intensity value, i.e., the element value, of the i-th element in the encoding vector; n represents the number of columns of the encoding vector, i.e., the total number of instructions represented by the encoding vector.

7. The XR control method based on a brain-computer interface according to claim 2, characterized in that, The process of the normalization processing includes: , represents the encoding probability vector at time t, represents the encoding vector of the EEG signal at time t, represents the instruction response intensity value of the i-th element in the encoding vector, that is, the element value; n represents the number of columns of the encoding vector, that is, the total number of instructions represented by the encoding vector.

8. The XR control method based on a brain-computer interface according to claim 1, wherein It also includes: The hierarchical threshold is obtained by calculation in the following manner ; , where and represent the mean value and the standard deviation respectively, * represents the set of normalized SSVEP response intensities represented by different instructions under multiple training sets, that is, the set of coding probability vectors corresponding to the EEG signals within multiple preset time lengths; m is the maximum value that the elements in the eigenstate instruction response intensity vector can take, m is an even number, the maximum value that the elements in the steady-state instruction response intensity vector can take is m / 2, and the number of thresholds in the hierarchical threshold is m / 2; the hierarchical threshold is a set; is the th threshold in the hierarchical threshold, .

9. The XR control method based on a brain-computer interface according to claim 1, wherein The steps of filtering the eigenstate instruction response intensity vector or the steady-state instruction response intensity vector to obtain an instruction output probability vector and determining the preset instructions and the number of instructions to be output based on the instruction output probability vector include: Setting the slope control parameters for each of the preset instructions; Based on the slope control parameters of each of the preset instructions, iteratively calculating the encoding probability vector generated at the current generation refresh point with the eigenstate instruction response intensity vector or the steady-state instruction response intensity vector generated at the previous generation refresh point to obtain the instruction output probability vector at the current generation refresh point; Based on the instruction output probability vector at the current generation refresh point, determining the preset instructions and the number of instructions to be output at the current generation refresh point.

10. The XR control method based on a brain-computer interface according to claim 9, wherein Further included are: Setting the output thresholds for each of the preset instructions; Based on the output thresholds of the preset instructions and the instruction output probability vector at the current generation refresh point, determining the preset instructions and the number of instructions to be output at the current generation refresh point.

11. The XR control method based on a brain-computer interface according to claim 9, wherein Further included are: The slope control parameter includes an ascending control parameter and a descending control parameter; Setting the ascending control parameter and the descending control parameter for each of the preset instructions, where the ascending control parameter is used to control the delay duration of the execution of the corresponding preset instruction, and the descending control parameter is used to control the exit duration of the execution of the corresponding preset instruction; During the iterative calculation process, adjusting the increasing amplitude of the execution probability of the corresponding preset instruction based on the ascending control parameter, and adjusting the decreasing amplitude of the execution probability of the corresponding preset instruction based on the descending control parameter.

12. The XR control method based on a brain-computer interface according to claim 1, wherein, Further included are: When there are multiple preset instructions to be output, combining and outputting the multiple preset instructions simultaneously.

13. The XR control method based on a brain-computer interface according to claim 1, wherein Further included are: Obtaining a user action sensing signal and an environmental device sensing signal; Analyzing and processing the user action sensing signal and the environmental device sensing signal to identify the device selected by the user action as the device to be controlled; Triggering the display of the control stimulation interface of the device to be controlled.

14. An XR control device based on a brain-computer interface, characterized in that, Included are: A display module for displaying the control stimulation interface of the device to be controlled; An electroencephalogram acquisition module for acquiring the electroencephalogram signals of the user when watching the control stimulation interface; A processing module communicatively connected to the display module and the electroencephalogram acquisition module, encoding the electroencephalogram signals within a preset time length to generate an encoding probability vector of the electroencephalogram signals; Decoding the encoding probability vector to generate a steady-state instruction response intensity vector; where each element of the steady-state instruction response intensity vector represents the response intensity of a preset instruction; the steady-state instruction response intensity vector is a set of the response intensities of the electroencephalogram signals at a certain refresh point corresponding to each preset instruction; Performing hierarchical threshold processing on the steady-state instruction response intensity vector to obtain an eigenstate instruction response intensity vector; Filtering the eigenstate instruction response intensity vector or the steady-state instruction response intensity vector to obtain an instruction output probability vector, and determining the preset instructions and the number of instructions to be output based on the instruction output probability vector; A communication module communicatively connected to the processing module, transmitting the output preset instructions to the device to be controlled, so that the device to be controlled executes the preset instructions.

15. An electronic device, characterized in that, Included are: One or more processors; And One or more memories, wherein computer-readable code is stored in the memories, and when the computer-readable code is run by the one or more processors, the XR control method based on a brain-computer interface according to any one of claims 1 to 13 is implemented.

16. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the XR control method based on a brain-computer interface according to any one of claims 1 to 13 is implemented.

Images