Multi-interface fusion dynamic brain-computer interaction method and terminal based on frequency phase information multiplexing

Abstract

The invention relates to the technical field of brain-computer interfaces, and discloses a multi-interface fusion dynamic brain-computer interaction method based on frequency phase information multiplexing and a terminal. Based on an offline electroencephalogram experiment of a user, the most brain electrical stimulation classification number U of the user for achieving the optimal electroencephalogram stimulation classification effect, frequency-phase information of the brain electrical stimulation of the U classification and a first corresponding relation between the brain electrical stimulation of the U classification and electroencephalogram signal characteristics are obtained, and an interactive interface after multi-interface fusion is dynamically partitioned; the method comprises the following steps: acquiring Z sub-regions and brain control instructions included in each sub-region, distributing frequency-phase information of U-class brain electrical stimulation to each brain control instruction, acquiring brain electrical stimulation classification based on a first corresponding relation on the basis of brain electrical signal characteristics induced by a flicker rule of brain electrical stimulation corresponding to a certain brain control instruction when a user watches the brain control instruction, and determining a sub-region based on the user sight drop point coordinate, and outputting a brain control instruction in combination with the brain electrical stimulation classification and the determined sub-region. According to the invention, the brain-computer interaction work efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the field of brain-computer interface technology, and in particular to a multi-interface fusion dynamic brain-computer interaction method and terminal based on frequency and phase information multiplexing. Background Art

[0002] Brain-computer interface (BCI) technology is a cross-disciplinary technology involving neuroscience, signal processing, pattern recognition, etc. It can directly convert information sent by the brain into commands that can drive external devices, and replace human body or language to achieve communication between people and the outside world and control the external environment. Currently, BCI technology is widely used in the field of intelligent control and can realize the control of various devices such as robotic arms, wheelchairs, exoskeletons and drones.

[0003] Nowadays, with the expansion of the control dimension of the controlled object and the installation of combined equipment, the number of brain control command classifications in the interactive interface continues to increase, and the demand for the types of stimulation frequency-phase coding combinations has increased. However, the number of available stimulation coding combinations is still very limited, especially in the case of high-frequency stimulation with high user comfort but low classification accuracy. It is difficult to meet the user's needs for large instruction classification stimulation coding. The frequency phase information with small differences between stimulation blocks will reduce the accuracy of user classification. For example, in the case of online control of drones, users need to take into account the environmental information returned by the first-person video at the same time. The user's EEG load is large. If large instructions are always classified, the user's fatigue level will increase and the EEG load will be overloaded, resulting in low user control efficiency and poor control accuracy. At the same time, as the application of brain-computer interaction systems gradually moves from indoors to outdoors and from single laboratory scenes to complex outdoor scenes, the environmental information that users need to pay attention to increases, and the control process becomes more complicated. How to reduce the user's EEG load, limit user misoperation, and improve the efficiency of brain-computer control during the control process has become an urgent problem to be solved.

[0004] In complex time-varying scenarios, as the scene information is continuously updated, the user only covers the control needs of specific dimensional actions at certain moments, and only involves the classification of a small number of brain control instructions. The user interface only needs to present a specific number of brain control stimuli. However, today, traditional static interactive interfaces cannot present different stimuli in different time periods. All stimuli are presented at the beginning of the task, and the number, position, and frequency information of the stimuli must remain unchanged until the end of the task. This method will lead to a high redundancy of brain control instructions, a large EEG load of the user, and increased fatigue of the user in complex time-varying scenarios. In addition, redundant and irrelevant instructions will increase the user's misoperation, which poses a greater risk to both the user and the controlled object. In addition, the existing brain-computer interaction interface is no longer satisfied with a single display device. It often requires multiple display devices to present the multi-channel return video information of the controlled object and the combined carrier in different regions, and display the brain control stimuli corresponding to specific operations in specific regions. At the same time, with the update of scene information during the interaction process, parameter information such as the position, size, and mapping instruction name of stimuli in different regions also needs to be updated in a time-varying manner. In addition, during the process of inducing EEG stimulation, the synchronization of EEG tags sent by the interface must be maintained at all times, and during the process of online EEG decoding, the timestamps of all areas of all devices must be kept consistent. Therefore, how to synchronize the clocks of multiple distributed devices has also become an important issue for the brain-computer interaction interface to present devices from a single device to an expanded device.

[0005] In view of this, there is an urgent need to develop a dynamic brain-computer interaction interface that can solve the problems of high command redundancy, poor timing flexibility, great coding difficulty, small amount of environmental information and single device display in existing brain-computer interaction interfaces. Summary of the invention

[0006] The purpose of the present invention is to provide a multi-interface fusion dynamic brain-computer interaction method and terminal based on frequency and phase information multiplexing, which can filter out irrelevant control instructions according to actual conditions, merge multiple interfaces into one interactive interface and dynamically partition them, and multiplex and allocate frequency and phase information of brain control instructions in sub-areas. It can greatly improve the interface interaction speed, reduce the user's task load, limit user misoperation, shorten the output command time, and improve the efficiency of brain-computer interaction operations.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] In the first aspect, the present invention provides a multi-interface fusion dynamic brain-computer interaction method based on frequency-phase information multiplexing. First, a user offline EEG experiment is configured to obtain the maximum number of EEG stimulation classifications U for the user to achieve the optimal EEG stimulation classification effect, the frequency-phase information of the U-classified EEG stimulation, and the first corresponding relationship between the U-classified EEG stimulation and the EEG signal feature; the dynamic brain-computer interaction method includes:

[0009] S10. Merge multiple interfaces into one interactive interface, align the clock frequencies of the multiple interfaces, and unify the EEG tag timestamps of the interactive interfaces;

[0010] S11. Merging the resolutions of multiple interfaces to obtain the resolution of the interactive interface;

[0011] S12. When the brain control instructions change over time, configure N brain control instructions that need to be presented at a certain time t, assign brain electrical stimulation to the N brain control instructions, and obtain a second corresponding relationship between the brain control instructions and the brain electrical stimulation;

[0012] S13. Dynamically partitioning the interactive interface based on the resolution of the interactive interface and the maximum number of brain electrical stimulation categories U to obtain Z sub-areas and brain control instructions included in each sub-area;

[0013] S14. Assign the frequency-phase information of the U-classified EEG stimulation to each brain control instruction in each sub-region. At this time, the EEG stimulation corresponding to each brain control instruction in the sub-region flashes according to the assigned frequency-phase information, and obtains the flashing pattern of the EEG stimulation;

[0014] At time S15.t, the user gazes at a certain brain control instruction, and the flickering pattern of the electroencephalogram stimulation corresponding to the brain control instruction induces an electroencephalogram signal feature. Based on the electroencephalogram signal feature and the first corresponding relationship, a corresponding electroencephalogram stimulation classification is obtained;

[0015] S16. Obtain the pixel coordinates of the point of sight on the interactive interface when the user is gazing at the brain control command at time t, and determine the sub-area where the point of sight is located;

[0016] S17. Based on the EEG stimulation classification obtained in S15 and the sub-areas determined in S16, the brain control instructions that the user needs to execute are determined and output to achieve dynamic brain-computer interaction.

[0017] As a possible implementation method, S14 is specifically as follows: according to the ascending order of the brain control command coordinates in each sub-region, the frequency-phase information of U-type electroencephalographic stimulation is allocated to the brain control commands in each sub-region in turn, and the frequency-phase information corresponding to the brain control commands in the same sub-region is not repeated.

[0018] As a possible implementation method, the following method is used to obtain the resolution of the interactive interface:

[0019] When multiple interfaces are arranged in a single column, the resolution of the interactive interface is [X, Y], where X = R i , Among them, R i is the minimum number of row pixels in the multi-interface, C k is the number of pixels in the column of the kth interface, M is the total number of interfaces, X is the total number of pixels in the row of the interactive interface, and Y is the total number of pixels in the column of the interactive interface;

[0020] When multiple interfaces are arranged in a single row, the resolution of the interactive interface is [X, Y]. Y=C i ; Among them, R k is the number of row pixels of the kth interface, C i is the minimum number of column pixels in multiple interfaces, M is the total number of interfaces, X is the total number of row pixels of the interactive interface, and Y is the total number of column pixels of the interactive interface.

[0021] As a possible implementation, S13 includes: dynamic partitioning of brain control command positions that are not fixed, specifically:

[0022] S130. dividing the number of brain control instructions N by the maximum number of brain electrical stimulation categories U, and rounding up the quotient to obtain Z sub-areas;

[0023] S131. Allocate N brain control instructions to Z sub-regions in sequence, the number of brain control instructions in the first Z-1 sub-regions is U, and the number of brain control instructions in the Zth sub-region is NU×(Z-1).

[0024] As a possible implementation manner, S131 includes the following sub-steps:

[0025] S1310. Based on the resolution of the interactive interface and the number of sub-regions Z, the number of pixels covered by each sub-region is obtained: Z i represents the number of pixels covered by the i-th sub-region, i = 1, 2, ..., Z;

[0026] S1311. If the interactive interface is partitioned by row pixels, the row pixel range of each sub-area is The column pixel range is (0, Y];

[0027] If the interactive interface is partitioned by column pixels, the row pixel range of each sub-area is (0, X], and the column pixel range is Wherein, X is the total number of row pixels of the interactive interface, Y is the total number of column pixels of the interactive interface, Z is the number of sub-regions, i represents the i-th sub-region, i=1, 2, ..., Z;

[0028] S1312. Determine the center position coordinates of each brain control command in each sub-area, specifically:

[0029] If the interactive interface is partitioned by rows of pixels, then the vth brain control command i in the i-th sub-area v The center coordinates are:

[0030]

[0031] If the interactive interface is partitioned by columns of pixels, then the vth brain control command i in the i-th sub-area v The center coordinates are:

[0032]

[0033] In the formula, i represents the i-th sub-region, i is less than or equal to Z, i v represents the vth brain control instruction in the ith sub-region, S i Represents the number of brain control instructions in the i-th sub-area. The brain control instructions are squares. l represents the side length of the brain control instructions, and d represents the spacing between the brain control instructions.

[0034] As a possible implementation, S13 includes: dynamic partitioning with fixed brain control command positions. At this time, the center position coordinates of each brain control command are known, specifically:

[0035] S133. Read the center position coordinates of each brain control instruction, and sort the N brain control instructions according to their center position coordinates;

[0036] S134. For the i-th sub-region, determine the size of the number of brain control instructions to be divided into different sub-regions and U; when the number of brain control instructions to be divided into different sub-regions is greater than U, execute S135; when the number of brain control instructions to be divided into different sub-regions is less than or equal to U, execute S136;

[0037] S135. Divide the first U brain control instructions in the brain control instructions to be divided into the same sub-region, and determine whether the pixel range of the sub-region overlaps with the pixel range of the U+1th brain control instruction in the brain control instructions to be divided. If not, the number of brain control instructions in the sub-region is U, and i=i+1 is set to continue to execute S134; otherwise, U=U-1 is set to re-execute S135;

[0038] S136. Divide the brain control instructions to be partitioned into the same sub-region, complete dynamic partitioning, and obtain Z sub-regions and the brain control instructions included in each sub-region.

[0039] As a possible implementation, the pixel range of the sub-region in S135 is obtained based on the center coordinates of the brain control instruction divided into the sub-region, specifically:

[0040] S1350. Obtain the minimum value x of the row pixel coordinates in the center position coordinates of the brain control command divided into the sub-area min and the maximum value x max , and the minimum value y of the column pixel coordinates min and the maximum value y max ;

[0041] S1351. The row pixel range of this sub-region is [x min -l,xmax +l), the column pixel range is [y min -l,y max +l); wherein the brain control command is a square, and l represents the side length of the brain control command.

[0042] As a possible implementation method, S133 is specifically: reading the center position coordinates of each brain control instruction, arranging the N brain control instructions in ascending order of the column pixel coordinates in their center position coordinates, and when the column pixel coordinates are the same, arranging them in ascending order of the row pixel coordinates.

[0043] In a second aspect, the present invention provides a terminal including a processor and a communication interface coupled to the processor, wherein the processor is used to run a computer program or instruction to implement the multi-interface fusion dynamic brain-computer interaction method based on frequency and phase information multiplexing provided in the first aspect.

[0044] Compared with the prior art, the beneficial effects of the present invention are:

[0045] 1. The multi-interface fusion dynamic brain-computer interaction method based on frequency-phase information multiplexing provided by the present invention solves the problems of limited field of view and limited presentation of environmental information in existing brain-computer interaction interfaces through independent decoupling and transparent penetration mode settings. At the same time, this method aligns the EEG label timestamps of multiple interfaces, solves the problem of inconsistent labels in cross-interface EEG stimulation paradigms, and drives the graphics card performance to the maximum extent, reduces device display delay, improves the refresh rate of EEG stimulation flashing frames, and achieves good induction of EEG signals under the cross-interface stimulation paradigm.

[0046] 2. The multi-interface fusion dynamic brain-computer interaction method based on frequency-phase information reuse provided by the present invention is based on the optimal classification results obtained by the user's offline experiment, which ensures the individual differences of the subjects. Regardless of whether the subjects' EEG classification ability is superior, it can optimize user behavior to the greatest extent and reduce user misoperation. At the same time, this dynamic partitioning method can adapt to various complex scenarios, broaden the application scenarios of brain-computer interaction interfaces, promote brain control systems from indoors to outdoors, and improve the stability and robustness of brain-computer interaction systems.

[0047] 3. The multi-interface fusion dynamic brain-computer interaction method based on frequency-phase information multiplexing provided by the present invention adopts frequency-phase information multiplexing of visual evoked stimulation, which can be applied to a variety of visual stimulation paradigms and is applicable to both low-frequency and high-frequency conditions. It solves the problem of limited number of stimulation coding combinations in the classification of large EEG instructions, especially in high-frequency stimulation with high comfort. When the regional information can be accurately divided, this method can greatly reduce the number of EEG stimulation classifications and ensure the optimal EEG stimulation classification results. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0049] Figure 1 It is a traditional brain-controlled drone interface in an embodiment of the present invention;

[0050] Figure 2 This is a flow chart of a multi-interface fusion dynamic brain-computer interaction method based on frequency and phase information multiplexing in an embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram of the partitioning of the interactive interface by rows of pixels in the case of dynamic partitioning where the position of the brain control command is not fixed in an embodiment of the present invention;

[0052] Figure 4 This is a schematic diagram of pixel partitioning by column in an interactive interface in a case of dynamic partitioning where the position of a brain control command is not fixed in an embodiment of the present invention;

[0053] Figure 5 This is a schematic diagram of a dynamic partition with a fixed position of a brain control command in an embodiment of the present invention;

[0054] Figure 6 A schematic diagram of allocating frequency-phase information for brain control commands in each sub-area when the interactive interface is partitioned by columns of pixels in the case of dynamic partitioning where the position of the brain control command is not fixed in an embodiment of the present invention;

[0055] Figure 7 A schematic diagram of allocating frequency-phase information for brain control commands in each sub-area in the case of dynamic partitioning with a fixed position of brain control commands in an embodiment of the present invention;

[0056] Figure 8 It is a schematic diagram that the positions of the sub-areas in the embodiment of the present invention need to be fixed according to a preset setting;

[0057] Fig. 9 This is a schematic diagram of merging two interfaces into an interactive interface in an embodiment of the present invention;

[0058] Fig.10 The dynamic partitioning and frequency-phase information allocation results of the brain control command with non-fixed position in the embodiment of the present invention;

[0059] Fig.11 This is the dynamic partitioning and frequency-phase information allocation result of the brain control command with fixed position in the embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to clearly describe the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, words such as "first" and "second" are used to distinguish the same items or similar items with basically the same functions and effects. For example, the first threshold and the second threshold are only used to distinguish different thresholds, and their order is not limited. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0061] It should be noted that, in the present invention, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0062] In the present invention, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The following at least one item (items) or similar expressions thereof refer to any combination of these items, including any combination of single items (items) or plural items (items). For example, at least one item (items) of a, b or c can mean: a, b, c, the combination of a and b, the combination of a and c, the combination of b and c, or the combination of a, b and c, where a, b, c can be single or multiple.

[0063] Nowadays, with the continuous expansion of BCI system application scenarios and the continuous increase in control dimensions, traditional brain-computer interaction interfaces have gradually failed to meet the application requirements of current BCI systems. Taking brain-controlled drones as an example, Figure 1As shown in the traditional brain-controlled drone interface, in the traditional static brain-computer interaction interface, the center of the interface keeps presenting the drone's first-person perspective video stream, surrounded by steady-state visual evoked potential (SSVEP) stimulation flashing blocks, 12 command classification EEG stimulation mapping corresponds to the drone's 12 flight actions, and the stimulation blocks correspond to 12 different frequencies and specific phases according to joint frequency-phase modulation (JFPM). During the drone's task operation, the stimulation blocks must be presented from beginning to end, and the user has been performing the EEG 12 command classification task during the experiment. The traditional static brain-computer interaction interface will lead to a high degree of redundancy of brain control commands, a large EEG load on the user, and increased user fatigue in complex time-varying scenarios, and redundant and irrelevant commands will increase the user's misoperation, which is a great risk for both the user and the controlled object.

[0064] The present invention proposes a multi-interface fusion dynamic brain-computer interaction method and terminal based on frequency and phase information multiplexing, which can filter out irrelevant control instructions according to actual conditions, merge multiple interfaces into one interactive interface and dynamically partition them, and perform frequency and phase information multiplexing and allocation of brain control instructions in sub-areas. It can greatly improve the interface interaction speed, reduce the user's task load, limit user misoperation, shorten the instruction output time, and improve the efficiency of brain-computer interaction operations.

[0065] In a first aspect, the present invention provides a multi-interface fusion dynamic brain-computer interaction method based on frequency-phase information multiplexing, firstly configuring a user's offline EEG experiment, obtaining the maximum number of EEG stimulation classifications U for the user to achieve the optimal EEG stimulation classification effect, the frequency-phase information of the U-classified EEG stimulation, and the first corresponding relationship between the U-classified EEG stimulation and the EEG signal feature;

[0066] As an example, suppose the number of EEG stimulation categories is U s When and the number of EEG stimulation categories is U l When , the optimal EEG classification results obtained in the experiment are the same, but U s Less than U l , then the maximum number of brain electrical stimulation categories U is equal to U l , which is the maximum value between the two. The U-type brain electrical stimulation is recorded as: U1, U2, ..., U u The frequency-phase information of each type of EEG stimulation is recorded as: (Fre u , Pha u ), u=1,2,…,U, and record the first corresponding relationship as shown in Table 1:

[0067] Table 1 The first correspondence between U-classified EEG stimulation and EEG signal features

[0068] Classification of EEG EEG signal characteristics <![CDATA[U1]]> Feature 1 <![CDATA[U2]]> Feature 2 <![CDATA[U3]]> Feature 3 <![CDATA[U4]]> Feature 4 …… ……

[0069] The present invention is based on the optimal classification results obtained by offline user experiments, ensuring the individual differences of the subjects. Regardless of whether the subjects have superior EEG classification capabilities, the user behavior can be optimized to the greatest extent and user misoperation can be reduced. At the same time, the dynamic partitioning method can adapt to various complex scenarios, broaden the application scenarios of brain-computer interaction interfaces, promote brain control systems from indoors to outdoors, and improve the stability and robustness of brain-computer interaction systems.

[0070] See also Figure 2 , dynamic brain-computer interaction methods include:

[0071] S10. Merge multiple interfaces into one interactive interface, align the clock frequencies of the multiple interfaces, and unify the EEG tag timestamps of the interactive interfaces;

[0072] As an example, assuming that the total number of interface display devices is M, the interface canvas is merged across multiple display devices and set to transparent penetration mode so that multiple interfaces are integrated into one interactive interface, which can be independently superimposed on any static or dynamic background; M display devices are connected to the same graphics card, and the clock frequencies of multiple interfaces are aligned through the graphics card surround setting to keep the interfaces refreshed synchronously and unify the EEG tag timestamps of the interactive interfaces.

[0073] The present invention aligns the EEG label timestamps of multiple interfaces, solves the problem of inconsistent labels in cross-interface EEG stimulation paradigms, drives graphics card performance to the maximum extent, reduces device display delay, improves the refresh rate of EEG stimulation flashing frames, and achieves good induction of EEG signals under a cross-interface stimulation paradigm.

[0074] S11. Merging the resolutions of multiple interfaces to obtain the resolution of the interactive interface;

[0075] As a possible implementation method, the following method is used to obtain the resolution of the interactive interface:

[0076] When multiple interfaces are arranged in a single column, the resolution of the interactive interface is [X, Y], where X = R i , Among them, R i is the minimum number of row pixels in the multi-interface, C k is the number of pixels in the column of the kth interface, M is the total number of interfaces, X is the total number of pixels in the row of the interactive interface, and Y is the total number of pixels in the column of the interactive interface;

[0077] When multiple interfaces are arranged in a single row, the resolution of the interactive interface is [X, Y]. Y=C i; Among them, R k is the number of row pixels of the kth interface, C i is the minimum number of column pixels in multiple interfaces, M is the total number of interfaces, X is the total number of row pixels of the interactive interface, and Y is the total number of column pixels of the interactive interface.

[0078] S12. When the brain control instructions change over time, configure N brain control instructions that need to be presented at a certain time t, assign brain electrical stimulation to the N brain control instructions, and obtain a second corresponding relationship between the brain control instructions and the brain electrical stimulation;

[0079] As an example, assuming that the number of brain control instructions to be presented at time t is 10, these 10 brain control instructions are named A to J, and brain electrical stimulation is assigned to the 10 brain control instructions. The second corresponding relationship between the brain control instructions and the brain electrical stimulation is shown in Table 2:

[0080] Table 2 The second correspondence between brain control instructions and brain electrical stimulation

[0081] Brain control commands Electrical brain stimulation A Stimulus 1 B Stimulus 2 C Stimulus 3 D Stimulus 4 …… …… J Stimulus 10

[0082] S13. Dynamically partitioning the interactive interface based on the resolution of the interactive interface and the maximum number of brain electrical stimulation categories U to obtain Z sub-areas and brain control instructions included in each sub-area;

[0083] As a possible implementation method, dynamic partitioning includes: dynamic partitioning in which the position of the brain control command is not fixed. In this case, the display position of the brain control command on the interactive interface can be freely arranged. Specifically:

[0084] S130. The quotient obtained by dividing the number of brain control instructions N by the maximum number of brain electrical stimulation categories U is rounded up to obtain Z sub-areas; that is, Z=1,2,…

[0085] S131. Allocate N brain control instructions to Z sub-regions in sequence, the number of brain control instructions in the first Z-1 sub-regions is U, and the number of brain control instructions in the Zth sub-region is NU×(Z-1).

[0086] As a possible implementation manner, S131 includes the following sub-steps:

[0087] S1310. Based on the resolution of the interactive interface and the number of sub-regions Z, the number of pixels covered by each sub-region is obtained: Z i represents the number of pixels covered by the i-th sub-region, i=1, 2, ..., Z, X is the total number of row pixels of the interactive interface, and Y is the total number of column pixels of the interactive interface;

[0088] S1311. If the interactive interface is partitioned by row pixels, the row pixel range of each sub-area is The column pixel range is (0, Y]; from S1310, it can be seen that the number of pixels covered by each sub-region is equal. Therefore, when partitioning by row pixels, the column pixel range of each sub-region is (0, Y], while the row pixel range is different.

[0089] As an example of pixel partitioning by row, see Figure 3 , assuming that the interactive interface is divided into Z sub-areas from left to right, the row pixel range of the first sub-area is The row pixel range of the second sub-area is By analogy, the row pixel range of the i-th sub-region is

[0090] If the interactive interface is partitioned by column pixels, the row pixel range of each sub-area is (0, X], and the column pixel range is Wherein, X is the row pixel of the interactive interface, Y is the column pixel of the interactive interface, Z is the number of sub-regions, i represents the i-th sub-region, i=1, 2, ..., Z;

[0091] S1312. Determine the center position coordinates of each brain control command in each sub-area, specifically:

[0092] If the interactive interface is partitioned by rows of pixels, then the vth brain control command i in the i-th sub-area v The center coordinates are:

[0093]

[0094] If the interactive interface is partitioned by columns of pixels, then the vth brain control command i in the i-th sub-area v The center coordinates are:

[0095]

[0096] In the formula, i represents the i-th sub-region, i is less than or equal to Z, i v represents the vth brain control instruction in the ith sub-region, S i Represents the number of brain control instructions in the i-th sub-area. The brain control instructions are squares. l represents the side length of the brain control instructions, and d represents the spacing between the brain control instructions.

[0097] As an example of pixel partitioning by column, see Figure 4 , assuming U is 4, assign 10 brain control commands A to J to In the sub-areas, the number of brain control commands in the first two sub-areas is 4, and the number of brain control commands in the third sub-area is 2. In general, in order to maintain a good classification result, the size and spacing of brain control commands are kept consistent, and most brain control commands are square, with l representing the side length of the brain control command and d representing the spacing of the brain control command. In order to maximize the user's classification accuracy, when the position of the brain control command is not fixed, the brain control command is placed in the central field of view of each sub-area as much as possible, and the number of column pixels in each sub-area is Therefore, the distance between the middle point of each sub-region column pixel and the upper and lower edges of the sub-region is the same, both Therefore, the pixel coordinates of the center point column of the vth brain control command in the ith sub-area can be obtained as:

[0098] The number of row pixels in each sub-area is X, and the midpoint of the row pixels is To ensure that the brain control command is placed in the central field of view of each sub-area as much as possible, the midpoint of the pixel in the area row To start stimulating from the center line of the regional row pixels to both ends. Figure 4 Taking area 1 as an example, the pixel coordinates of the B instruction line closest to the midline are The pixel coordinates of the C instruction line are Secondly, the pixel coordinates of the A instruction row are The pixel coordinates of the D instruction line are Therefore, the row pixel coordinates of the center point of the vth brain control command in the ith sub-area can be obtained as:

[0099] See also Figure 4 , the solution is that the center position coordinates of the third brain control command in the second sub-area are Since the brain control instructions are distributed in sequence, the third brain control instruction, that is, the seventh brain control instruction, which should be placed in the second sub-area is G. Therefore, the brain control instruction G can be placed at its center position coordinates until the dynamic partitioning of all instructions is completed.

[0100] As a possible implementation method, dynamic partitioning includes: dynamic partitioning with fixed brain control command positions, that is, brain control commands need to be fixed at a certain position on the interface according to actual needs. At this time, the center position coordinates of each brain control command are known, specifically:

[0101] S133. Read the center position coordinates of each brain control instruction, and sort the N brain control instructions according to their center position coordinates;

[0102] As a possible implementation method, N brain control instructions are sorted according to their center position coordinates, specifically: the center position coordinates of each brain control instruction are read, and the N brain control instructions are arranged in ascending order of the column pixel coordinates in their center position coordinates; when the column pixel coordinates are the same, they are arranged in ascending order of the row pixel coordinates.

[0103] As an example, see Figure 5 , the position coordinates of 10 brain control commands A~J are known, and they are arranged in ascending order of column coordinates in their center position coordinates as ABECFIGDJH. Among them, the column coordinates in the position coordinates of G and D are the same, so the two are arranged in ascending order of row pixel coordinates, that is, G is in front and D is in the back.

[0104] S134. For the i-th sub-region, determine the size of the number of brain control instructions to be divided into different sub-regions and U; when the number of brain control instructions to be divided into different sub-regions is greater than U, execute S135; when the number of brain control instructions to be divided into different sub-regions is less than or equal to U, execute S136;

[0105] S135. Divide the first U brain control instructions in the brain control instructions to be divided into the same sub-region, and determine whether the pixel range of the sub-region overlaps with the pixel range of the U+1th brain control instruction in the brain control instructions to be divided. If not, the number of brain control instructions in the sub-region is U, and i=i+1 is set to continue to execute S134; otherwise, U=U-1 is set to re-execute S135;

[0106] As a possible implementation, the pixel range of the sub-region is obtained based on the center coordinates of the brain control command divided into the sub-region, specifically:

[0107] S1350. Obtain the minimum value x of the row pixel coordinates in the center position coordinates of the brain control command divided into the sub-area min and the maximum value x max , and the minimum value y of the column pixel coordinates min and the maximum value y max ;

[0108] S1351. The row pixel range of this sub-region is [x min -l,x max +l), the column pixel range is [y min -l,y max +l); wherein the brain control command is a square, and l represents the side length of the brain control command.

[0109] S136. Divide the brain control instructions to be partitioned into the same sub-region, complete dynamic partitioning, and obtain Z sub-regions and the brain control instructions included in each sub-region.

[0110] As an example of dynamic partitioning with a fixed position of brain control instructions, assuming that the value of U is 4, when the first partition is performed, the brain control instructions to be partitioned are ABECFIGDJH, and the number is 10, which is greater than 4. Therefore, the first four brain control instructions ABEC are divided into the same sub-area, and it is determined whether the pixel range of the sub-area overlaps with the pixel range of the fifth brain control instruction, i.e., F, among the brain control instructions to be partitioned. Figure 5 It can be seen that among the center coordinates of the four ABEC brain control commands, the brain control command with the smallest row pixel coordinate value is A, and its row pixel coordinate value is The brain control command with the largest row pixel coordinate value is C, and its row pixel coordinate value is The brain control command with the smallest column pixel coordinate value is A, and its column pixel coordinate value is The brain control command with the largest column pixel coordinate value is C, and its column pixel coordinate value is Therefore, the row pixel range of this sub-region is The column pixel range is It should be explained that when using an eye tracker or computer vision technology to obtain pixel information of the gaze area, there is a certain degree of accuracy error, so it is necessary to keep the accuracy of the area when dividing the area. If the value after -l is a negative number, it starts from zero. The center position coordinates of the brain control command F are Therefore, the row pixel range is The column pixel range is Determine the row pixel range of the sub-area Column pixel range The row pixel range of the brain control command F Column pixel range Is there any overlap? If there is no overlap, the number of brain control instructions in this sub-area is 4, that is: ABEC, and the number of brain control instructions to be partitioned is updated to 6: FIGDJH. The first 4 of these 6 brain control instructions are divided into the next sub-area, and the above operation is repeated. If there is overlap, the number of brain control instructions in this sub-area is updated to 3, that is: ABE. It is determined whether the pixel range of the sub-area where these three brain control instructions are located overlaps with the pixel range of the fourth brain control instruction, that is, C. If not, the number of brain control instructions in this sub-area is 3, that is: ABE, and the number of brain control instructions to be partitioned is updated to 7: CFIGDJH. If there is overlap, the number of brain control instructions in this sub-area is updated to 2, that is: AB, and continue to determine whether the pixel range of the sub-area where these two brain control instructions are located overlaps with the pixel range of the third brain control instruction, that is, E.

[0111] When the number of brain control instructions to be partitioned is less than 4, for example, the number of brain control instructions to be partitioned is 2, that is: JH, see Figure 5, then JH is divided into the same sub-region, and dynamic partitioning is completed. In this example, a total of 5 sub-regions are obtained. The number of brain control instructions in the first sub-region is 4: ABEC, the number of brain control instructions in the second sub-region is 1: F, the number of brain control instructions in the third sub-region is 1: I, the number of brain control instructions in the fourth sub-region is 2: GD, and the number of brain control instructions in the fifth sub-region is 2: JH.

[0112] S14. Assign the frequency-phase information of the U-classified EEG stimulation to each brain control instruction in each sub-region. At this time, the EEG stimulation corresponding to each brain control instruction in the sub-region flashes according to the assigned frequency-phase information, and obtains the flashing pattern of the EEG stimulation;

[0113] As a possible implementation method, the frequency-phase information of the U-type electroencephalographic stimulation is assigned to the brain control commands in each sub-region in ascending order of the brain control command coordinates in each sub-region, and the frequency-phase information corresponding to the brain control commands in the same sub-region is not repeated.

[0114] As an example, assuming that the number of U is 4, the frequency-phase information of the U-type EEG stimulation recorded during the offline EEG experiment is: U1: (Fre1, Pha1), U2: (Fre2, Pha2), U3: (Fre3, Pha3), U4: (Fre4, Pha4). Figure 4 What is shown is the result of column pixel partitioning in the case of dynamic partitioning when the position of brain control commands is not fixed. Therefore, this partitioning result is continued here to show how to assign the frequency-phase information of U-classified EEG stimulation to each brain control command in each sub-region in the case of dynamic partitioning when the position of brain control commands is not fixed. For each sub-region, the frequency-phase information of U-class EEG stimulation is assigned to the brain control commands in ascending order of the coordinates of the brain control commands. In the results of column pixel partitioning, the column pixels of the center position coordinates of the brain control commands in each sub-region are the same, so they are assigned in ascending order of the row pixels in the center position coordinates. For the assignment results, see Figure 6, the frequency-phase information obtained by brain control instructions A to D in the first sub-region is: (Fre1, Pha1), (Fre2, Pha2), (Fre3, Pha3), (Fre4, Pha4), the frequency-phase information obtained by brain control instructions E to H in the second sub-region is: (Fre1, Pha1), (Fre2, Pha2), (Fre3, Pha3), (Fre4, Pha4), and the frequency-phase information obtained by brain control instructions I to J in the third sub-region is: (Fre1, Pha1), (Fre2, Pha2). Combined with Table 2, it can be seen that the brain electrical stimulation corresponding to each brain control instruction, for example, the brain electrical stimulation corresponding to brain control instructions A to D in the first sub-region is stimulation 1, stimulation 2, stimulation 3, and stimulation 4, respectively, and the brain electrical stimulation corresponding to brain control instructions E to H in the second sub-region is stimulation 5, stimulation 6, stimulation 7, and stimulation 8, respectively. Since the frequency-phase information assigned to brain control instructions A and E is the same, after the assignment is completed, the flickering rules of stimulation 1 and stimulation 5 are also the same.

[0115] As an example, since Figure 5 The partition results are shown in the case of dynamic partitioning with fixed brain control command positions. Therefore, this partition result is used to show how to allocate the frequency-phase information of U-classified electroencephalographic stimulation to each brain control command in each sub-area. Since the dynamic partitioning with fixed brain control command positions has been arranged in ascending order of the coordinates of the center position of the brain control command before partitioning, the frequency-phase information of the brain control commands in each sub-area can be allocated according to the order of their arrangement positions. The allocation results can be seen in Figure 7 In the first sub-region, the frequency-phase information obtained by the brain control instructions A, B, E, and C are: (Fre1, Pha1), (Fre2, Pha2), (Fre3, Pha3), and (Fre4, Pha4), respectively; the frequency-phase information obtained by the brain control instruction F in the second sub-region is: (Fre1, Pha1), the frequency-phase information obtained by the brain control instruction I in the third sub-region is: (Fre1, Pha1), the frequency-phase information obtained by the brain control instructions G and D in the fourth sub-region is: (Fre1, Pha1), (Fre2, Pha2), and the frequency-phase information obtained by the brain control instructions J and H in the fifth sub-region is: (Fre1, Pha1), (Fre2, Pha2), respectively. The brain control instructions that obtain the same frequency-phase information have the same flickering law of the corresponding electroencephalographic stimulation.

[0116] The present invention reuses frequency and phase information of visual stimulation and can be applied to a variety of visual stimulation paradigms, both in low-frequency and high-frequency situations. It solves the problem of limited number of stimulation coding combinations in the classification of large EEG instructions, especially in high-frequency stimulation with high comfort. When regional information can be accurately divided, the method can greatly reduce the number of EEG stimulation classifications and ensure the optimal EEG stimulation classification results.

[0117] At time S15.t, the user gazes at a certain brain control instruction, and the flickering pattern of the electroencephalogram stimulation corresponding to the brain control instruction induces an electroencephalogram signal feature. Based on the electroencephalogram signal feature and the first corresponding relationship, a corresponding electroencephalogram stimulation classification is obtained;

[0118] As an example, in the case of dynamic partitioning where the position of the brain control command is not fixed, assuming that the EEG signal feature induced by the flickering law of the EEG stimulation corresponding to the brain control command that the user is watching is feature 1, according to Table 1, the EEG stimulation classification corresponding to the EEG signal feature is U1, because the frequency-phase information of the U1 type EEG stimulation is (Fre1, Pha1). Figure 6 The brain control instructions with frequency-phase information (Fre1, Pha1) are: A, E, I.

[0119] S16. Obtaining the pixel coordinates of the point of sight on the interactive interface when the user is looking at a certain brain control command at time t, and determining the sub-area where the point of sight is located;

[0120] S17. Based on the EEG stimulation classification obtained in S15 and the sub-areas determined in S16, the brain control instructions that the user needs to execute are determined and output to achieve dynamic brain-computer interaction.

[0121] As an example, assuming that when the user looks at the brain control command, the pixel coordinates of the line of sight on the interactive interface are in the second sub-area. Since among the brain control commands A, E and I, E is in the second sub-area, it can be determined that the brain control command the user wants to output is E.

[0122] The same method can also be used in the case of dynamic partitioning where the position of the brain control command is fixed to determine the brain control command that the user wants to output, which will not be elaborated here.

[0123] When the sub-area position needs to be fixed as preset, see Figure 8, the interactive interface presets 5 fixed sub-areas (the 6 brain control instructions in the solid line frame of sub-area 4 and the 4 brain control instructions in the dotted line frame appear alternately according to different timings). Assuming that in this case, according to the user's offline EEG experiment, the maximum number of EEG stimulation classifications for the user to achieve the optimal EEG stimulation classification effect is 6, and the number of brain control instructions in each sub-area does not exceed 6, there is no need to perform partitioning operations, and the frequency-phase information is assigned to each brain control instruction to continue to execute subsequent steps. If the number of brain control instructions in a sub-area exceeds 6, choose to partition according to the fixed or non-fixed position of the brain control instructions based on actual needs, and execute the corresponding dynamic partitioning steps.

[0124] In a second aspect, the present invention provides a terminal including a processor and a communication interface coupled to the processor, wherein the processor is used to run a computer program or instruction to implement the multi-interface fusion dynamic brain-computer interaction method based on frequency and phase information multiplexing provided in the first aspect.

[0125] In order to more clearly describe the technical solution of the present invention, it is further elaborated with specific examples below.

[0126] Assume that there are two interfaces, and the display devices that present the two interfaces are of the same model, both Dell Alienware 27-inch monitors (resolution 2560*1440 pixels, refresh rate 240Hz). Connect the two display devices to the same graphics card of the workstation host. The workstation host model is Intel Xeon 6226R Golden CPU@3.9GHz, and the graphics card model is NVIDA3090. Set the unified device clock frequency through the graphics card surround to keep the interface refresh synchronously and align the device timestamps. Merge the interactive interface canvas across multiple monitors and set it to transparent penetration mode. According to the single-column arrangement of the display device, the resolution of the interactive interface is calculated to be 2560*2880, such as Fig. 9 shown.

[0127] Based on the user's offline EEG experiment, the maximum number of EEG stimulation categories that achieve the optimal EEG stimulation classification effect for the user is 4, and the frequency-phase information of the 4-category EEG stimulation is recorded as follows: first category: (31HZ, 0), second category: (32HZ, 0.35π), third category: (33HZ, 0.7π), fourth category: (34HZ, 1.05π). The first correspondence between the 4-category EEG stimulation and the EEG signal features is recorded as follows: first category → feature 1, second category → feature 2, third category → feature 3, fourth category → feature 4.

[0128] Assume that 19 brain control instructions A to S need to be presented at time t (in order to verify the method proposed in the present invention to the greatest extent, a large number of brain control instructions need to be presented in a simulated complex scenario) .

[0129] In the case of dynamic partitioning where the position of brain control commands is not fixed, the number of partitions is obtained The number of pixels covered by each sub-region is calculated as: Pixel points, assuming that the sub-area is partitioned by column pixels, the row pixel range of each sub-area is (0,2560], the column pixel range of the first sub-area is (0,576], the column pixel range of the second sub-area is (1×576,2×576], the column pixel range of the third sub-area is (2×576,3×576], and so on.

[0130] The 19 brain control instructions are sequentially distributed to 5 sub-areas. The number of brain control instructions in the first 4 sub-areas is 4, and the number of brain control instructions in the 5th sub-area is 3. Assuming that the brain control instruction is a square, the side length l and the spacing d of the brain control instruction are both 200 pixels. The center position coordinates of each brain control instruction in each sub-area can be calculated, and the brain control instructions are divided into the corresponding coordinate positions of the sub-areas. Frequency-phase information is distributed to each brain control instruction in each sub-area. The partitioning and distribution results are shown in Fig.10 .

[0131] In the case of dynamic partitioning with fixed brain control command positions, read the center position coordinates of the 19 brain control commands in, The 19 brain control instructions are arranged in columns of pixels. Arrange in ascending order. If the column pixels are the same, sort them by row pixels. Arranged in ascending order, the order of brain control instructions obtained is: ADLKENRSHJIBCFGHMOP. The first four brain control instructions ADLK are selected as the first sub-region. Assuming that the minimum value of the row pixel coordinates in the center position coordinates of these four brain control instructions is 450 and the maximum value is 2000, the minimum value of the column pixel coordinates is 280 and the maximum value is 480, and the side length l of the brain control instruction is 200, then the row pixel range of this sub-region is [250, 2200) and the column pixel range is [80, 680). In order to avoid overlapping of sub-regions, it is necessary to determine whether the pixel range of this sub-region overlaps with the pixel range of the fifth brain control instruction E during the partitioning process. If there is no overlap, the brain control instruction in this sub-region is ADLK; if there is an overlap, ADL is used as the first sub-region and the above partitioning process is repeated. This iterative partitioning strategy can ensure good classification results. Frequency-phase information is assigned to each brain control instruction in each sub-region. See the partitioning and allocation results for details. Fig.11 .

[0132] For the interactive interface that completes dynamic partitioning, the frequency-phase information obtained by brain control commands with the same order in each sub-area is the same, and the frequency-phase information obtained by each brain control command in a single sub-area is independent of each other. Fig.10 and Fig.11 In the figure, brain control instructions with the same frequency-phase information are marked with the same grayscale.

[0133] At time t, the user looks at a brain control instruction. The flickering pattern of the EEG stimulation corresponding to the brain control instruction induces an EEG signal feature. Assuming that the induced EEG signal feature is feature 2, it can be known that the EEG stimulation classification corresponding to the EEG signal feature is the second classification, Fig.11 Taking the partition results shown as an example, it can be seen that the brain control instructions corresponding to the electrical stimulation belonging to the second category are D, N, I, C, H, and O. Assuming that the pixel coordinates of the point of sight on the interactive interface when the user looks at the brain control instruction are in the fourth sub-area, it can be determined that the brain control instruction the user wants to execute is C; assuming that the pixel coordinates of the point of sight on the interactive interface when the user looks at the brain control instruction are in the second sub-area, it can be determined that the brain control instruction the user wants to execute is N.

[0134] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art may understand and implement other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the drawings, etc. In the specification, the word "comprising" does not exclude other components or steps, and "one" or "an" does not exclude multiple situations. A single processor or other unit can implement several functions listed in the specification. Certain measures are recorded in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0135] Although the present invention has been described in conjunction with specific features and embodiments thereof, it is apparent that various modifications and combinations may be made thereto without departing from the spirit and scope of the present invention. Accordingly, this specification and the accompanying drawings are merely exemplary illustrations of the present invention and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present invention. Obviously, those skilled in the art may make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, the present invention is intended to include such modifications and variations if they fall within the scope of the present invention and its equivalents.

Claims

1. A multi-interface fusion dynamic brain-computer interaction method based on frequency-phase information multiplexing, characterized in that: Configure an offline EEG experiment for the user to obtain the maximum number of EEG classifications U for achieving the optimal EEG classification effect for the user, the frequency-phase information of the U-classified EEG stimulations, and the first correspondence between the U-classified EEG stimulations and the EEG signal features; The dynamic brain-computer interaction method comprises: S10. Merging multiple interfaces into one interactive interface, aligning the clock frequencies of the multiple interfaces, and unifying the EEG tag timestamps of the interactive interfaces; S11. Merging the resolutions of the multiple interfaces to obtain the resolution of the interactive interface; S12. When the brain control instructions change over time, configure N brain control instructions that need to be presented at a certain time t, assign brain electrical stimulation to the N brain control instructions, and obtain a second correspondence between the brain control instructions and the brain electrical stimulation; S13. Dynamically partitioning the interactive interface based on the resolution of the interactive interface and the maximum number of brain electrical stimulation categories U to obtain Z sub-areas and brain control instructions included in each sub-area; S14. Assign the frequency-phase information of the U-classified electroencephalographic stimulation to each brain control instruction in each sub-region. At this time, the electroencephalographic stimulation corresponding to each brain control instruction in the sub-region flashes according to the assigned frequency-phase information, and obtain the flashing law of the electroencephalographic stimulation; At time S15.t, the user gazes at a certain brain control instruction, and the flickering pattern of the electroencephalogram stimulation corresponding to the brain control instruction induces an electroencephalogram signal feature, and based on the electroencephalogram signal feature and the first corresponding relationship, a corresponding electroencephalogram stimulation classification is obtained; S16. Obtaining the pixel coordinates of the point of sight on the interactive interface when the user is gazing at the brain control command at time t, and determining the sub-area where the point of sight is located; S17. Based on the EEG stimulation classification obtained in S15 and the sub-areas determined in S16, the brain control instructions that the user needs to execute are determined and output to achieve dynamic brain-computer interaction.

2. The multi-interface fusion dynamic brain-computer interaction method based on frequency-phase information multiplexing according to claim 1 is characterized in that: Specifically, S14 includes allocating the frequency-phase information of the U-type electroencephalographic stimulation to the brain control instructions in each sub-area in ascending order of the coordinates of the brain control instructions in each sub-area, and the frequency-phase information corresponding to the brain control instructions in the same sub-area is not repeated.

3. The multi-interface fusion dynamic brain-computer interaction method based on frequency-phase information multiplexing according to claim 1 is characterized in that: The resolution of the interactive interface is obtained by the following method: When multiple interfaces are arranged in a single column, the resolution of the interactive interface is [X, Y], where X = R i , Among them, R i is the minimum number of row pixels in the multi-interface, C k is the number of column pixels of the kth interface, M is the total number of interfaces, X is the total number of row pixels of the interactive interface, and Y is the total number of column pixels of the interactive interface; When multiple interfaces are arranged in a single row, the resolution of the interactive interface is [X, Y]. Y=C i ; Among them, R k is the number of row pixels of the kth interface, C i is the minimum number of column pixels in multiple interfaces, M is the total number of interfaces, X is the total number of row pixels of the interactive interface, and Y is the total number of column pixels of the interactive interface.

4. The multi-interface fusion dynamic brain-computer interaction method based on frequency-phase information multiplexing according to claim 1 is characterized in that: The S13 includes: dynamic partitioning of brain control command positions that are not fixed, specifically: S130. dividing the number of brain control instructions N by the maximum number of brain electrical stimulation categories U, and rounding up the quotient to obtain Z sub-areas; S131. Allocate the N brain control instructions to the Z sub-regions in sequence, the number of brain control instructions in the first Z-1 sub-regions is U, and the number of brain control instructions in the Zth sub-region is NU×(Z-1).

5. The multi-interface fusion dynamic brain-computer interaction method based on frequency-phase information multiplexing according to claim 4 is characterized in that: The S131 includes the following sub-steps: S1310. Based on the resolution of the interactive interface and the number of sub-regions Z, the number of pixels covered by each sub-region is obtained: Z i represents the number of pixels covered by the i-th sub-region, i = 1, 2, ..., Z; S1311. If the interactive interface is partitioned by row pixels, the row pixel range of each sub-area is The column pixel range is (0, Y]; If the interactive interface is partitioned by column pixels, the row pixel range of each sub-area is (0, X], and the column pixel range is Wherein, X is the total number of row pixels of the interactive interface, Y is the total number of column pixels of the interactive interface, Z is the number of sub-regions, i represents the i-th sub-region, i=1, 2, ..., Z; S1312. Determine the center position coordinates of each brain control command in each sub-area, specifically: If the interactive interface is partitioned by rows of pixels, then the vth brain control instruction i in the i-th sub-area v The center coordinates are: If the interactive interface is partitioned by columns of pixels, then the vth brain control instruction i in the i-th sub-area v The center coordinates are: In the formula, i represents the i-th sub-region, i is less than or equal to Z, i v represents the vth brain control instruction in the ith sub-region, S i Represents the number of brain control instructions in the i-th sub-area. The brain control instructions are squares. l represents the side length of the brain control instructions, and d represents the spacing between the brain control instructions.

6. The multi-interface fusion dynamic brain-computer interaction method based on frequency-phase information multiplexing according to claim 1 is characterized in that: S13 includes: dynamic partitioning with fixed brain control command positions. At this time, the center position coordinates of each brain control command are known, specifically: S133. Read the center position coordinates of each brain control instruction, and sort the N brain control instructions according to their center position coordinates; S134. For the i-th sub-region, determine the size of the number of brain control instructions to be divided into different sub-regions and U; when the number of brain control instructions to be divided into different sub-regions is greater than U, execute S135; when the number of brain control instructions to be divided into different sub-regions is less than or equal to U, execute S136; S135. Divide the first U brain control instructions in the brain control instructions to be divided into the same sub-region, and determine whether the pixel range of the sub-region overlaps with the pixel range of the U+1th brain control instruction in the brain control instructions to be divided. If not, the number of brain control instructions in the sub-region is U, and i=i+1 is set to continue to execute S134; otherwise, U=U-1 is set to re-execute S135; S136. Divide the brain control instructions to be partitioned into the same sub-region, complete dynamic partitioning, and obtain Z sub-regions and the brain control instructions included in each sub-region.

7. The multi-interface fusion dynamic brain-computer interaction method based on frequency-phase information multiplexing according to claim 6 is characterized in that: S135 The pixel range of the sub-region is obtained based on the central coordinates of the brain control instruction divided into the sub-region, specifically: S1350. Obtain the minimum value x of the row pixel coordinates in the center position coordinates of the brain control command divided into the sub-area min and the maximum value x max , and the minimum value y of the column pixel coordinates min and the maximum value y max ; S1351. The row pixel range of this sub-region is [x min -l,x max +l), the column pixel range is [y min -l,y max +l); wherein the brain control command is a square, and l represents the side length of the brain control command.

8. The multi-interface fusion dynamic brain-computer interaction method based on frequency-phase information multiplexing according to claim 6 is characterized in that: The S133 is specifically as follows: reading the center position coordinates of each brain control instruction, arranging the N brain control instructions in ascending order of the column pixel coordinates in the center position coordinates, and arranging them in ascending order of the row pixel coordinates when the column pixel coordinates are the same.

9. A terminal, characterized in that: It includes a processor and a communication interface coupled to the processor, and the processor is used to run a computer program or instruction to implement the multi-interface fusion dynamic brain-computer interaction method based on frequency and phase information multiplexing as described in any one of claims 1 to 8.

Images