A brain-computer interface device for fine-grained reading and interpretation of brain power

Abstract

The present invention provides a brain-computer interface device for fine-grained reading and interpretation of mental workload, which relates to the field of mental workload detection. The device includes: an EEG signal acquisition module, which collects EEG signals to form EEG signal samples when the user being tested performs a fast sequence visual presentation task with different fine-grained targets occupying the search space ratio; a time domain propagation module, which extracts the time domain propagation graph embedding features of the EEG signal samples; a frequency domain evolution module, which extracts the frequency domain evolution summary graph embedding features of the EEG signal samples; an information sharing lead dynamic selection module, which fuses the time domain propagation graph embedding features and the frequency domain evolution summary graph embedding features to obtain fused features; a mental workload prediction module, which calls a mental workload classifier to decode the fused features to obtain a mental workload evaluation result. Through this application, the problem of difficulty in predicting the fine gradient changes of mental workload during mental workload detection and low detection accuracy caused by ignoring the dynamic change factors of mental workload is solved.

Description

Technical Field

[0001] The present invention relates to the field of mental workload detection, and in particular to a brain-computer interface device for fine-grained reading and interpretation of mental workload. Background Art

[0002] The brain-computer interface (BCI) system collects and processes brain signals from operators during tasks, decodes the current brain state, and implements real-time assessment of mental workload. Real-time monitoring of the mental workload of operators is of great significance for timely identification of abnormal conditions to prevent human errors.

[0003] In the study of mental workload, due to the limitation of experimental induction conditions, the experimental paradigm designed in the existing technology is difficult to induce fine-grained mental workload. Although researchers divide mental workload into two categories (low load, high load) or three categories (low load, medium load, high load) for induction and identification, this method of classifying mental workload into two or three categories is difficult to capture more subtle gradient changes. The mental workload identification model in the existing technology usually assumes that the mental workload in the sample is steady-state, that is, a static model is used to evaluate the mental workload level, but in fact, the mental workload of a person changes dynamically, which further limits the accurate detection of mental workload. Summary of the invention

[0004] The present invention provides a brain-computer interface device for fine-grained reading and interpretation of mental workload, so as to solve the problem in the prior art that it is difficult to predict the fine gradient changes of mental workload during mental workload detection, and the dynamic changes of mental workload are ignored, resulting in low detection accuracy.

[0005] The present invention provides a brain-computer interface device for fine-grained reading and interpretation of mental power, the device comprising the following modules: an electroencephalogram signal acquisition module, a time domain propagation module, a frequency domain evolution module, an information sharing lead dynamic selection module, and a mental load prediction module;

[0006] The EEG signal acquisition module is used to set the fine-grained target-occupying search space ratio, and to acquire the EEG signal of the user under test when the user under test performs a fast sequence visual presentation task with the target occupying the search space ratio, so as to obtain an EEG signal sample;

[0007] The time domain propagation module is used to construct a time domain feature dynamic graph according to the EEG signal sample, and call a graph convolutional neural network to extract time domain features from the time domain feature dynamic graph to obtain embedded features of the time domain propagation graph of the EEG signal sample;

[0008] The frequency domain evolution module is used to construct a frequency domain feature dynamic graph according to the EEG signal samples, and call a graph convolutional neural network group to extract frequency domain features from the frequency domain feature dynamic graph to obtain embedded features of the frequency domain evolution summary graph of the EEG signal samples;

[0009] The information sharing lead dynamic selection module is used to determine the time domain weight and frequency domain weight of the EEG signal sample according to the self-attention mechanism, and perform feature fusion on the time domain propagation graph embedding feature and the frequency domain evolution summary graph embedding feature through the time domain weight and the frequency domain weight to obtain the fusion feature of the EEG signal sample;

[0010] The mental workload prediction module is used to call the mental workload classifier to decode the fusion feature to obtain the mental workload assessment result of the user being tested.

[0011] In some embodiments, the EEG signal acquisition module is further used to pre-process the acquired EEG signals to obtain EEG signal samples of the tested user. The EEG signal pre-processing process includes:

[0012] The collected EEG signals are sampled and processed based on the bilateral mastoid points, and the EEG signals are frequency band filtered by using a bandpass filter with a fixed frequency range after the signal re-reference processing.

[0013] The EEG signal after frequency band filtering is denoised based on independent component analysis, and the denoised EEG signal is downsampled to obtain the EEG signal sample of the tested user.

[0014] In some embodiments, in the time domain propagation module, constructing a time domain feature dynamic graph according to the EEG signal sample includes:

[0015] Dividing the EEG signal sample into 7 equally divided EEG signal segments according to the time step;

[0016] For each EEG signal, the EEG leads in the EEG signal are used as time domain nodes, and the correlation between the EEG leads is used as time domain edges, wherein each time domain node corresponds to a time domain node feature, and the time domain node features include activity, mobility, complexity, high-order zero crossing times, and fractal dimension;

[0017] Constructing a time domain feature graph of the electroencephalogram signal according to the time domain nodes and the time domain edges;

[0018] The time domain feature map sequence formed by the time domain feature maps corresponding to the 7 segments of EEG signals is used as the time domain feature dynamic map.

[0019] In some embodiments, in the time domain propagation module, the calling graph convolutional neural network performs time domain feature extraction on the time domain feature dynamic graph to obtain the time domain propagation graph embedding feature of the EEG signal sample, including:

[0020] In the time domain feature dynamic graph, a time domain difference feature graph between every two adjacent time domain feature graphs is determined as the time domain difference feature graph of each propagation time step;

[0021] The initial feature map at the head of the sequence in the time domain feature dynamic map is dynamically fused with the time domain differential feature map of each propagation time step in sequence, and finally the time domain propagation map embedding feature is obtained.

[0022] Among them, when dynamic fusion is performed at each propagation time step, the initial feature map is fused with the time domain difference feature map of the corresponding propagation time step to obtain a time domain propagation feature map, and a graph convolutional neural network is called to extract time domain features from the time domain propagation feature map to obtain an initial feature map for dynamic fusion at the next propagation time step.

[0023] In some embodiments, in the frequency domain evolution module, constructing a frequency domain feature dynamic graph according to the EEG signal sample includes:

[0024] Dividing the EEG signal sample into three equally divided EEG signal segments according to the time step;

[0025] Performing Fourier transform on each segment of the EEG signal to obtain five EEG frequency bands of the EEG signal;

[0026] In each EEG frequency band of the EEG signal, the EEG leads of the EEG signal are used as frequency domain nodes, and the intra-band correlation between the EEG leads is used as a frequency domain edge, wherein each of the frequency domain nodes corresponds to a frequency domain node feature, and the frequency domain node features include power spectral density, differential entropy, power, sample entropy, and Shannon information entropy;

[0027] For each EEG frequency band, a frequency domain feature map is constructed according to the frequency domain nodes and the frequency domain edges, and the frequency domain feature map combination of the five EEG frequency bands is used as the frequency domain feature map combination of the EEG signal;

[0028] The frequency domain feature map sequence formed by combining the frequency domain feature maps corresponding to the three segments of EEG signals is used as the frequency domain feature dynamic map.

[0029] In some embodiments, in the frequency domain evolution module, the calling graph convolutional neural network group performs frequency domain feature extraction on the frequency domain feature dynamic graph to obtain the frequency domain evolution summary graph embedding feature of the EEG signal sample, including:

[0030] In the frequency domain feature dynamic graph, determining an initial frequency domain feature graph combination at the head of the sequence and a frequency domain difference feature graph combination of a second frequency domain feature graph combination in the sequence;

[0031] The frequency domain differential feature map combination is merged with the initial frequency domain feature map combination to obtain a frequency domain evolution feature map combination;

[0032] Calling a graph convolutional neural network group to perform frequency domain feature extraction on the frequency domain evolution feature graph combination to obtain a frequency domain evolution graph embedding feature combination;

[0033] For each EEG frequency band's frequency domain evolution graph embedding feature in the frequency domain evolution graph embedding feature combination, respectively determine the corresponding frequency band attention weight;

[0034] The frequency domain evolution graph embedding feature of each EEG frequency band is weighted by the frequency band attention weight, and the weighted result is subjected to frequency band feature fusion to obtain the frequency domain evolution summary graph embedding feature of the EEG signal sample.

[0035] In some embodiments, in the information sharing lead dynamic selection module, the time domain weight and frequency domain weight of the EEG signal sample include: the time domain weight of each EEG lead in the EEG signal sample for the time domain propagation graph embedding feature and the frequency domain weight for the frequency domain evolution summary graph embedding feature, and the feature fusion of the time domain propagation graph embedding feature and the frequency domain evolution summary graph embedding feature is performed by the time domain weight and the frequency domain weight to obtain the fusion feature of the EEG signal sample, including:

[0036] For each EEG lead in the EEG signal sample, the time domain weight for the time domain propagation graph embedding feature and the frequency domain weight for the frequency domain evolution summary graph embedding feature are fused by dual-domain attention to obtain the dual-domain fusion attention weight;

[0037] Embed the time domain propagation graph corresponding to each EEG lead with features, and perform dual-domain feature splicing with the frequency domain evolution summary graph embedding features to obtain a dual-domain feature splicing value matrix;

[0038] For each EEG lead in the EEG signal sample, the dual-domain feature splicing value matrix is ​​weighted by the dual-domain fusion attention weight to obtain the fusion feature of the EEG signal sample.

[0039] The present invention also provides a method for fine-grained reading and interpretation of mental power, the method comprising the following steps:

[0040] Setting a fine-grained target-occupying ratio of the search space, and collecting an electroencephalogram (EEG) signal of the user under test when the user under test performs a rapid sequence visual presentation task with the target occupying the search space ratio, to obtain an EEG signal sample;

[0041] Constructing a time domain feature dynamic graph according to the EEG signal sample, and calling a graph convolutional neural network to extract time domain features from the time domain feature dynamic graph to obtain a time domain propagation graph embedding feature of the EEG signal sample;

[0042] Constructing a frequency domain feature dynamic graph according to the EEG signal samples, and calling a graph convolutional neural network group to extract frequency domain features from the frequency domain feature dynamic graph, so as to obtain embedding features of the frequency domain evolution summary graph of the EEG signal samples;

[0043] Determine the time domain weight and frequency domain weight of the EEG signal sample according to the self-attention mechanism, and perform feature fusion on the time domain propagation graph embedding feature and the frequency domain evolution summary graph embedding feature through the time domain weight and the frequency domain weight to obtain the fusion feature of the EEG signal sample;

[0044] The mental workload classifier is called to decode the fusion feature to obtain a mental workload assessment result of the user being tested.

[0045] In some embodiments, the mental workload assessment result is obtained by calling a mental workload assessment model to predict an EEG signal sample of the user being tested. The mental workload assessment model is trained by EEG signal samples, and the EEG signal samples carry a real mental workload label. The training process of the mental workload assessment model includes:

[0046] By forward propagating the EEG signal sample in the mental workload assessment model, a mental workload prediction label of the EEG signal sample is obtained;

[0047] Based on the real mental workload label and the predicted mental workload label, constructing a cross entropy loss value of the mental workload assessment model;

[0048] Determining a time domain mutual information loss value and a frequency domain mutual information loss value of the EEG signal sample during forward propagation;

[0049] Based on the cross entropy loss value, the time domain mutual information loss value and the frequency domain mutual information loss value, a final loss value of the mental workload assessment model is constructed, and back propagation is performed in the mental workload assessment model through the final loss value to update the parameters of the mental workload assessment model.

[0050] In some embodiments, determining the time domain mutual information loss value and the frequency domain mutual information loss value of the EEG signal sample during the forward propagation process includes:

[0051] In the forward propagation process, the predicted time domain propagation graph embedding features of the EEG signal sample are determined, and the time domain mutual information loss value is constructed based on the mutual information between the predicted time domain propagation graph embedding features and the real time domain target feature graph, wherein the real time domain target feature graph is the feature graph at the end of the sequence in the time domain feature dynamic graph constructed in the forward propagation process;

[0052] During the forward propagation process, the predicted frequency domain evolution map embedding feature combination of the EEG signal sample is determined, and based on the mutual information between the predicted frequency domain evolution map embedding feature combination and the real frequency domain target feature map combination, a frequency domain mutual information loss value is constructed, wherein the real frequency domain target feature map combination is the feature map combination at the end of the sequence in the frequency domain feature dynamic map constructed during the forward propagation process.

[0053] The brain-computer interface device for fine-grained reading and interpretation of mental workload provided by the present invention collects the EEG signals of the user under test to form EEG signal samples, extracts the time domain propagation characteristics and frequency domain evolution characteristics of the EEG signal samples, and performs fusion processing of the time domain propagation characteristics and the frequency domain evolution characteristics based on the attention mechanism in the process of prediction and classification to evaluate the mental workload of the user under test. In this way, the dynamic change characteristics of mental workload are characterized by the time domain propagation characteristics and the frequency domain evolution characteristics during the classification of mental workload, thereby realizing accurate detection of mental workload. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0055] Figure 1 This is a framework diagram of the brain-computer interface device for fine-grained reading and interpretation of mental power provided by the present invention.

[0056] Figure 2 It is a schematic diagram of the principle of the time domain propagation module provided by the present invention.

[0057] Figure 3 It is a schematic diagram of the principle of the frequency domain evolution module provided by the present invention.

[0058] Figure 4 It is a flowchart of the method for fine-grained mental reading and interpretation provided by the present invention.

[0059] Figure 5 It is a schematic diagram of the application of the brain-computer interface device for fine-grained reading and interpretation of mental power provided by the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] Combine the following Figure 1-Figure 4 The present invention describes a brain-computer interface device for fine-grained reading and interpretation of mental power. Figure 1 This is a schematic diagram of the brain-computer interface device for fine-grained reading and interpretation of brain power provided by the present invention, such as Figure 1 As shown, the brain-computer interface device for fine-grained reading and interpretation of mental workload includes: an EEG signal acquisition module, a time domain propagation module, a frequency domain evolution module, an information sharing lead dynamic selection module, and a mental workload prediction module, which are described in detail below.

[0062] The EEG signal acquisition module is used to set the fine-grained target proportion in the search space, and to collect the EEG signals of the user under test when the user under test performs a fast sequence visual presentation task with the target proportion in the search space, so as to form an EEG signal sample.

[0063] Here, first of all, it is necessary to design a rapid sequence visual presentation task that the tested user needs to perform to induce the mental workload of the tested user. In the design process, the proportion of the target of the visual presentation task to the search space is determined according to the fine-grained level. The specific fine-grained target proportion of the search space can be designed as: 1%, 3%, 5%, 7%, 9%, 11%, 13%, 15%, 17%, 19%, 21%, 23%, 25%, 27%, 29%, a total of 15 target proportions to the search space, to achieve the induction of fine-grained mental workload. At the same time, it is also necessary to customize all fine-grained RSVP target retrieval tasks that the tested user needs to complete. During the whole process, the tested user needs to concentrate and sit in front of the center of the experimental screen to avoid other body movements except task response. The experiment includes a total of 60 groups (blocks) of experiments, each of which lasts for 2 minutes. During the experiment, one of the rapid sequence visual presentation tasks with a target proportion of the search space is randomly performed. Each target percentage of the search space appears twice in the whole experiment, and the adjacent blocks are set as rest.

[0064] During the experiment, the EEG signal acquisition module collects the EEG signals of the tested user in real time in the RSVP target retrieval task of various targets occupying the search space percentage. However, the collected EEG signals cannot be used directly, so the EEG signal acquisition module is also used to pre-process the collected EEG signals to obtain EEG signal samples of the tested user.

[0065] Specifically, firstly, the collected EEG signal is sampled. Here, an 8-second time window can be set, and the sample is sampled by sliding the time window. The sample capture time is 8 seconds, so an 8-second EEG signal is obtained. Then, the intercepted EEG signal is subjected to signal re-reference processing based on the bilateral mastoid points. The function of re-reference here is to replace the reference lead and reduce the influence of the original reference data recording method on the result. Then, a bandpass filter with a fixed frequency range is used to perform frequency band filtering on the EEG signal after the signal re-reference processing. The purpose of frequency band filtering is to remove the low- and high-frequency components in the EEG signal and retain the EEG signal in a specific frequency range.

[0066] Next, the EEG signal that has been frequency-band filtered is denoised based on independent component analysis. The purpose of denoising is to remove the interference of blinking and eye movements. Specifically, independent component analysis is used to remove the interference components in the EEG signal caused by blinking and eye movements. Of course, in some special cases, it is also necessary to remove interference components such as head movement, sweating, electrode movement, electrocardiogram, and electromyography.

[0067] Finally, the denoised EEG signal is downsampled. Since the sampling frequency of the EEG signal is very high, generally 1000 Hz, in order to reduce the amount of calculation, the frequency of the EEG signal is reduced to 200 Hz by frequency downsampling, and the 8-second EEG signal of the user under test is obtained as the EEG signal sample of the user under test.

[0068] The next step is the process of decoding EEG signal samples in the brain-computer interface device for fine-grained reading and interpretation of mental workload. The decoding process is implemented through the time domain propagation module, the frequency domain evolution module, the information sharing lead dynamic selection module, and the mental workload prediction module. Figure 1 illustrate.

[0069] like Figure 1 As shown, the time domain propagation module is used to construct a time domain feature dynamic graph based on the EEG signal samples, and call the graph convolutional neural network to extract the time domain features of the time domain feature dynamic graph to obtain the embedded features of the time domain propagation graph of the EEG signal samples.

[0070] In the brain workload decoding based on EEG signals, mining the dynamic changes of EEG signal data in samples over time can capture subtle changes in brain workload, thereby achieving fine-grained decoding of brain workload. Therefore, a time domain propagation module is designed in the brain-computer interface device for fine-grained reading and interpretation of brain workload provided by the present invention.

[0071] See also Figure 2 , Figure 2 is a schematic diagram of the principle of the time domain propagation module provided by the present invention, such as Figure 2As shown in , firstly, a time domain feature dynamic graph is constructed based on the EEG signal samples. In order to extract the characteristics of the EEG signal samples propagating over time, the EEG signal samples (i.e. Figure 2 The 8-second EEG sample in the video is divided into 7 equally divided EEG signals according to the propagation time step. Then, for each EEG signal, the EEG leads in the EEG signal are used as time domain nodes, and the correlation between the EEG leads is used as time domain edges. The correlation as time domain edges refers to the degree of correlation between the EEG leads. Among them, each time domain node corresponds to a time domain node feature, and the time domain node features include activity, mobility, complexity, high-order zero crossing times, and fractal dimension. Finally, the time domain feature map of the EEG signal sample is constructed based on the time domain nodes and time domain edges.

[0072] Here, based on each EEG signal, an undirected graph is constructed. , where the node set is the set of EEG leads, t represents time, and n represents the number of nodes. Each node represents an EEG lead, and each node ( ) all correspond to time domain node features ( ), where d is the dimension of the time domain node features. The time domain node features include the Hiorth parameters (including activity, mobility, complexity), high-order zero crossing times, and fractal dimension of the EEG signal samples. Therefore, the dimension d is taken as 5. Represents the degree of correlation between two EEG leads, forming a set of time domain edges, which can be obtained through an adjacency matrix To express.

[0073] In this way, a time domain feature map can be constructed for each EEG signal, and finally the time domain feature map sequence formed by the time domain feature maps corresponding to the 7 EEG signals is Here, the time domain feature maps corresponding to the 7 EEG signals are constructed into a time domain feature map sequence with a length of 7 as the time domain feature dynamic map.

[0074] like Figure 1 As shown in the figure, in the time domain propagation module, after constructing the time domain feature dynamic graph, the process of extracting the time domain features is introduced next, that is, calling the graph convolutional neural network to extract the time domain features of the time domain feature dynamic graph, and obtaining the embedded features of the time domain propagation graph of the EEG signal sample.

[0075] like Figure 2 As shown, there are 7 time-domain feature dynamic graphs in total. Here, the process of time-domain dynamic change is divided into three stages. The starting stage is the head of the time-domain feature dynamic graph (that is, the time-domain feature graph sequence). The initial time domain feature map , the middle 5 time domain feature maps It is used in the propagation stage, which is divided into 5 propagation time steps to propagate in sequence. The target feature map G is Figure 2 The actual target map shown in FIG. 1 is used as the target feature map in the final stage.

[0076] In the propagation stage, in order to characterize the time domain dynamic changes of EEG signals with the propagation time step, in the time domain feature dynamic graph, every two adjacent time domain feature graphs (i.e. and ) , as the time domain differential feature map of each propagation time step. That is, in the 7 time domain feature maps, calculate every two adjacent time domain feature maps (i.e. and ) , which is used to quantify the change in time domain features between adjacent time windows. Thus, 6 time domain differential feature maps can be calculated. Since the last propagation time step is the end stage, only the first 5 time domain differential feature maps are used as the time domain differential feature maps corresponding to the 5 propagation time steps in the propagation stage. Figure 2 The 5 difference graphs shown in .

[0077] The calculation process of the time domain difference feature map is as follows: the two adjacent time domain feature maps (i.e. and ) According to the time domain node (i.e., EEG leads) are subtracted, that is, the node features between two adjacent time domain feature maps (i.e. and ) to subtract features and obtain differential node features , and update the time domain edge weight to the adjacency matrix of the current time domain feature graph , to reflect the changes of time domain characteristics with time series.

[0078] The graph convolutional neural network uses a network structure consisting of five graph convolution blocks (GCN blocks) connected in series. Figure 2 As shown in the figure, each graph convolution block consists of a normalization layer, a graph convolution layer, and a nonlinear layer with the Relu function as the activation function. The five graph convolution blocks are used to extract time domain features at five propagation time steps. Specifically, the input time domain propagation feature map is subjected to layer normalization, graph convolution, and nonlinear activation processing in sequence to extract the corresponding time domain propagation feature map embedding.

[0079] Next, the initial time domain feature map of the sequence head in the time domain feature dynamic map , and the time domain difference feature map of each propagation time step Dynamic fusion is performed step by step to finally obtain the embedding features of the time domain propagation graph Here, in each propagation time step of the propagation stage, the time domain difference feature map representing the time domain dynamic changes of the EEG signal with the propagation time step is Gradually integrated into the initial time domain features That is, the five time-domain difference feature maps are gradually integrated into the initial feature map middle.

[0080] Among them, when dynamic fusion is performed at each propagation time step, the initial feature map is fused with the time domain difference feature map of the corresponding propagation time step to obtain the time domain propagation feature map, and the graph convolutional neural network is called to extract the time domain features of the time domain propagation feature map to obtain the initial feature map for dynamic fusion at the next propagation time step. Therefore, in the propagation stage, after 5 propagation time steps in sequence, the embedded features of the time domain propagation map are finally obtained.

[0081] Specific as Figure 2 As shown in Figure 2, at the first propagation time step of the propagation phase, the initial time domain feature map is transformed into Time-domain difference feature map corresponding to the first propagation time step Fusion is performed to obtain the time domain propagation feature map , and then call the first graph convolutional block of the graph convolutional neural network to perform time domain propagation feature map Perform time domain feature extraction to obtain the initial time domain propagation graph embedding features , as the initial time domain feature map for dynamic fusion in the second propagation time step.

[0082] Here, the fused time domain propagation feature map satisfy .like Figure 2 As shown, in the dynamic fusion unit, the calculation result Y of the processing function DF (i.e. ) is expressed as , which is represented by the two features and Projected into a shared space for fusion, W is the projection matrix. Therefore, the projection result obtained by the dynamic fusion unit is expressed as The dynamic fusion unit and the graph convolution block jointly realize the dynamic fusion of temporal features at different time steps in the propagation stage, capturing the detailed information of the dynamic changes of features over time.

[0083] At the second propagation time step, the initial time domain feature map of the second propagation time step (that is, the initial time domain propagation map obtained at the first propagation time step) is embedded into the feature map through the dynamic fusion unit DF. ) Time domain difference feature map corresponding to the second propagation time step Fusion is performed to obtain the corresponding time domain propagation feature map , and then input it into the second graph convolution block of the graph convolutional neural network for time domain feature extraction to obtain the second time domain propagation graph embedding feature , as the initial time domain feature map for dynamic fusion in the third propagation time step.

[0084] And so on, until the fifth propagation time step, the initial time domain feature map of the fifth propagation time step (i.e., the fourth time domain propagation map) is embedded into the feature map through the dynamic fusion unit DF. ) Time domain difference feature map corresponding to the fifth propagation time step The fused features are then input into the fifth graph convolution block of the graph convolutional neural network to obtain the embedding features of the fifth time domain propagation graph. . This marks the end of the propagation phase.

[0085] After the propagation stage, the final time domain features That is, as the embedding feature of the time domain propagation graph, it can be recorded as , which includes the time-domain propagation node embedding of each time-domain node , represents the feature dimension, C is the number of time domain nodes (i.e., EEG leads), which is used to input into the information sharing lead dynamic selection module for feature fusion.

[0086] like Figure 1 As shown, the frequency domain evolution module is used to construct a frequency domain feature dynamic graph based on the EEG signal samples, and call the graph convolutional neural network group to extract the frequency domain features of the frequency domain feature dynamic graph to obtain the embedded features of the frequency domain evolution summary graph of the EEG signal samples.

[0087] Considering that frequency domain characteristics can reveal the periodic components and frequency distribution in EEG signals, and frequency domain evolution can reflect the dynamic changes of the periodic components of EEG signals, the temporal evolution of different rhythms can be explored through the division of frequency bands. Therefore, a frequency domain evolution module is also designed in the brain-computer interface device for fine-grained reading and interpretation of brain power provided by the present invention.

[0088] See also Figure 3 , Figure 3 is a schematic diagram of the principle of the frequency domain evolution module provided by the present invention, such as Figure 3 As shown in , firstly, a frequency domain feature dynamic graph is constructed based on the EEG signal samples. Considering that a wider time window can retain the subtle spectral features in each frequency band, it is helpful to improve the accuracy of frequency domain feature extraction. Here, the EEG signal samples (i.e. Figure 3 The 8-second EEG sample in the image is divided into 3 equal segments according to the time step, and then Fourier transform is performed on each segment to obtain 5 EEG frequency bands of the EEG signal. Here, each EEG signal is transformed by Fourier and divided into 5 different EEG frequency bands.

[0089] In each EEG frequency band of the EEG signal, the EEG leads of the EEG signal are taken as frequency domain nodes, and each frequency domain node corresponds to a frequency domain node feature, which includes power spectrum density, differential entropy, power, sample entropy, and Shannon information entropy. The correlation between EEG leads in the frequency band is taken as a frequency domain edge. Here, the correlation of the frequency domain edge refers to the functional connection relationship between different EEG leads, which is essentially a PLV matrix, which is used here as an adjacency matrix to represent the frequency domain edge.

[0090] For each EEG frequency band f, the frequency domain feature map of the EEG frequency band is constructed according to the frequency domain nodes and frequency domain edges, and the frequency domain feature map combination of the five EEG frequency bands is used as the frequency domain feature map combination of the EEG signal, which is recorded as Finally, the frequency domain feature map sequence formed by combining the frequency domain feature maps corresponding to the three EEG signals is used as the frequency domain feature dynamic map, which can be recorded as .

[0091] In the frequency domain evolution module, after constructing the frequency domain feature dynamic graph based on the EEG signal samples, the process of extracting frequency domain features is introduced next, that is, calling the graph convolutional neural network group to extract the frequency domain features of the frequency domain feature dynamic graph, and obtaining the embedded features of the frequency domain evolution summary graph of the EEG signal samples.

[0092] like Figure 3 As shown, there are 3 frequency domain feature map combinations in the frequency domain feature dynamic map. Here, the process of frequency domain dynamic change is also divided into three stages. The initial stage is the initial frequency domain feature map combination at the head of the sequence in the frequency domain feature dynamic map (i.e., the frequency domain feature map sequence). , the frequency domain feature map combination of the intermediate process For the propagation stage, the target feature map combination at the end of the sequence in the frequency domain feature dynamic map (i.e., frequency domain feature map sequence) ,Right now Figure 3 The real target map shown is used as the target feature map combination in the final stage.

[0093] When extracting frequency domain features, in the frequency domain feature dynamic graph In the example, the initial frequency domain feature map combination of the sequence head is determined , combined with the second frequency domain feature map in the sequence The frequency domain difference feature map combination between ,Right now Figure 3 Then in the dynamic fusion unit, the frequency domain differential feature map is combined Combined with the initial frequency domain features Fusion is performed to obtain a combination of frequency domain evolution feature maps , and then call the graph convolutional neural network group (including multiple graph convolution blocks) to combine the frequency domain evolution feature graph Perform frequency domain feature extraction to obtain the frequency domain evolution graph embedding feature combination .

[0094] Here, the frequency domain difference feature map combination ,satisfy , Represents the initial frequency domain feature map combination corresponding to the five EEG frequency bands f Combined with the intermediate frequency domain feature map , according to the time domain node The result of feature subtraction of the EEG leads is represents the updated frequency domain edge used to connect the nodes. The specific calculation method is similar to the calculation method of the time domain difference feature map mentioned above, and will not be repeated here. The frequency domain evolution feature map combination obtained by the dynamic fusion unit is satisfy The processing process of the dynamic fusion unit here is similar to that of the dynamic fusion unit in the time domain propagation module, and will not be repeated here.

[0095] The graph convolutional neural network group is a network structure composed of 5 graph convolution blocks in parallel. The frequency domain evolution feature map of each EEG frequency band in the frequency domain evolution feature map combination , each of which is extracted through a graph convolution block to obtain the frequency domain single band evolution graph embedding feature Then, the frequency domain single band evolution graph corresponding to the five EEG frequency bands is embedded into the feature Combine them to get the embedding features of all frequency band evolution graphs in the frequency domain, that is, the embedding feature combination of the frequency domain evolution graph The composition structure of the graph convolution block here is similar to the graph convolution block for time domain feature extraction mentioned above, and will not be repeated here.

[0096] like Figure 3 As shown, the frequency domain evolution graph embeds the feature combination (Right now Figure 3 The frequency domain evolution map of each EEG frequency band f in the frequency domain all frequency band features in , respectively determine the corresponding frequency band attention weights (Right now Figure 3 The frequency band corresponding weight in ( ), through the frequency band attention weight Embed features into the frequency domain evolution graph of each EEG frequency band The weighted features are then summarized. Here, the weighted results are fused with frequency band features. The summarized frequency domain features are the frequency domain evolution summary graph of the EEG signal samples. , expressed as the following formula (1):

[0097] (1)

[0098] In the above formula (1), represents the frequency band attention weight of each EEG frequency band, The frequency domain evolution map representing each EEG frequency band is embedded with features.

[0099] Embedding features in the frequency domain evolution summary graph of EEG signal samples In the example, each frequency domain node is embedded in the frequency band after time propagation. , represents the feature dimension, C is the frequency domain node (i.e., the number of EEG leads), which is used to input into the information sharing lead dynamic selection module for feature fusion.

[0100] like Figure 1 As shown, the information sharing lead dynamic selection module is used to determine the time domain weight and frequency domain weight of the EEG signal sample according to the self-attention mechanism, and to fuse the time domain propagation graph embedding features and the frequency domain evolution summary graph embedding features through the time domain weight and frequency domain weight to obtain the fused features of the EEG signal sample.

[0101] The information sharing lead dynamic selection module receives the time domain propagation graph embedding features extracted by the time domain propagation module , and the frequency domain evolution summary graph embedded features extracted by the frequency domain evolution module After that, the time domain weight and frequency domain weight of the EEG signal sample are determined according to the self-attention mechanism. Here, the time domain weight and frequency domain weight of the EEG signal sample include: each EEG lead in the EEG signal sample embeds features for the time domain propagation graph The time domain weight And embedding features for frequency domain evolution summary graph The frequency domain weight The following describes the process of calculating weights based on the self-attention mechanism.

[0102] First, embed the time domain propagation graph of each time domain node Perform time domain attention calculation to obtain time domain weight , expressed as the following formula (2) to formula (4):

[0103] (2)

[0104] (3)

[0105] (4)

[0106] In the above formulas (2) to (4), is the query vector calculated for temporal attention, represents the key vector of time-domain attention calculation, C represents the number of time-domain nodes (i.e., EEG leads), and F represents the feature dimension of the mapping space. represents the query weight matrix, represents the key weight matrix, is the characteristic dimension of the time domain, is the scaling factor, is the temporal attention weight.

[0107] Frequency domain evolution graph embedding for each frequency domain node Perform frequency domain attention calculation to obtain frequency domain weights , expressed as the following formulas (5) to (7):

[0108] (5)

[0109] (6)

[0110] (7)

[0111] In the above formulas (5) to (7), is the query vector for frequency domain attention calculation, represents the key vector of frequency domain attention calculation, C represents the number of frequency domain nodes (i.e., EEG leads), and F represents the feature dimension of the mapping space. represents the query weight matrix, represents the key weight matrix, is the characteristic dimension in the frequency domain, is the scaling factor, is the frequency domain attention weight.

[0112] Determine the temporal weight of each EEG lead for the temporal propagation map embedding feature And the frequency domain weights for embedding features of the frequency domain evolution summary graph Then, through the time domain weight and frequency domain weights Embedding features into time domain propagation graph And frequency domain evolution summary graph embedding features Perform feature fusion to obtain the fusion features of the EEG signal samples.

[0113] Specifically, for each EEG lead in the EEG signal sample, the time domain propagation map is embedded with features The time domain weight Embedding features with frequency domain evolution summary graph The frequency domain weight Perform dual-domain attention fusion to obtain dual-domain fusion attention weights , expressed as the following formula (8):

[0114] (8)

[0115] In the above formula (8), Represents the time domain weight The matrix and frequency domain weights Perform element-wise addition on matrices.

[0116] Next, the time domain propagation graph corresponding to each EEG lead is embedded in the feature, and the frequency domain evolution summary graph is embedded in the feature to obtain the dual-domain feature splicing value matrix , expressed as the following formula (9) to formula (11):

[0117] (9)

[0118] (10)

[0119] (11)

[0120] In the above formulas (9) to (11), is the value vector calculated for the time domain attention, represents the value vector of frequency domain attention calculation, C represents the number of frequency domain nodes (i.e., EEG leads), and F is the feature dimension of the mapping space. represents the time domain value weight matrix, represents the frequency domain value weight matrix, is the characteristic dimension of the time domain, is the feature dimension in the frequency domain, and concat represents the processing function of feature concatenation.

[0121] Finally, for each EEG lead in the EEG signal sample, the attention weight is fused by dual domains. The dual domain feature concatenation value matrix After weighted processing, the fusion feature H of the EEG signal sample is obtained, which is expressed as the following formula (12):

[0122] (12)

[0123] Fusion features of EEG signal samples , which is used to be sent to the mental workload prediction module for mental workload prediction processing. Figure 1As shown, the brain-computer interface device for fine-grained mental workload reading and interpretation also includes a mental workload prediction module, which is used to call the mental workload classifier to decode the fusion feature H to obtain the mental workload assessment result of the user being tested.

[0124] Here, in the mental workload prediction module, the fusion feature H is sent to the mental workload classifier to achieve fine decoding of the mental workload. The mental workload classifier is implemented using a convolutional neural network. First, in the feature dimension, the input fusion feature H is reduced using a one-dimensional convolution and average pooling layer. Then, in the channel dimension, the causal convolution layer is used to extract and mine the close connection between the fusion features of adjacent EEG leads. Next, the features output by the causal convolution layer are input into the nonlinear layer using the Relu activation function for activation processing. The processed results are finally mapped through the linear layer and the Softmax layer to obtain the classification probability corresponding to the mental workload classifier. The user being tested can evaluate the degree of his or her mental workload when performing the rapid sequence visual presentation task based on the classification probability output by the mental workload classifier. This achieves accurate identification of the mental workload of the user being tested.

[0125] The brain-computer interface device for fine-grained reading and interpretation of mental workload provided by the present invention collects the EEG signals of the user under test to form EEG signal samples, extracts the time domain propagation characteristics and frequency domain evolution characteristics of the EEG signal samples, and performs fusion processing of the time domain propagation characteristics and the frequency domain evolution characteristics based on the attention mechanism in the process of prediction and classification to evaluate the mental workload of the user under test. In this way, the dynamic change characteristics of mental workload are characterized by the time domain propagation characteristics and the frequency domain evolution characteristics during the classification of mental workload, thereby realizing accurate detection of mental workload.

[0126] See also Figure 4 The embodiment of the present invention also provides a method for fine-grained reading and interpretation of mental power, including the following steps 101 to 105. Figure 4 Explain one by one.

[0127] Step 101 , setting a fine-grained target-occupying search space ratio, and collecting the EEG signal of the user under test when the user under test performs a rapid sequence visual presentation task with a target-occupying search space ratio, to form an EEG signal sample.

[0128] Step 102: construct a time domain feature dynamic graph based on the EEG signal samples, and call the graph convolutional neural network to extract time domain features from the time domain feature dynamic graph to obtain the time domain propagation graph embedding features of the EEG signal samples.

[0129] Step 103: construct a frequency domain feature dynamic graph based on the EEG signal samples, and call the graph convolutional neural network group to extract frequency domain features from the frequency domain feature dynamic graph to obtain embedded features of the frequency domain evolution summary graph of the EEG signal samples.

[0130] Step 104: determine the time domain weight and frequency domain weight of the EEG signal sample according to the self-attention mechanism, and fuse the time domain propagation graph embedding features and the frequency domain evolution summary graph embedding features through the time domain weight and frequency domain weight to obtain the fused features of the EEG signal sample.

[0131] Step 105: Call the mental workload classifier to decode the fusion feature to obtain the mental workload assessment result of the user being tested.

[0132] It should be noted that the data collection process of step 101 here can refer to the data collection process of the EEG signal collection module in the fine-grained reading and interpretation device for mental power, and the time domain feature extraction process and frequency domain feature extraction process of step 102 and step 103 can refer to the processing process of the time domain propagation module and the frequency domain evolution module in the fine-grained reading and interpretation device for mental power, respectively. The feature fusion process in step 104 and the decoding process in step 105 can also refer to the processing process of the information sharing lead dynamic selection module and the mental workload prediction module in the fine-grained reading and interpretation device for mental power, respectively. Therefore, the implementation method of each step will not be repeated here.

[0133] Moreover, the beneficial effects of the fine-grained mental reading and interpretation method here and the fine-grained mental reading and interpretation device mentioned above can be referred to each other, so the beneficial effects of the fine-grained mental reading and interpretation method will not be repeated here.

[0134] In some embodiments, the mental workload assessment result obtained in step 105 can also call the mental workload assessment model, and is obtained by predicting the EEG signal samples of the tested user. The mental workload assessment model is trained with EEG signal samples, and the EEG signal samples carry the real label of mental workload. The training process of the mental workload assessment model is introduced below.

[0135] First, a large number of EEG signals of the tested users are collected in advance to form EEG signal samples, and the EEG signal samples are forward propagated in the mental workload assessment model to obtain the mental workload prediction label of the EEG signal samples. The forward propagation process here is to use the EEG signal samples as EEG signal data, and then execute the above steps 101 to 105 in sequence to obtain the corresponding mental workload assessment result. This mental workload assessment result is used as the mental workload prediction label after forward propagation, recorded as .

[0136] Next, based on the real label of mental workload Tags with Brain Workload Prediction , construct the cross entropy loss value of the mental workload evaluation model , that is, the cross entropy is calculated based on the predicted mental workload prediction label and the actual mental workload real label, and the cross entropy loss value of the mental workload evaluation model is constructed through the cross entropy, which is expressed as the following formula (13):

[0137] (13)

[0138] In the above formula (13), C is the number of EEG signal samples, represents the true label of mental workload, Represents the mental workload prediction label.

[0139] During forward propagation, the embodiment of the present invention also needs to determine the time domain mutual information loss value and the frequency domain mutual information loss value of the EEG signal samples during the forward propagation process.

[0140] Specifically, if Figure 2 As shown in the figure, during the forward propagation process, the predicted time domain propagation graph embedding feature of the EEG signal sample is determined. This predicted time domain propagation graph embedding feature is the time domain feature obtained at the end of the propagation stage, which is the time domain propagation graph embedding feature. In addition, at the end of the time domain propagation, there is also a time domain feature dynamic graph (i.e., time domain feature graph sequence) at the end of the sequence The target feature map G is Figure 2 The real target map shown in the figure is used as the real time domain target feature map to calculate the loss. Here we first determine the embedding feature of the predicted time domain propagation map The mutual information J with the real time domain target feature map G is constructed based on the mutual information J, and the time domain mutual information loss value is constructed. The calculation process can be expressed as the following formula (14):

[0141] (14)

[0142] like Figure 3 As shown in the figure, during the forward propagation process, it is also necessary to determine the predicted frequency domain evolution graph embedding feature combination of the EEG signal sample. This predicted frequency domain evolution graph embedding feature combination is the frequency domain evolution graph embedding feature combination finally obtained at the end of the propagation stage. In addition, at the end of the frequency domain propagation, there is also a target feature map combination at the end of the sequence in the frequency domain feature dynamic map (i.e., frequency domain feature map sequence) , which is the target feature map combination as the final stage, such as Figure 3 The real target graph is shown in Figure 1. This real target graph is the real frequency domain target feature graph combination, which is used to calculate the loss. Here we first determine the predicted frequency domain evolution graph embedding feature combination Combined with the real frequency domain evolution feature map The mutual information I of the EEG frequency band is calculated and then accumulated to obtain the frequency domain mutual information loss value. , the calculation process can be expressed as the following formula (15):

[0143] (15)

[0144] In the above formula (15), They represent the five EEG frequency bands corresponding to the EEG signal samples. , , , , Respectively represent the mutual information loss values ​​of the frequency domain evolution of the five EEG frequency bands.

[0145] Finally, based on the cross entropy loss value , time domain mutual information loss value And the frequency domain mutual information loss value , construct the final loss value of the mental workload assessment model. Here, we can set the corresponding time domain loss weights according to the emphasis of time domain and frequency domain. And the frequency domain loss weight , and then through the time domain loss weight The time domain mutual information loss value Weighted, through the frequency domain loss weight Mutual information loss in frequency domain Weighted, finally add the two weighted results, and then add the cross entropy loss value , the final loss value Loss of the mental workload assessment model is obtained, which is expressed as the following formula (16):

[0146] (16)

[0147] Determine the final loss value After that, through the final loss value Back propagation is performed in the mental workload assessment model. During the back propagation process, the gradient of the model is calculated, and the gradient optimization algorithm is used to optimize the gradient to update the parameters of the mental workload assessment model.

[0148] During the training process of the mental workload assessment model, it will undergo continuous iterations. One iteration is a training round. Each iteration performs a forward propagation and a backward propagation, and calculates the final loss value. , update the parameters of the mental workload assessment model. After a certain iteration, the final loss value The training ends when it starts to converge or reaches the preset total number of training rounds.

[0149] The trained mental workload assessment model predicts the EEG signal samples, thereby assessing the mental workload and obtaining the corresponding assessment results. Figure 5 As shown, Figure 5 Schematic diagram of the application scenario of the mental workload assessment model provided by the present invention. The EEG signal of the person to be assessed is collected by an EEG acquisition device to form an EEG signal sample, which is then transmitted to a computer. The mental workload assessment model (i.e. Figure 5 The decoding model in the example is used to evaluate the mental workload, and the mental workload evaluation result is transmitted to an external display device to inform the person to be evaluated of the final mental workload evaluation result.

[0150] The present invention provides a brain-computer interface device and a method for fine-grained reading and interpretation of mental workload, which collects the EEG signals of the user under test to form EEG signal samples, and extracts the time domain propagation characteristics and frequency domain evolution characteristics of the EEG signal samples. In the process of prediction and classification, the time domain propagation characteristics and the frequency domain evolution characteristics are fused based on the attention mechanism to evaluate the mental workload of the user under test. In this way, the dynamic change characteristics of mental workload are characterized by the time domain propagation characteristics and the frequency domain evolution characteristics during the classification of mental workload, thereby realizing accurate detection of mental workload.

[0151] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0152] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A brain-computer interface device for fine-grained reading and interpretation of mental power, characterized in that: The device comprises: an EEG signal acquisition module, a time domain propagation module, a frequency domain evolution module, an information sharing lead dynamic selection module, and a mental workload prediction module; The EEG signal acquisition module is used to set a fine-grained target-occupying search space ratio, and to acquire the EEG signal of the user under test when the user under test performs a fast sequence visual presentation task with the target occupying the search space ratio, to form an EEG signal sample; The time domain propagation module is used to construct a time domain feature dynamic graph according to the EEG signal sample, and call a graph convolutional neural network to extract time domain features from the time domain feature dynamic graph to obtain embedded features of the time domain propagation graph of the EEG signal sample; The frequency domain evolution module is used to construct a frequency domain feature dynamic graph according to the EEG signal samples, and call a graph convolutional neural network group to extract frequency domain features from the frequency domain feature dynamic graph to obtain embedded features of the frequency domain evolution summary graph of the EEG signal samples; The information sharing lead dynamic selection module is used to determine the time domain weight and frequency domain weight of the EEG signal sample according to the self-attention mechanism, and perform feature fusion on the time domain propagation graph embedding feature and the frequency domain evolution summary graph embedding feature through the time domain weight and the frequency domain weight to obtain the fusion feature of the EEG signal sample; The mental workload prediction module is used to call the mental workload classifier to decode the fusion feature to obtain the mental workload assessment result of the user being tested; In the time domain propagation module, the step of constructing a time domain feature dynamic graph according to the EEG signal samples includes: Dividing the EEG signal sample into 7 equally divided EEG signal segments according to the propagation time step; For each EEG signal, the EEG leads in the EEG signal are used as time domain nodes, and the correlation between the EEG leads is used as time domain edges, wherein each time domain node corresponds to a time domain node feature, and the time domain node features include activity, mobility, complexity, high-order zero crossing times, and fractal dimension; Constructing a time domain feature graph of the electroencephalogram signal according to the time domain nodes and the time domain edges; The time domain feature map sequence formed by the time domain feature maps corresponding to the 7 segments of EEG signals is used as the time domain feature dynamic map; In the frequency domain evolution module, the step of constructing a frequency domain feature dynamic graph according to the EEG signal sample includes: Dividing the EEG signal sample into three equally divided EEG signal segments according to the time step; Performing Fourier transform on each segment of the EEG signal to obtain five EEG frequency bands of the EEG signal; In each EEG frequency band of the EEG signal, the EEG leads of the EEG signal are used as frequency domain nodes, and the intra-band correlation between the EEG leads is used as a frequency domain edge, wherein each of the frequency domain nodes corresponds to a frequency domain node feature, and the frequency domain node features include power spectral density, differential entropy, power, sample entropy, and Shannon information entropy; For each EEG frequency band, a frequency domain feature map is constructed according to the frequency domain nodes and the frequency domain edges, and the frequency domain feature map combination of the five EEG frequency bands is used as the frequency domain feature map combination of the EEG signal; The frequency domain feature map sequence formed by combining the frequency domain feature maps corresponding to the three segments of EEG signals is used as the frequency domain feature dynamic map.

2. The brain-computer interface device for fine-grained reading and interpretation of mental power according to claim 1, characterized in that: The EEG signal acquisition module is also used to pre-process the collected EEG signals to obtain EEG signal samples of the tested user. The EEG signal pre-processing process includes: The collected EEG signals are sampled and processed based on the bilateral mastoid points, and the EEG signals are frequency band filtered by using a bandpass filter with a fixed frequency range after the signal re-reference processing. The EEG signal after frequency band filtering is denoised based on independent component analysis, and the denoised EEG signal is downsampled to obtain the EEG signal sample of the tested user.

3. The brain-computer interface device for fine-grained reading and interpretation of mental power according to claim 1, characterized in that: In the time domain propagation module, the calling graph convolutional neural network performs time domain feature extraction on the time domain feature dynamic graph to obtain the time domain propagation graph embedding feature of the EEG signal sample, including: In the time domain feature dynamic graph, a time domain difference feature graph between every two adjacent time domain feature graphs is determined as the time domain difference feature graph of each propagation time step; The initial time domain feature map at the head of the sequence in the time domain feature dynamic map is dynamically fused with the time domain difference feature map of each propagation time step in sequence, and finally the time domain propagation map embedding feature is obtained; Among them, when dynamic fusion is performed at each propagation time step, the initial time domain feature map is fused with the time domain difference feature map of the corresponding propagation time step to obtain a time domain propagation feature map, and a graph convolutional neural network is called to extract time domain features from the time domain propagation feature map to obtain an initial time domain feature map for dynamic fusion at the next propagation time step.

4. The brain-computer interface device for fine-grained reading and interpretation of mental power according to claim 1, characterized in that: In the frequency domain evolution module, the calling graph convolutional neural network group performs frequency domain feature extraction on the frequency domain feature dynamic graph to obtain the frequency domain evolution summary graph embedding feature of the EEG signal sample, including: In the frequency domain feature dynamic graph, determining an initial frequency domain feature graph combination at the head of the sequence and a frequency domain difference feature graph combination of a second frequency domain feature graph combination in the sequence; The frequency domain differential feature map combination is merged with the initial frequency domain feature map combination to obtain a frequency domain evolution feature map combination; Calling a graph convolutional neural network group to perform frequency domain feature extraction on the frequency domain evolution feature graph combination to obtain a frequency domain evolution graph embedding feature combination; For each EEG frequency band's frequency domain evolution graph embedding feature in the frequency domain evolution graph embedding feature combination, respectively determine the corresponding frequency band attention weight; The frequency domain evolution graph embedding feature of each EEG frequency band is weighted by the frequency band attention weight, and the weighted result is subjected to frequency band feature fusion to obtain the frequency domain evolution summary graph embedding feature of the EEG signal sample.

5. The brain-computer interface device for fine-grained reading and interpretation of mental power according to claim 1, characterized in that: In the information sharing lead dynamic selection module, the time domain weight and frequency domain weight of the EEG signal sample include: the time domain weight of each EEG lead in the EEG signal sample for the time domain propagation graph embedding feature and the frequency domain weight for the frequency domain evolution summary graph embedding feature, and the feature fusion of the time domain propagation graph embedding feature and the frequency domain evolution summary graph embedding feature is performed through the time domain weight and the frequency domain weight to obtain the fusion feature of the EEG signal sample, including: For each EEG lead in the EEG signal sample, the time domain weight for the time domain propagation graph embedding feature and the frequency domain weight for the frequency domain evolution summary graph embedding feature are fused by dual-domain attention to obtain the dual-domain fusion attention weight; Embed the time domain propagation graph corresponding to each EEG lead with features, and perform dual-domain feature splicing with the frequency domain evolution summary graph embedding features to obtain a dual-domain feature splicing value matrix; For each EEG lead in the EEG signal sample, the dual-domain feature splicing value matrix is ​​weighted by the dual-domain fusion attention weight to obtain the fusion feature of the EEG signal sample.

6. A method for fine-grained reading and interpretation of mental power, characterized in that: The method comprises: Setting a fine-grained target-occupying ratio of the search space, and collecting EEG signals of the user under test when the user under test performs a rapid sequence visual presentation task with the target occupying the search space ratio, to form EEG signal samples; Constructing a time domain feature dynamic graph according to the EEG signal sample, and calling a graph convolutional neural network to extract time domain features from the time domain feature dynamic graph to obtain a time domain propagation graph embedding feature of the EEG signal sample; Constructing a frequency domain feature dynamic graph according to the EEG signal samples, and calling a graph convolutional neural network group to extract frequency domain features from the frequency domain feature dynamic graph, so as to obtain embedding features of the frequency domain evolution summary graph of the EEG signal samples; Determine the time domain weight and frequency domain weight of the EEG signal sample according to the self-attention mechanism, and perform feature fusion on the time domain propagation graph embedding feature and the frequency domain evolution summary graph embedding feature through the time domain weight and the frequency domain weight to obtain the fusion feature of the EEG signal sample; Calling a mental workload classifier to decode the fusion feature to obtain a mental workload assessment result of the user being tested; The step of constructing a time domain feature dynamic graph according to the EEG signal sample comprises: Dividing the EEG signal sample into 7 equally divided EEG signal segments according to the propagation time step; For each EEG signal, the EEG leads in the EEG signal are used as time domain nodes, and the correlation between the EEG leads is used as time domain edges, wherein each time domain node corresponds to a time domain node feature, and the time domain node features include activity, mobility, complexity, high-order zero crossing times, and fractal dimension; Constructing a time domain feature graph of the electroencephalogram signal according to the time domain nodes and the time domain edges; The time domain feature map sequence formed by the time domain feature maps corresponding to the 7 segments of EEG signals is used as the time domain feature dynamic map; The step of constructing a frequency domain feature dynamic graph according to the EEG signal sample comprises: Dividing the EEG signal sample into three equally divided EEG signal segments according to the time step; Performing Fourier transform on each segment of the EEG signal to obtain five EEG frequency bands of the EEG signal; In each EEG frequency band of the EEG signal, the EEG leads of the EEG signal are used as frequency domain nodes, and the intra-band correlation between the EEG leads is used as a frequency domain edge, wherein each of the frequency domain nodes corresponds to a frequency domain node feature, and the frequency domain node features include power spectral density, differential entropy, power, sample entropy, and Shannon information entropy; For each EEG frequency band, a frequency domain feature map is constructed according to the frequency domain nodes and the frequency domain edges, and the frequency domain feature map combination of the five EEG frequency bands is used as the frequency domain feature map combination of the EEG signal; The frequency domain feature map sequence formed by combining the frequency domain feature maps corresponding to the three segments of EEG signals is used as the frequency domain feature dynamic map.

7. A method for fine-grained reading and interpretation of mental power, characterized in that: The method comprises: Setting a fine-grained target-occupying ratio of the search space, and collecting EEG signals of the user under test when the user under test performs a rapid sequence visual presentation task with the target occupying the search space ratio, to form EEG signal samples; The mental workload assessment model is called to predict the EEG signal samples to obtain the mental workload assessment result. The mental workload assessment model is trained by the EEG signal samples. The EEG signal samples carry the real mental workload label. The training process of the mental workload assessment model includes: By forward propagating the EEG signal sample in the mental workload assessment model, a mental workload prediction label of the EEG signal sample is obtained; Based on the real mental workload label and the predicted mental workload label, constructing a cross entropy loss value of the mental workload assessment model; Determining a time domain mutual information loss value and a frequency domain mutual information loss value of the EEG signal sample during forward propagation; Based on the cross entropy loss value, the time domain mutual information loss value and the frequency domain mutual information loss value, a final loss value of the mental workload assessment model is constructed, and back propagation is performed in the mental workload assessment model through the final loss value to update the parameters of the mental workload assessment model.

8. The method for fine-grained mental reading and interpretation according to claim 7, characterized in that: The determining of the time domain mutual information loss value and the frequency domain mutual information loss value of the EEG signal sample in the forward propagation process includes: In the forward propagation process, the predicted time domain propagation graph embedding feature of the EEG signal sample is determined, and the time domain mutual information loss value is constructed based on the mutual information between the predicted time domain propagation graph embedding feature and the real time domain target feature graph, wherein the real time domain target feature graph is the feature graph at the end of the sequence in the time domain feature dynamic graph constructed in the forward propagation process; During the forward propagation process, the predicted frequency domain evolution map embedding feature combination of the EEG signal sample is determined, and based on the mutual information between the predicted frequency domain evolution map embedding feature combination and the real frequency domain target feature map combination, a frequency domain mutual information loss value is constructed, wherein the real frequency domain target feature map combination is the feature map combination at the end of the sequence in the frequency domain feature dynamic map constructed during the forward propagation process.

Images