A method and apparatus for perceiving brain state staging fluctuations

Abstract

The application discloses a brain state staging fluctuation perception method and device, relates to the technical field of robots and artificial intelligence, and comprises the following steps: acquiring human brain electrical signals and behavior data of a person to be measured when the person to be measured performs a task; inputting the human brain electrical signals into a brain state short-term fluctuation perception model to obtain brain load changes of the person to be measured; inputting the behavior data into a brain state long-term fluctuation perception model to obtain skill level changes of the person to be measured; and inputting the brain load changes and the skill level changes into a brain state short-term and long-term fluctuation comprehensive information fusion model to generate a comprehensive output result of the person to be measured. The application can improve brain-computer mutual adaptation capability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robots and artificial intelligence, in particular to a brain state staging fluctuation perception method and device. BACKGROUND

[0002] The future new generation of robots needs to achieve "human-robot symbiosis", which is not only a key core technology, but also an urgent and necessary development direction. Human-robot symbiosis will make robots more intelligent, flexible and adaptable, and can better communicate, cooperate and interact with humans. Only through the continuous development and application of human-robot symbiosis technology, can robots be widely used in various fields and create greater value for humans. Therefore, the research and promotion of human-robot symbiosis technology is imperative.

[0003] However, in a complex human-robot symbiosis scenario, the fluctuation of brain state is particularly prominent, and there is a staging fluctuation phenomenon, which increases the difficulty of brain-machine mutual adaptation, which has become a major problem and a frontier hotspot in the international robot field. Brain state fluctuation is divided into short-term and long-term fluctuations. Short-term fluctuations are changes in brain load that affect behavior, while long-term fluctuations are changes in neural plasticity that affect skill levels. Short-term fluctuations have a small time span and often occur within each work period, while long-term fluctuations have a large time span and often accompany a person's entire career cycle. Existing research focuses on developing more accurate brain state perception methods, ignoring the staging fluctuation phenomenon of brain state, making human-robot symbiosis technology in large-scale time series encounter bottlenecks. In summary, existing research mainly has two defects: 1) existing methods ignore the staging fluctuation phenomenon of brain state, the interaction mechanism between human brain and machine brain is unclear, and it is difficult to perceive the staging fluctuation of brain state; 2) there is a lack of brain state staging fluctuation perception devices, which cannot efficiently detect the staging fluctuation of brain state from the hardware. SUMMARY

[0004] The purpose of the present application is to provide a brain state staging fluctuation perception method and device, which can solve the problem of brain state staging fluctuation perception in a human-robot symbiosis scenario and improve the brain-machine mutual adaptation capability.

[0005] To achieve the above-mentioned purpose, the present application provides the following solutions:

[0006] In a first aspect, the present application provides a brain state staging fluctuation perception method, comprising:

[0007] Obtaining human brain electrical signals and behavioral data of a person to be tested when performing a task.

[0008] Inputting the human brain electrical signals into a brain state short-term fluctuation perception model to obtain the change in brain load of the person to be tested.

[0009] inputting the behavior data into a brain state long-term fluctuation perception model to obtain the skill level change of the person to be tested.

[0010] inputting the brain load change and the skill level change into a brain state short-term and long-term fluctuation comprehensive information fusion model to generate a comprehensive output result of the person to be tested.

[0011] Optionally, after the comprehensive output result of the person to be tested is generated, the method further comprises:

[0012] updating a person to be tested database according to the comprehensive output result of the person to be tested and the human brain electrical signals and the behavior data of the person to be tested when performing the task.

[0013] Optionally, the construction method of the brain state short-term fluctuation perception model comprises:

[0014] acquiring the human brain electrical signals of the person to be tested when controlling the robot to perform the target task by using a wireless brain electrical signal acquisition device.

[0015] preprocessing the human brain electrical signals; the preprocessing comprises removing a trend item, adopting polynomial fitting discrete data points, filter processing and down-sampling processing to obtain the processed human brain electrical signals.

[0016] based on wavelet decomposition and reconstruction, adopting a fifth-order vanishing moment Daubechies wavelet basis function to decompose and reconstruct five kinds of rhythm waves from the processed human brain electrical signals.

[0017] calculating differential entropy characteristic values of each frequency band according to the rhythm waves; the differential entropy characteristic values are used to analyze and judge whether the extracted differential entropy characteristics are effective representations of brain load according to mutual information parameters.

[0018] using equidistant azimuth projection method and Clough-Tocher interpolation method to perform spatial coding on the differential entropy characteristics to generate a two-dimensional feature matrix.

[0019] taking the two-dimensional feature matrix as input and taking effective representations of brain load as output to construct a brain state short-term fluctuation perception model; the brain state short-term fluctuation perception model is composed of a lightweight convolutional neural network classifier; the lightweight convolutional neural network classifier adopts a small convolutional neural network architecture design, including four convolutional layers, a maximum value pooling layer, a convolutional layer, a maximum value pooling layer and three fully connected layers, and ReLU, Tanh and Softmax activation function processing.

[0020] Optionally, the calculation formula of the differential entropy characteristic values is:

[0021] .

[0022] In the formula, a probability density function representing a time series; μ and σ represent the mean and standard deviation of a Gaussian distribution, respectively, L denotes the number of decomposition layers, x j denotes the first j band of the electroencephalogram rhythm wave.

[0023] Optionally, the downsampling processing is to reduce the sampling frequency of the human electroencephalogram signal from 1000 Hz to 256 Hz.

[0024] Optionally, the method for constructing the brain state long-term fluctuation perception model is:

[0025] Obtain the behavior data of the person to be tested when controlling the robot to perform the target task; the behavior data includes total training time, single training time mean, training times, training interval length, interval length mean, interval length standard deviation, self-evaluation and task performance score.

[0026] Preprocess the behavior data; the preprocessing includes normalization processing and polynomial fitting.

[0027] Take the feature matrix of skill level as input and the result of skill level as output to construct a brain state long-term fluctuation perception model; the brain state long-term fluctuation perception model is composed of a self-organizing neural network classifier; the self-organizing neural network classifier adopts an ART2 type neural network model; the ART2 type neural network model is composed of an attention subsystem and a directional subsystem; the attention subsystem includes an input comparison layer and a recognition layer.

[0028] Optionally, the formula expression input in the input comparison layer is:

[0029] .

[0030] In the formula, N xi denotes the value of each node in the network, x ∈{ s , p , u , v , q,z};C a , C b and C e are constants, and their values are 10, 10, and 1e-8, respectively; s ni (i=1,……, I) is the i-th component of the input vector; |Z|, |V|, |P| are the modules of the corresponding layers; is the connection weight from the recognition layer to the input comparison layer; the function h(·) represents a nonlinear transformation on the transmitted signal, , and α is a threshold parameter, , d is the network gain, j， represents a neuron, is the output of the jth neuron in the F2 layer.

[0031] Optionally, the formula expression of the comprehensive output result is:

[0032] .

[0033] wherein a is the brain load recognition result, λ is the skill level recognition result, , w1 and w2 are weight coefficients of the mean R ave and the standard deviation R sta .

[0034] In a second aspect, the application provides a perception device based on the perception method of the brain state staging fluctuation, comprising:

[0035] an electroencephalogram collector, an electroencephalogram preprocessor, an electroencephalogram feature extractor, a brain load recognizer, a behavioral data reader, a behavioral data preprocessor, a skill level recognizer, and a recognition result coupler.

[0036] The electroencephalogram collector is configured to acquire the human electroencephalogram signal of the person to be tested.

[0037] The electroencephalogram preprocessor is configured to preprocess the human electroencephalogram signal.

[0038] The electroencephalogram feature extractor is configured to extract the features containing the brain load information in the electroencephalogram.

[0039] The brain load recognizer is configured to recognize and classify different brain load states.

[0040] The behavioral data reader is configured to acquire the behavioral data of the person to be tested.

[0041] The behavioral data preprocessor is configured to preprocess the read behavioral data.

[0042] The skill level recognizer is configured to recognize and separate different skill levels.

[0043] The recognition result coupler is configured to couple the output results of the brain load recognizer and the skill level recognizer to obtain the comprehensive output result of the person to be tested.

[0044] Optionally, it further comprises a data storage configured to update the person to be tested database according to the comprehensive output result of the person to be tested and the human electroencephalogram signal and the behavioral data of the person to be tested when performing the task.

[0045] According to the specific embodiments provided in the present application, the present application discloses the following technical effects:

[0046] The present application provides a brain state staging fluctuation perception method and device. First, by obtaining the human brain electrical signal and behavioral data of the person to be tested when performing a task, the brain activity state and behavior performance of the person to be tested under a specific task can be understood. Then, the human brain electrical signal is input into a brain state short-term fluctuation perception model to obtain the brain load change of the person to be tested. By analyzing the change of the brain electrical signal, the cognitive load state of the person to be tested when performing the task can be reflected in real time, thereby helping the brain-computer interface system to better understand and adapt to the brain power demand of the individual. Then, the behavioral data is input into a brain state long-term fluctuation perception model to obtain the skill level change of the person to be tested. By analyzing the long-term trend of the behavioral data, the change of the person to be tested in the skill learning and development process can be revealed, which helps the brain-computer interface system to adjust the interaction strategy according to the skill level of the individual and improve the interaction efficiency. Finally, the brain load change and the skill level change are input into a brain state short-term and long-term fluctuation comprehensive information fusion model to generate the comprehensive output result of the person to be tested. By fusing the short-term and long-term brain state information, the current state and development trend of the person to be tested can be comprehensively reflected, providing more accurate and comprehensive individual state information for the brain-computer interface system, thereby further improving the brain-computer interface adaptation ability. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0048] Figure 1 A brain state staging fluctuation perception method block diagram is provided for an embodiment of the present application.

[0049] Figure 2 An electrode arrangement position diagram of an electroencephalogram signal acquisition device is provided for an embodiment of the present application.

[0050] Figure 3 A brain state staging fluctuation perception network diagram is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0051] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0052] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0053] Embodiment one

[0054] The embodiment provides a brain state staging fluctuation perception method, comprising:

[0055] Step 1: obtaining human brain electrical signals and behavior data of a person to be tested when performing a task.

[0056] Step 2: inputting the human brain electrical signals into a brain state short-term fluctuation perception model to obtain a change in mental load of the person to be tested.

[0057] Step 3: inputting the behavior data into a brain state long-term fluctuation perception model to obtain a change in skill level of the person to be tested.

[0058] Step 4: inputting the change in mental load and the change in skill level into a brain state short-term and long-term fluctuation comprehensive information fusion model to generate a comprehensive output result of the person to be tested.

[0059] After generating the comprehensive output result of the person to be tested, further comprising:

[0060] Step 5: updating a person to be tested database according to the comprehensive output result of the person to be tested and the human brain electrical signals and behavior data of the person to be tested when performing a task.

[0061] In some embodiments, a brain state short-term fluctuation (mental load) perception model is constructed, which can be as follows:

[0062] Step 11: collecting human brain electrical signals of a person to be tested when controlling a robot to perform a target task by using a wireless electroencephalogram acquisition device; an electrode arrangement position diagram of the electroencephalogram acquisition device is shown in Figure 2 .

[0063] Step 12: preprocessing the human brain electrical signals; the preprocessing includes removing a trend item, adopting polynomial fitting discrete data points, filtering processing and down-sampling processing to obtain processed human brain electrical signals.

[0064] Step 13: Based on wavelet decomposition and reconstruction, the Daubechies wavelet basis function with the fifth-order vanishing distance is used to decompose and reconstruct five rhythm waves from the processed human EEG signal.

[0065] Step 14: Calculate the differential entropy characteristic value of each frequency band based on the rhythm waves described above; the differential entropy characteristic value is used to determine whether the extracted differential entropy feature is an effective characterization of mental load based on mutual information parameter analysis.

[0066] Step 15: Spatial encoding of differential entropy features is performed using the equidistant azimuth projection method and the Kraf-Tocher interpolation method to generate a two-dimensional feature matrix.

[0067] Step 16: Using a two-dimensional feature matrix as input and an effective representation of mental workload as output, construct a short-term brain state fluctuation perception model; the short-term brain state fluctuation perception model is composed of a lightweight convolutional neural network classifier; the lightweight convolutional neural network classifier adopts a small convolutional neural network architecture design, including four convolutional layers, a max pooling layer, a convolutional layer, a max pooling layer and three fully connected layers, as well as ReLU, Tanh and Softmax activation functions.

[0068] Specifically, in step 11, a 32-lead wireless EEG acquisition device is used to acquire human brain signals.

[0069] Specifically, in step 12, the EEG signal is preprocessed, which mainly consists of three steps: 1) removing the trend term by using a polynomial to fit discrete data points, forming a trend term curve, and then subtracting the trend term curve; 2) filtering by using a fourth-order Butterworth bandpass filter to process the EEG signal, retaining the frequency band signal of 0.5-45.0Hz, thereby eliminating the interference of low-frequency electrooculography and high-frequency electromyography signals to a certain extent; 3) downsampling by reducing the sampling frequency of the collected signal from 1000Hz to 256Hz to improve the efficiency of subsequent operations.

[0070] Specifically, steps 13-14 involve feature extraction from the EEG signals, which mainly consists of three steps:

[0071] 1) Wavelet decomposition and reconstruction. The Daubechies wavelet basis function with the fifth-order vanishing distance is used to decompose and reconstruct five rhythm waves from the EEG signal. The wavelet decomposition formula is shown in Equation (1).

[0072] (1).

[0073] In the formula, Indicates the first j The rhythmic waves of EEG signals in a frequency band are discontinuous signals of finite length; L Indicates the number of decomposition levels; Aj representing the approximate component of the j th frequency band; D j representing the detailed component of the j th frequency band with different scales. The resulting five kinds of rhythm waves are δ (0.5-3 Hz), θ (4-7 Hz), α (8-15 Hz), β (16-31 Hz) and γ (>32 Hz).

[0074] 2) Calculate the differential entropy feature. The differential entropy (Differential Entropy, DE ) feature value of each frequency band can be used to measure the complexity of the time series signal. For a fixed length of electroencephalogram signal sequence, the approximate calculation DE can be carried out as shown in equation (2): DE

[0075] (2).

[0076] In the formula, denotes the probability density function of the time series; μ and σ respectively denote the mean and standard deviation of the Gaussian distribution.

[0077] 3) Perform mutual information (Mutual Information, MI ) parameter analysis to determine whether the extracted features can effectively represent mental load. Calculate DE the mutual information between MI and the mental load label. In probability theory and information theory, the MI of two random variables refers to the degree of measurement of the mutual dependence between the variables. Two discrete random variables MI representing the X F and Y L mutual information is defined as equation (3):

[0078] (3).

[0079] In the formula, is the joint probability distribution function of X F and Y L ; and and are the marginal probability distribution functions of X F and Y L respectively.

[0080] In step 15, the mental load is identified, mainly divided into two steps:​

[0081] 1) Time-domain-frequency-domain-space trinity encoding processing. The spatial encoding of the multi-lead electroencephalogram signal features is performed using the Azimuthal Equidistant Projection (AEP) and the Clough-Tocher (CT) interpolation method. The data containing the three-dimensional spatial distribution structure information of the electrodes are mapped to form a two-dimensional feature matrix, thereby realizing the extraction of the feature matrix containing the time-domain-frequency-domain-space trinity from the electroencephalogram signal.

[0082] 2) A lightweight convolutional neural network classifier is built to classify and identify mental load. The input of the classifier is the feature matrix representing the mental load (denoted by Sm), and the output is the result of the mental load (denoted by a). Based on the small Convolutional Neural Network (CNN) architecture, a lightweight mental state perceiver is designed, and the main structure and the series relationship of each module are as follows: four convolutional layers → maximum pooling layer → convolutional layer → maximum pooling layer → three fully connected layers. The convolutional layer is mainly used for convolution operation on Feature Maps to extract high-level features of electroencephalogram signals. All convolutional layers use a two-dimensional kernel with a size of 3×3. By stacking convolutional layers with small receptive fields, a higher-dimensional effective receptive field is formed, and the parameter quantity of the network is reduced. In addition, the convolutional layer input is filled with 1 pixel to maintain the spatial resolution after convolution. The maximum pooling layer is added to compress the overall feature matrix, highlight important feature information, reduce the amount of calculation, and alleviate the problem of overfitting. The pooling layer uses a two-dimensional kernel with a size of 2×2. Each convolutional layer is equipped with a ReLU activation function, the fully connected layer is equipped with a Tanh activation function, and the last output layer is equipped with a Softmax processing. Through the measures of compressing the network structure, designing convolutional layers with small receptive fields, and adding maximum pooling layers, the model achieves lightweight operation to a certain extent. The loss function of the mental state perceiver uses the FocalLoss function, which aims to solve the class imbalance problem in classification. The perceiver based on the FocalLoss function can reduce the weight of easily identifiable class samples during the training process, and improve the weight of difficult-to-classify sample classes, thereby improving the recognition accuracy of each class and enhancing the generalization ability of the model.

[0083] In some embodiments, a brain state long-term fluctuation (skill level) perception model is constructed, which can be specifically as follows:

[0084] Obtaining behavior data of the to-be-tested personnel when controlling the robot to perform the target task; the behavior data includes total training time, single training time average, training times, training interval length, interval length average, interval length standard deviation, self-evaluation and task performance score.

[0085] Preprocessing the behavior data; the preprocessing includes normalization processing and polynomial fitting.

[0086] Taking the feature matrix of skill level as input and the result of skill level as output, a brain state long-term fluctuation perception model is constructed; the brain state long-term fluctuation perception model is composed of a self-organizing neural network classifier; the self-organizing neural network classifier adopts an ART2 type neural network model; the ART2 type neural network model is composed of an attention subsystem and a directional subsystem; the attention subsystem includes an input comparison layer and an identification layer.

[0087] Specifically, the construction process includes three steps:

[0088] 1. Automatically recording the behavior data of the person performing the task by the computer, including three parts of parameters related to training time T (total training time T sum , single training time average T sig , training times T num , training interval length T int , interval length average T ave , and interval length standard deviation T sta ), self-evaluation V of the to-be-tested personnel and task performance score R (highest value R max , lowest value R min , average value R ave , standard deviation R sta ).

[0089] 2. Preprocessing the behavior data, mainly divided into two steps: 1) normalization processing of the behavior data, normalizing the behavior data of each to-be-tested personnel so that all data are between 0 and 1; 2) polynomial fitting, fitting discrete data points with a 3rd order polynomial to form a spline curve, thereby effectively reducing the influence of noise and mining the inherent change law of the data.

[0090] 3. Identifying the skill level, mainly divided into two steps: 1) correlation analysis; 2) building a self-organizing neural network classifier to classify and identify the skill level, the input of the classifier being a feature matrix representing the skill level (denoted by S n , and the output being the result of the skill level (denoted by λ). An ART2 type model with an input that can be an arbitrary analog vector is adopted. The ART2 neural network is composed of an attention subsystem and a directional subsystem, and the network structure is as follows: Figure 3The input is shown (only one neuron is shown in each sublayer). The attention subsystem can be divided into the input comparison layer (F1 layer) and the recognition layer (F2 layer). The F1 layer is composed of two positive feedback loops with six sublayers, which are used to normalize the input, suppress noise and enhance useful signals. The F2 layer is used to let neurons compete with each other and select the pattern prototype that best matches the input pattern. The orientation subsystem compares the similarity of the input with the pattern model, and if the threshold value ρ set by the alarm parameter is exceeded, the resonance learning module is entered to adjust the connection weights from the F1 layer to the F2 layer and the weights from the F2 layer to the F1 layer. If the test is not passed, the activated neuron in the F2 layer is shielded, and the winning neuron is again competed and compared with the input. If all the neurons that have stored pattern prototypes do not pass the test, a new neuron is created to store the current pattern. The skill level (the degree of brain plasticity) is divided into 10 levels (ranging from 1 to 2, with a resolution of 0.1), and each time the sensor identifies a new category, it is considered that the skill level has changed, and λ is set to increase by 1 level. Figure 3 Each vector is represented by the ith component, the variables in the hollow circle are added or subtracted, and the variables in the solid circle are modulus operations. The skill level sensor effectively simulates the self-organization and self-growth characteristics of neuron cells in the brain by constructing a self-organizing neural network. The bionic sensor network realizes the unification of artificial structure and biological structure. The input of each neuron in the F1 layer is as shown in formula (4):

[0091] (4).

[0092] In the formula, N xi , wherein V represents the value of each node in the network, x ∈{ s , p , u , v , q,z}, C a , C b and C e are constants, and their values are 10, 10, 1e-8 respectively; s ni (i=1,……, I) is the ith component of the input vector; |Z|, |V|, |P| are the modules of the corresponding layers; is the connection weight from the F2 layer to the F1 layer; the function h(·) represents a nonlinear transformation of the transmitted signal, which is generally represented as formula (5), wherein α is a threshold parameter, and its value is 0 to 1.

[0093] (5).

[0094] The function g(·) in formula (4) makes the winning neuron output and other neurons are inhibited, if the neuron wins in the competition, as shown in formula (6):

[0095] (6).

[0096] In the formula, d is the network gain, which is a constant, and its value is 0.75; the role of the F2 layer is to receive the input of the F1 layer, and the output of the F2 layer is calculated by the connection weight value from the F1 layer to the F2 layer The output of the jth neuron of the F2 layer is calculated as (j=1,……, J); The output of the winning neuron (i.e. λ ) is shown in detail in formula (7):

[0097] (7).

[0098] The input of the ith neuron of the R layer is N ri , and the modulus of the layer vector is |R|, which is shown in detail in formula (8):

[0099] (8).

[0100] In the formula, C c is a constant, and its value is 0.3; |U| is the modulus of the corresponding layer; a warning parameter ρ (0<ρ<1) is set, if |R|>ρ, the similarity test is passed, otherwise the reorganization signal is sent by the directional subsystem. If the test is passed, the weight values and need to be adjusted. It should be noted that only the weight value connected to the winning neuron will be adjusted. The adjustment method in the fast learning mode is shown in formula (9):

[0101] (9).

[0102] Regarding the input quantity s n , the embodiment calculates the training time related parameters (T sum , T sig , T num , T int , T ave , T sta ) respectively, and the Pearson correlation coefficient between the self-evaluation V two types of parameters and λ, and the absolute threshold of the correlation coefficient is greater than 0.5. The parameters are T sum , T sig , T int and V, which are used as input vector parameters of the skill level sensor, and s n is obtained by combining the four parameters, as shown in formula (10):

[0103] (10).

[0104] The output of the skill level sensor is λ, and the skill level (brain plasticity) factor λ is evaluated by the task performance score. It is found through research that λ is only related to the mean R ave and the standard deviation R sta of the task performance score, and is specifically expressed as formula (11):

[0105] (11).

[0106] In the formula, w1 and w2 are weight coefficients of the mean R ave and the standard deviation R sta , and w1 = 2 and w2 = -1 are set.

[0107] In some embodiments, a brain state short-term and long-term fluctuation comprehensive information fusion model, i.e., a fusion model of the mental workload sensor and the skill level sensor, is constructed. First, the mental workload identification result a and the skill level identification result λ are collected; then the respective results are coupled, and the coupling mode is shown in formula 12, so as to obtain a comprehensive output result c.

[0108] (12).

[0109] Finally, an input and output data storage model is constructed. After identification, the data generated in this operation are stored, which are: input data (human brain electrical signals, behavioral data), output data (mental workload data, skill level data); and then the database for each to-be-tested person is updated.

[0110] Embodiment Two

[0111] As shown in Figure 1 , the embodiment provides a sensing device based on the sensing method of brain state short-term and long-term fluctuation, which comprises:

[0112] a brain electrical signal collector, a brain electrical signal preprocessor, a brain electrical signal feature extractor, a mental workload identifier, a behavioral data reader, a behavioral data preprocessor, a skill level identifier, and an identification result coupler.

[0113] The brain electrical signal collector is used to acquire the human brain electrical signals of a to-be-tested person.

[0114] The brain electrical signal preprocessor is used to pre-process the human brain electrical signals.

[0115] The brain electrical signal feature extractor is used to extract the features of the mental workload information contained in the brain electrical signals.

[0116] The brain load identifier is used to identify and classify different brain load states.

[0117] The behavior data reader is used to obtain the behavior data of the person to be tested.

[0118] The behavior data preprocessor is responsible for preprocessing the read behavior data.

[0119] The skill level identifier is used to identify and separate different skill levels.

[0120] The identification result coupler is used to couple the output results of the brain load identifier and the skill level identifier to obtain the comprehensive output result of the person to be tested.

[0121] Further comprising a data storage for updating the database of the person to be tested according to the comprehensive output result of the person to be tested and the human brain electrical signals and behavior data of the person to be tested when performing the task.

[0122] Specifically, the specific logical relationship is as shown in Figure 1

[0123] The EEG preprocessor is responsible for preprocessing the EEG signals, mainly including removing the trend item of the EEG signals, filtering to remove noise and reducing sampling; the EEG feature extractor is responsible for extracting the features of the brain load information contained in the EEG, mainly including wavelet decomposition and reconstruction of the EEG signals, calculation of differential entropy features and mutual information analysis; the brain load identifier is responsible for identifying and classifying different brain load states, mainly including time domain-frequency domain-space domain three-in-one coding processing, and using a lightweight convolutional neural network classifier to identify the brain load of the person to be tested; the behavior data reader is responsible for obtaining the behavior data of the person; the behavior data preprocessor is responsible for preprocessing the read behavior data, mainly including normalization operation and polynomial fitting; the skill level identifier is responsible for identifying and separating different skill levels, mainly including correlation analysis and using a self-organizing neural network to identify the skill level of the person to be tested; the identification result coupler is responsible for coupling the results of the two identifiers to output the comprehensive state of the person to be tested; the data storage is responsible for recording the input data (EEG data, behavior data), output data (brain load data, skill level data), and updating the database for each person to be tested in real time.

[0124] In summary, the present application has the following technical effects:

[0125] ​The application provides a brain state staging fluctuation perception method and device, which realizes comprehensive detection and analysis of brain state staging fluctuation by respectively designing a brain state short-term fluctuation perceiver, a long-term fluctuation perceiver, and an identification result coupler, solves two defects of the prior art, i.e., (1) brain state staging fluctuation is ignored, the interaction mechanism of the human brain and the machine brain is unknown, and it is difficult to realize the perception of brain state staging fluctuation; and (2) there is a lack of brain state staging fluctuation perception devices, and brain state staging fluctuation cannot be efficiently detected from the hardware. The method combining electroencephalogram and behavior data breaks through the problems of the prior art, improves the performance of brain state perception, and provides a technical implementation approach for promoting the adaptation of the machine brain to the human and realizing the human-machine integration goal.

[0126] The technical features of the above embodiments can be combined in any manner. To make the description concise, all possible combinations of the technical features in the above embodiments are not described, but as long as the combinations of the technical features do not have contradictions, they should be considered as the scope of the present application.

[0127] The principles and implementation modes of the present application are described by using specific examples in the present application. The above embodiments are only used to help understand the method of the present application and its core idea. For those skilled in the art, the specific implementation modes and application ranges can be changed according to the idea of the present application. In summary, the content of the present application should not be understood as a limitation.

Claims

1. A method of awareness of brain state staging fluctuations, characterized by, The method comprises the following steps: obtaining the human brain electrical signals and the behavior data of the person to be tested when performing a task; inputting the human brain electrical signals into a brain state short-term fluctuation perception model to obtain the mental load change of the person to be tested; inputting the behavior data into a brain state long-term fluctuation perception model to obtain the skill level change of the person to be tested; inputting the mental load change and the skill level change into a brain state short-term and long-term fluctuation comprehensive information fusion model to generate the comprehensive output result of the person to be tested.

2. The method of claim 1, wherein the method further comprises: After generating the comprehensive output result of the person to be tested, the method further comprises the following steps: updating the person to be tested database according to the comprehensive output result of the person to be tested and the human brain electrical signals and the behavior data of the person to be tested when performing a task.

3. The method of claim 1, wherein the method further comprises: The construction method of the brain state short-term fluctuation perception model comprises the following steps: collecting the human brain electrical signals of the person to be tested when controlling a robot to perform a target task by using a wireless electroencephalogram acquisition device; preprocessing the human brain electrical signals; the preprocessing comprises the following steps: removing a trend item, adopting a polynomial fitting discrete data point, filtering processing, and reducing sampling processing to obtain the processed human brain electrical signals; based on wavelet decomposition and reconstruction, adopting a five-order vanishing moment Daubechies wavelet base function to decompose and reconstruct five rhythm waves from the processed human brain electrical signals; calculating the differential entropy characteristic value of each frequency band according to each rhythm wave; the differential entropy characteristic value is used to determine whether the extracted differential entropy characteristic is an effective representation of the mental load according to the mutual information parameter analysis; using the equidistant azimuth projection method and the Clough-Tocher interpolation method to perform spatial coding on the differential entropy characteristic to generate a two-dimensional feature matrix; taking the two-dimensional feature matrix as the input and taking the effective representation of the mental load as the output to construct the brain state short-term fluctuation perception model; the brain state short-term fluctuation perception model is composed of a lightweight convolutional neural network classifier; the lightweight convolutional neural network classifier adopts a small convolutional neural network architecture design, and comprises four convolutional layers, a maximum value pooling layer, a convolutional layer, a maximum value pooling layer, three fully connected layers, and ReLU, Tanh and Softmax activation function processing.

4. The method of claim 3, wherein the brain state staging fluctuation is sensed by a method comprising: The calculation formula of the differential entropy characteristic value is as follows: ； wherein denotes the probability density function of the time series; μ and σ denote the mean and the standard deviation of the Gaussian distribution, respectively, L denotes the number of decomposition levels, x j denotes the rhythm wave of the electroencephalogram signal of the j th frequency band.

5. The method of claim 4, wherein the method further comprises: the sampling frequency of the human brain electrical signals is reduced from 1000 Hz to 256 Hz.

6. The method of claim 1, wherein the method further comprises: The construction method of the brain state long-term fluctuation perception model comprises the following steps: obtaining the behavior data of the person to be tested when controlling a robot to perform a target task; the behavior data comprises the total training time, the single training time average, the training frequency, the training interval time, the interval time average, the interval time standard deviation, the self-evaluation and the task performance score; preprocessing the behavior data; the preprocessing comprises normalization processing and polynomial fitting; taking the feature matrix of the skill level as the input and taking the result of the skill level as the output to construct the brain state long-term fluctuation perception model; the brain state long-term fluctuation perception model is composed of a self-organizing neural network classifier; the self-organizing neural network classifier adopts an ART2 type neural network model; the ART2 type neural network model comprises an attention subsystem and a directional subsystem; the attention subsystem comprises an input comparison layer and an identification layer.

7. The method of claim 6, wherein the method further comprises: The formula expression input in the input comparison layer is: ； wherein, N xi denotes the value of each node in the network, x ∈{ s , p , u , v , q,z}; C a , C b and C e are constants whose values are 10, 10, 1e-8 respectively; s ni (i=1, …, I) is the i-th component of the input vector; |Z|, |V|, |P| are the modules of the corresponding layers respectively; is the connection weight of the identification layer to the input comparison layer; the function h(·) represents that the transmitted signal has been nonlinearly transformed, , α is the threshold parameter, , d is the network gain, j， denotes the neuron, is the output of the j-th neuron of the F2 layer.

8. The method of claim 1, wherein the method further comprises: The formula expression of the comprehensive output result is: ； Wherein, a is the brain load recognition result, λ is the skill level recognition result, , w1 and w2 are weight coefficients of the mean value R ave and the standard deviation R sta of the task performance score respectively.

9. A perception device based on the perception method of a brain state staging fluctuation according to any one of claims 1 to 8, characterized in that, Comprise: An electroencephalogram collector, an electroencephalogram preprocessor, an electroencephalogram feature extractor, a mental workload identifier, a behavioral data reader, a behavioral data preprocessor, a skill level identifier, and an identification result coupler; The electroencephalogram collector is configured to acquire the human electroencephalogram signal of the person to be tested; The electroencephalogram preprocessor is configured to preprocess the human electroencephalogram signal; The electroencephalogram feature extractor is configured to extract the features containing the mental workload information in the electroencephalogram; The mental workload identifier is configured to identify and classify different mental workload states; The behavioral data reader is configured to acquire the behavioral data of the person to be tested; The behavioral data preprocessor is configured to preprocess the read behavioral data; The skill level identifier is configured to identify and separate different skill levels; The identification result coupler is configured to couple the output results of the mental workload identifier and the skill level identifier to obtain the comprehensive output result of the person to be tested.

10. The perception device of claim 9, wherein, Further comprise: A data storage is configured to update the person to be tested database according to the comprehensive output result of the person to be tested and the human electroencephalogram signal and the behavioral data of the person to be tested when performing the task.

Images