Cognitive state recognition method and device based on multi-modal signals, equipment and medium

Abstract

The present application relates to educational information technology, and discloses a cognitive state recognition method, device, equipment and medium based on multi-modal signals. The method comprises: performing feature extraction operations on preprocessed electroencephalogram and peripheral image sets of a target user, respectively, to obtain first feature sequences and second feature sequence sets; performing channel self-attention weight configuration and two-dimensional mapping operations based on the first feature sequences to obtain first mapping feature sequences; performing time-based multiple linear projection attention weight configuration and one-dimensional mapping operations based on the second feature sequence sets to obtain second mapping feature sequence sets; mapping the first mapping feature sequences and the second mapping feature sequence sets to a target space to obtain a multi-modal fusion feature sequence set, and performing full connection classification judgment on the multi-modal fusion feature sequence set to obtain high and low order cognitive state recognition results of the target user. The present application can improve the accuracy of high and low order cognitive state recognition of users.

Description

Technical Field

[0001] This invention relates to the field of educational informatization technology, and in particular to a cognitive state recognition method, apparatus, device, and computer-readable storage medium based on multimodal signals. Background Technology

[0002] With the development of artificial intelligence, the recognition of users' cognitive states can be applied to all aspects of our lives. For example, in user experience research, the effectiveness of products or services can be evaluated by recognizing users' cognitive states; in the field of education, the recognition of high and low-order cognitive states can be used to assess students' learning and emotional states; in the field of automotive driving, cognitive state recognition can be used to monitor drivers' attention, fatigue, and other states. Currently, in the field of educational informatization, there is no method to accurately identify learners' high and low-order cognitive states. Summary of the Invention

[0003] This invention provides a cognitive state recognition method, device, equipment, and storage medium based on multimodal signals. Its main purpose is to reduce the heterogeneity gap between various modal data while representing the complex synergistic relationship between multiple physiological signals through attention mechanisms, thereby improving the accuracy of recognizing the user's high and low-order cognitive states.

[0004] To achieve the above objectives, the present invention provides a cognitive state recognition method based on multimodal signals, comprising:

[0005] The initial EEG and the initial peripheral image set of the preset peripheral physiological signals of the target user are obtained, and the initial EEG and the initial peripheral image set are subjected to noise reduction and cleaning operations according to the preset image preprocessing strategy to obtain the EEG and peripheral image set.

[0006] Using a pre-trained recurrent neural network layer, feature extraction is performed on the electroencephalogram to obtain a first feature sequence, and feature extraction is performed on the peripheral image set to obtain a second feature sequence set;

[0007] Using the first feature mapping network in the pre-trained feature mapping layer, the first feature sequence is configured with self-attention weights based on each channel to obtain a channel attention-enhanced feature sequence, and the channel attention-enhanced feature sequence is subjected to a time-channel based two-dimensional feature mapping operation to obtain a first mapped feature sequence.

[0008] Using the second feature mapping network in the feature mapping layer, the second feature sequence set is configured with multiple linear projection attention weights based on time steps to obtain a time attention-enhanced feature sequence set, and a one-dimensional feature mapping operation based on time steps is performed on the time attention-enhanced feature sequence set to obtain a second mapped feature sequence set.

[0009] Using a pre-trained fusion and classification layer, the first mapping feature sequence and the second mapping feature sequence set are mapped to the target space to obtain a multimodal fusion feature sequence set. The multimodal fusion feature sequence set is then subjected to fully connected classification to obtain the high and low order cognitive state recognition results of the target user.

[0010] Optionally, the step of using the first feature mapping network in the pre-trained feature mapping layer to configure the self-attention weights of each channel on the first feature sequence to obtain a channel attention-enhanced feature sequence, and then performing a time-channel based two-dimensional feature mapping operation on the channel attention-enhanced feature sequence to obtain a first mapped feature sequence, includes:

[0011] The first feature sequence is transformed by the first multilayer perceptual layer in the first feature mapping network based on the self-attention of each channel to obtain the weight configuration feature sequence. The first feature sequence and the weight configuration feature sequence are then connected by a residual skip connection to obtain the channel attention enhanced feature sequence.

[0012] Using the two-dimensional convolutional neural network in the first feature mapping network, an adaptive number of two-dimensional double-layer convolutional pooling operations are performed on the channel attention enhancement feature sequence to obtain a dimension-reduced enhancement feature sequence;

[0013] Using the second multilayer perception layer in the first feature mapping network, the dimensionality reduction and enhancement feature sequence is mapped into a time attention feature sequence and a channel attention feature sequence according to the preset time-channel dimension, and the time attention feature sequence and the channel attention feature sequence are fused to obtain the fused feature result.

[0014] The fused feature result is subjected to a fully convolutional operation using the fully convolutional network in the first feature mapping network to obtain the first mapping feature sequence.

[0015] Optionally, the step of using the first multilayer perceptual layer in the first feature mapping network to perform a nonlinear transformation on the first feature sequence based on self-attention of each channel to obtain a weighted feature sequence includes:

[0016] The first feature sequence is subjected to average pooling to obtain a first average pooled feature sequence, and the first feature sequence is subjected to max pooling to obtain a first max pooled feature sequence.

[0017] Using the first multilayer perceptron in the first feature mapping network, attention weights are calculated on the first average pooling feature sequence and the first max pooling feature sequence respectively to obtain the average attention sequence and the max attention sequence respectively.

[0018] The average attention sequence and the maximum attention sequence are summed and averaged to obtain the attention fusion sequence. The attention fusion sequence is then output using the sigmoid activation function to obtain the weight configuration feature sequence.

[0019] Optionally, the step of using the second feature mapping network in the feature mapping layer to perform multiple linear projection attention weight configurations on the second feature sequence set based on time steps to obtain a time-attention-enhanced feature sequence set includes:

[0020] The second feature sequence set is subjected to two linear transformations using the multilayer perceptron in the second feature mapping network to obtain the transformed feature result.

[0021] The attention distribution is obtained by normalizing the transformed feature results using the softmax function.

[0022] The second feature sequence set is weighted according to the attention distribution to obtain the time attention-enhanced feature sequence set.

[0023] Optionally, before performing feature extraction on the electroencephalogram using the pre-trained recurrent neural network layer, the method further includes:

[0024] Obtain a cognitive state recognition model that includes a recurrent neural network layer, a feature mapping layer, and a fusion and classification layer, and obtain a training sample set that includes self-collected data and a DEAP physiological signal database;

[0025] A target sample is extracted sequentially from the training sample set, and the cognitive state recognition model is used to perform forward network computation on the target sample to obtain the cognitive prediction result;

[0026] Using the cross-entropy loss algorithm, the loss value between the true label corresponding to the target sample and the cognitive prediction result is calculated. Then, the loss value is minimized according to the gradient descent method to obtain the network model parameters when the loss value is minimized. Finally, the network model parameters are updated by performing a network inverse parameter update operation to obtain an updated cognitive state recognition model.

[0027] Determine the convergence of the loss value;

[0028] When the loss value fails to converge, return to the above steps of extracting a target sample from the training sample set in sequence, and use the new target sample to iteratively train the updated cognitive state recognition model.

[0029] When the loss value converges, the training process stops, and the trained recurrent neural network layer, feature mapping layer, and fusion and classification layer are obtained.

[0030] Optionally, the step of using a pre-trained recurrent neural network layer to perform feature extraction on the electroencephalogram to obtain a first feature sequence includes:

[0031] The long short-term memory network in the recurrent neural network layer is used to perform feature extraction on the electroencephalogram to obtain the feature extraction results;

[0032] The feature extraction results are filtered for important information using a gating state filtering algorithm to obtain the first feature sequence.

[0033] Optionally, the fusion and classification layer includes a pre-built joint representation framework, an intermodal attention encoder, a forward feedback attention encoder, and a two-layer fully connected layer classifier.

[0034] To address the above problems, the present invention also provides a cognitive state recognition device based on multimodal signals, the device comprising:

[0035] The multimodal signal image acquisition module is used to acquire the initial EEG of the target user and the initial peripheral image set of the preset peripheral physiological signals, and to perform noise reduction and cleaning operations on the initial EEG and the initial peripheral image set according to the preset image preprocessing strategy to obtain the EEG and peripheral image set.

[0036] The feature encoding module is used to perform feature extraction operations on the electroencephalogram using a pre-trained recurrent neural network layer to obtain a first feature sequence, and to perform feature extraction operations on the peripheral image set to obtain a second feature sequence set.

[0037] The feature mapping module is used to configure the self-attention weights of the first feature sequence based on each channel using the first feature mapping network in the pre-trained feature mapping layer to obtain a channel attention-enhanced feature sequence, and to perform a time-channel based two-dimensional feature mapping operation on the channel attention-enhanced feature sequence to obtain a first mapped feature sequence. It also uses the second feature mapping network in the feature mapping layer to configure the attention weights of the second feature sequence set based on multiple linear projections over time steps to obtain a time attention-enhanced feature sequence set, and to perform a time-step based one-dimensional feature mapping operation on the time attention-enhanced feature sequence set to obtain a second mapped feature sequence set.

[0038] The cognitive state recognition module is used to map the first mapping feature sequence and the second mapping feature sequence set to the target space using a pre-trained fusion and classification layer to obtain a multimodal fusion feature sequence set, and to perform fully connected classification judgment on the multimodal fusion feature sequence set to obtain the high and low order cognitive state recognition results of the target user.

[0039] To address the above problems, the present invention also provides an electronic device, the electronic device comprising:

[0040] At least one processor; and,

[0041] A memory communicatively connected to the at least one processor; wherein,

[0042] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the cognitive state recognition method based on multimodal signals described above.

[0043] To address the aforementioned problems, the present invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the cognitive state recognition method based on multimodal signals described above.

[0044] This invention first uses a first feature mapping network to configure multi-channel attention weights for EEG data, and then obtains a first mapping feature sequence based on time and channel through two-dimensional convolution operations. Next, a second feature mapping network is used to configure time-step attention for signals other than EEG, such as ECG, blood pressure, and respiratory signals, and then maps them through one-dimensional convolution operations to obtain a second mapping feature sequence set. The complementary nature of multimodal physiological signals reduces the negative impact of factors such as anomalies in single-modal data on the recognition results. Finally, feature fusion is performed to address the differences between different modal signals, and finally, fully connected classification is used to obtain the recognition results for high and low-order cognitive states, improving the recognition accuracy. Therefore, this invention provides a method, device, equipment, and storage medium for recognizing cognitive states based on multimodal signals. This method can represent the complex collaborative relationships between multiple physiological signals through an attention mechanism while reducing the heterogeneity gap between different modal data, thereby improving the accuracy of recognizing users' high and low-order cognitive states. Attached Figure Description

[0045] Figure 1 This is a flowchart illustrating a cognitive state recognition method based on multimodal signals provided in an embodiment of the present invention.

[0046] Figure 2 This is a detailed flowchart illustrating one step of a cognitive state recognition method based on multimodal signals provided in an embodiment of the present invention.

[0047] Figure 3 This is a detailed flowchart illustrating one step of a cognitive state recognition method based on multimodal signals provided in an embodiment of the present invention.

[0048] Figure 4 A functional block diagram of a cognitive state recognition device based on multimodal signals provided in an embodiment of the present invention;

[0049] Figure 5 This is a schematic diagram of the structure of an electronic device that implements the cognitive state recognition method based on multimodal signals, according to an embodiment of the present invention.

[0050] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0051] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0052] This application provides a cognitive state recognition method based on multimodal signals. In this application, the executing entity of the cognitive state recognition method based on multimodal signals includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the cognitive state recognition method based on multimodal signals can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0053] Reference Figure 1 The diagram shown is a flowchart illustrating a cognitive state recognition method based on multimodal signals according to an embodiment of the present invention. In this embodiment, the cognitive state recognition method based on multimodal signals includes:

[0054] S1. Obtain the initial EEG of the target user and the initial peripheral image set of the preset peripheral physiological signals, and perform noise reduction and cleaning operations on the initial EEG and the initial peripheral image set according to the preset image preprocessing strategy to obtain the EEG and peripheral image set.

[0055] In this embodiment of the invention, a pre-built cluster of devices, such as physiological sensor patches or radar devices, can be used to acquire initial electroencephalogram (EEG) images and initial peripheral images such as electrocardiograms, respiratory signals, and blood pressure of a target user when learning a certain aspect of learning content. The EEG can connect to different regions of the brain, thus corresponding to different channels.

[0056] The cognitive state recognition described in this embodiment of the invention is mainly used to detect the learning state of target users, and can be used in real-world teaching, distance learning, virtual games, metaverse teaching, and other fields.

[0057] Furthermore, in this embodiment of the invention, the image preprocessing strategy refers to transforming the initial image into a recognizable image, including operations such as noise reduction, grayscale conversion, and resolution changes, which can be adjusted according to the actual situation.

[0058] In this embodiment of the invention, the initial EEG and initial peripheral image set are denoised and cleaned using grayscale conversion and Gaussian filtering methods in the image preprocessing strategy to obtain the EEG and peripheral image set. Specifically, Gaussian filtering refers to achieving a smoothing effect by weighted averaging of each pixel and its surrounding pixels, which can reduce image noise; grayscale conversion refers to the process of converting a color image to a grayscale image, which can reduce the data volume and improve the processing efficiency of subsequent image data.

[0059] S2. Using a pre-trained recurrent neural network layer, perform feature extraction on the electroencephalogram to obtain a first feature sequence, and perform feature extraction on the peripheral image set to obtain a second feature sequence set.

[0060] This invention constructs a multi-modal physiological signal feature representation fusion model based on multi-level attention (PSFMMA, or cognitive state recognition model). The cognitive state recognition model includes a recurrent neural network layer, a feature mapping layer, and a fusion and classification layer. The recurrent neural network layer is used to extract features from the multimodal signals. The feature mapping layer is used to transform the features extracted from each modality into abstract features through attention weight configuration. The fusion and classification layer is used to fuse the abstract features into the same space and perform fully connected classification calculations to obtain the cognitive state recognition result.

[0061] In detail, in this embodiment of the invention, the step of using a pre-trained recurrent neural network layer to perform feature extraction on the electroencephalogram to obtain a first feature sequence includes:

[0062] The long short-term memory network in the recurrent neural network layer is used to perform feature extraction on the electroencephalogram to obtain the feature extraction results;

[0063] The feature extraction results are filtered for important information using a gating state filtering algorithm to obtain the first feature sequence.

[0064] The Long Short-Term Memory (LSTM) network is a commonly used Recurrent Neural Network (RNN) model. Compared to traditional RNNs, LSTM introduces a gating mechanism, which enables it to better handle long sequences of data and solve the vanishing / exploding gradient problem.

[0065] The above-mentioned gated state filtering algorithm, that is, the calculation process of the LSTM network filtering important information through gated states, is as follows:

[0066] N k =W k ·[S t-1 ,X t ]+b k

[0067] L t =sig(N1)*L t-1 +sig(N2)*tan(N3)

[0068] S t =sig(N4)*tan(L) t )

[0069] Where X t Let S be the input sequence at time t. t-1 S represents short-term memory at time t-1. t L represents short-term memory at time t. t-1 L represents long-term memory at time t-1. t W represents long-term memory at time t. k It is the weight matrix for various gated states, b k N represents the bias term for various gating states. k These are the main terms for various gating states. k can take values ​​of 1, 2, 3, and 4. sig() and tanh() are two activation functions. N1 and N4, after passing through their respective activation functions, form the forget gate and output gate respectively (where sig(N1) is the forget gate and sig(N4) is the output gate); N2 and N3, through their corresponding activation functions, together form the memory gate (where sig(N2)*tan(N3) is the memory gate). In the memory gate, tan(N3) is used to extract information, and sig(N2) controls how much information is memorized.

[0070] In this embodiment of the invention, the above-mentioned gating state filtering algorithm performs important information filtering operation on the feature extraction results to obtain the first feature sequence.

[0071] Similarly, based on the feature extraction process described above, the second feature sequence set is obtained.

[0072] Furthermore, in this embodiment of the invention, before performing feature extraction on the electroencephalogram using a pre-trained recurrent neural network layer, the method further includes:

[0073] Obtain a cognitive state recognition model that includes a recurrent neural network layer, a feature mapping layer, and a fusion and classification layer, and obtain a training sample set that includes self-collected data and a DEAP physiological signal database;

[0074] A target sample is extracted sequentially from the training sample set, and the cognitive state recognition model is used to perform forward network computation on the target sample to obtain the cognitive prediction result;

[0075] Using the cross-entropy loss algorithm, the loss value between the true label corresponding to the target sample and the cognitive prediction result is calculated. Then, the loss value is minimized according to the gradient descent method to obtain the network model parameters when the loss value is minimized. Finally, the network model parameters are updated by performing a network inverse parameter update operation to obtain an updated cognitive state recognition model.

[0076] Determine the convergence of the loss value;

[0077] When the loss value fails to converge, return to the above steps of extracting a target sample from the training sample set in sequence, and use the new target sample to iteratively train the updated cognitive state recognition model.

[0078] When the loss value converges, the training process stops, and the trained recurrent neural network layer, feature mapping layer, and fusion and classification layer are obtained.

[0079] The DEAP physiological signal database is a publicly available database of physiological signals used for emotion analysis. It is a project funded by the European Science Foundation, aiming to provide a standard dataset for studying the relationship between emotions and physiological signals. In this embodiment of the invention, a training sample set is constructed by combining self-collected data with the DEAP physiological signal database.

[0080] In this embodiment of the invention, a cognitive prediction result is first obtained through forward model computation. Then, the cross-entropy loss algorithm is used to calculate the loss value between the true label corresponding to the target sample and the cognitive prediction result. Next, the gradient descent algorithm is used to perform a reverse model update operation using the network model parameters when the loss value is minimized, resulting in an updated cognitive state recognition model. The cross-entropy loss algorithm is a commonly used loss function used to measure the difference between the prediction result of a classification model and the true label; it is widely used in classification tasks in machine learning and deep learning. The gradient descent algorithm is a commonly used optimization algorithm used to find the minimum or maximum value of a function. It updates parameters iteratively along the negative gradient direction of the function to gradually approach the optimal solution.

[0081] In this embodiment of the invention, the training progress of the model is controlled by the convergence of the loss value. When the model loss value has not converged, it indicates that the model training has made significant progress and can continue to be trained. When the loss value converges, it indicates that the model training effect is not significant. In order to avoid overfitting, the training process is stopped, and the trained recurrent neural network layer, feature mapping layer, and fusion and classification layer are obtained.

[0082] S3. Using the first feature mapping network in the pre-trained feature mapping layer, configure the self-attention weights of each channel on the first feature sequence to obtain the channel attention-enhanced feature sequence, and perform a time-channel based two-dimensional feature mapping operation on the channel attention-enhanced feature sequence to obtain the first mapped feature sequence.

[0083] In this embodiment of the invention, since EEG and peripheral physiological signals are associated with different nervous systems, the number of channels they contain and the data formats they represent differ significantly. Therefore, this invention designs two feature mapping architectures to extract and map features from EEG signals and peripheral physiological signals using different hierarchical structures, namely, a first feature mapping network and a second feature mapping network.

[0084] The first feature mapping network is used to process the first feature sequence corresponding to the electroencephalogram.

[0085] The second feature mapping network is used to process the second feature sequence set corresponding to peripheral physiological signals.

[0086] For details, please refer to Figure 2 As shown in the embodiment of the present invention, the operation step S3 includes:

[0087] S31. The first feature sequence is transformed nonlinearly based on the self-attention of each channel using the first multilayer perceptual layer in the first feature mapping network to obtain the weight configuration feature sequence, and the first feature sequence and the weight configuration feature sequence are connected by residual skip connection to obtain the channel attention enhanced feature sequence.

[0088] S32. Using the two-dimensional convolutional neural network in the first feature mapping network, perform an adaptive number of two-dimensional double-layer convolutional pooling operations on the channel attention enhancement feature sequence to obtain a dimension-reduced enhancement feature sequence.

[0089] S33. Using the second multilayer perception layer in the first feature mapping network, the dimensionality reduction and enhancement feature sequence is mapped into a time attention feature sequence and a channel attention feature sequence according to the preset time-channel dimension, and the time attention feature sequence and the channel attention feature sequence are subjected to feature fusion operation to obtain the fused feature result.

[0090] S34. The fully convolutional network in the first feature mapping network is used to perform a fully convolutional operation on the fused feature result to obtain the first mapping feature sequence.

[0091] Specifically, in this embodiment of the invention, the step of performing a nonlinear transformation based on the self-attention of each channel on the first feature sequence in S31 to obtain a weighted configuration feature sequence includes:

[0092] The first feature sequence is subjected to average pooling to obtain a first average pooled feature sequence, and the first feature sequence is subjected to max pooling to obtain a first max pooled feature sequence.

[0093] Using the first multilayer perceptron in the first feature mapping network, attention weights are calculated on the first average pooling feature sequence and the first max pooling feature sequence respectively to obtain the average attention sequence and the max attention sequence respectively.

[0094] The average attention sequence and the maximum attention sequence are summed and averaged to obtain the attention fusion sequence. The attention fusion sequence is then output using the sigmoid activation function to obtain the weight configuration feature sequence.

[0095] The first and second multilayer perceptrons are a type of multilayer perceptron (MLP), consisting of multiple neuron layers, each fully connected to the preceding and following neuron layers. Each neuron layer comprises multiple neurons, each receiving input from the neurons in the preceding layer, performing a nonlinear transformation through an activation function, and then passing the output to the neurons in the next layer.

[0096] In this embodiment of the invention, channel attention learns attention weights for each channel, thereby emphasizing a specific weight configuration feature sequence. Then, based on the above process, an identity mapping path is added to form a residual skip connection structure. This structure helps solve the network degradation problem, obtains channel attention-enhanced feature sequences, and facilitates network optimization.

[0097] Furthermore, in process S32 of this invention, the first feature mapping network selects a two-dimensional CNN as its core component. The two-dimensional CNN can filter noise and fully utilize the temporal (sequence length of the first feature sequence) and spatial (number of channels) information of the signal during the learning period to study complex patterns. In addition, the two-dimensional CNN can establish connections between different EEG signal channels and extract local temporal features within a time period; this characteristic is well-suited to the data format of EEG signals.

[0098] Specifically, for the input, the first feature mapping network first employs one-dimensional adaptive max pooling and one-dimensional adaptive average pooling to more comprehensively aggregate the feature information of each channel. The number of convolutional pooling operations can be adaptively configured so that the pooling result remains at a computationally convenient order of magnitude. The pooled feature sequence is then input into a multilayer perceptron (i.e., a multilayer perceptron layer), and the number of neurons in its hidden layers is reduced to simplify the network parameters.

[0099] Then, the multilayer perceptron outputs two different attention sequences (feature sequence length and number of channels), which are fused and activated by the sigmoid function to calculate the attention for each channel. Finally, the channel attention vectors are compared with the first input feature sequence to obtain the optimized weight configuration feature sequence.

[0100] In detail, the time attention feature sequence and the channel attention feature sequence are subjected to feature fusion in the above S33 process, and the calculation process is defined as follows:

[0101] H c (X)=σ(W1·W0·F avg (X)+W1·W0·F(X) max (X))

[0102]

[0103] Where X∈R n×c H is the first feature sequence of the input, where n is the length of the first feature sequence and c is the number of channels in the first feature sequence. c (X) represents the attention weight vector for each channel, W1 and W0 are the weight matrices in the multilayer perceptron, and F avg For adaptive average pooling, Fmax For adaptive max pooling, σ is the activation function. X′ represents the fused feature result after passing through the first feature mapping network.

[0104] Furthermore, in S3 of this invention, all convolutional layers employ the ReLU activation function. Then, after passing through the fully convolutional network structure designed at the end of the first feature mapping network, the output is the first mapping feature sequence representing the EEG modality.

[0105] S4. Using the second feature mapping network in the feature mapping layer, perform multiple linear projection attention weight configurations on the second feature sequence set based on time steps to obtain a time attention-enhanced feature sequence set, and perform a one-dimensional feature mapping operation on the time attention-enhanced feature sequence set based on time steps to obtain a second mapped feature sequence set.

[0106] The second feature mapping network is used to process the second feature sequence set corresponding to peripheral physiological signals.

[0107] For details, please refer to Figure 3 As shown in this embodiment of the invention, the step of using the second feature mapping network in the feature mapping layer to perform multiple linear projection attention weight configuration on the second feature sequence set based on time steps to obtain a time attention-enhanced feature sequence set includes:

[0108] S41. The second feature sequence set is subjected to two linear transformations using the multilayer perceptron in the second feature mapping network to obtain the transformed feature result;

[0109] S42. Normalize the transformed feature results using the softmax function to obtain the attention distribution;

[0110] S43. The second feature sequence set is weighted according to the attention distribution to obtain the time attention-enhanced feature sequence set.

[0111] The second feature mapping network is mainly composed of a mixture of multiple 1D CNN layers and multilayer perceptrons. The 1D CNN, with fewer parameters and a lighter weight, can train and infer faster, while its sliding kernel calculation can smooth the data and has a good effect on extracting local features of peripheral physiological signals.

[0112] In this embodiment of the invention, the input feature sequence undergoes two linear transformations, and then the result is normalized using the softmax function to obtain the attention distribution. Since the feature dimensions of physiological signals such as Electrodermal Activity (EDA) and Photoplethysmography (PPG) are singular at each time step, a multilayer perceptron is introduced to provide more learnable parameters to improve generalization ability, and a Dropout layer is added to each layer to prevent overfitting.

[0113] The detailed calculation process for the above-mentioned time-step-based multi-linear projection attention weight configuration is as follows:

[0114] B = G(W2G(W1X))

[0115]

[0116] Where X = [x1,…,x] n ]∈R n×1 n is the time step, X is the time step representation of the second feature sequence set, W1 and W2 are the weight matrices, and G is the activation function. B = [b1,…,b n ] is a sequence consistent with the shape dimension of X, representing the score value at each time step in the attention mechanism. μ i It is the sequence element value at the current time step after the attention mechanism.

[0117] After obtaining the time attention enhancement feature sequence set according to the above operations, the present invention performs a one-dimensional feature mapping operation based on time steps on the time attention enhancement feature sequence set to obtain a second mapped feature sequence set.

[0118] S5. Using the pre-trained fusion and classification layer, the first mapping feature sequence and the second mapping feature sequence set are mapped to the target space to obtain a multimodal fusion feature sequence set. The multimodal fusion feature sequence set is then subjected to fully connected classification judgment to obtain the high and low order cognitive state recognition results of the target user.

[0119] In this embodiment of the invention, the fusion and classification layer aims to learn the relationship between different physiological signal modalities to achieve modality fusion and complete the recognition of the learner's cognitive state. The modality representation vectors output by the aforementioned feature mapping layer reside in different subspaces. To reduce the heterogeneity gap between different modality vectors, this layer is designed to integrate the various subspaces into a single representation space. This paper adopts a model-level fusion strategy based on a joint representation framework, mapping the representations of each modality to a shared subspace.

[0120] In detail, in this embodiment of the invention, the fusion and classification layer includes a pre-built joint representation framework, an intermodal attention encoder, a forward feedback attention encoder, and a two-layer fully connected layer classifier.

[0121] This invention designs a two-layer transformer encoder, including intermodal attention calculation and a feedforward neural network. The feedforward neural network increases the model's fitting ability and extracts richer information between modalities. After the encoder, an averaging operation is used to synthesize all fused information to form a multimodal representation. At the end of the fusion and classification layers, a classifier is constructed using two fully connected layers to achieve cognitive state recognition. One of the fully connected layers uses the Tanh function activation to increase nonlinearity.

[0122] In addition, in another embodiment of the present invention, in order to reduce the computational complexity of attention without significantly degrading model performance, a non-softmax function method is used to remove nonlinear operations in the attention calculation process, thereby making the time and space complexity linearly related to the number of modes, while improving the efficiency of data utilization.

[0123] Therefore, in this embodiment of the invention, based on the structure in the above-mentioned fusion and classification layer, the multimodal fusion feature sequence set can be fully connected for classification judgment to obtain the high and low order cognitive state recognition results of the target user.

[0124] Specifically, in this embodiment, the recognition results of the target user's high- and low-order cognitive states are divided into high-order cognitive states and low-order cognitive states.

[0125] Specifically, in remote teaching scenarios, students wear various wearable devices to collect physiological signal data. The collected physiological data is then fed into the cognitive state recognition model to provide real-time feedback on the student's cognitive state. This data indicates whether the student's learning cognitive state is at a low-order or high-order level.

[0126] To facilitate teachers' management and monitoring of students' cognitive states during the learning process, the feedback data on students' cognitive states will be visualized for teachers and stored in log form. Teachers can use the visualized data to understand whether students are in higher-order cognitive states during learning, allowing for real-time intervention to promote the development of learning cognitive abilities. They can also analyze the cognitive log data from the teaching process to intervene with students after class or develop more appropriate teaching plans.

[0127] This invention first uses a first feature mapping network to configure multi-channel attention weights for EEG data, and then obtains a first mapping feature sequence based on time and channel through two-dimensional convolution operations. Next, a second feature mapping network is used to configure time-step attention for signals other than EEG, such as ECG, blood pressure, and respiratory signals, and then maps them through one-dimensional convolution operations to obtain a second mapping feature sequence set. The complementary nature of multimodal physiological signals reduces the negative impact of factors such as anomalies in single-modal data on the recognition results. Finally, feature fusion is performed to address the differences between different modal signals, and finally, fully connected classification is used to obtain the recognition results for high and low-order cognitive states, improving the recognition accuracy. Therefore, this invention provides a method, device, equipment, and storage medium for recognizing cognitive states based on multimodal signals. This method can represent the complex collaborative relationships between multiple physiological signals through an attention mechanism while reducing the heterogeneity gap between different modal data, thereby improving the accuracy of recognizing users' high and low-order cognitive states.

[0128] like Figure 4 The diagram shown is a functional block diagram of a cognitive state recognition device based on multimodal signals provided in an embodiment of the present invention.

[0129] The cognitive state recognition device 100 based on multimodal signals described in this invention can be installed in an electronic device. Depending on the functions implemented, the cognitive state recognition device 100 may include a multimodal signal image acquisition module 101, a feature encoding module 102, a feature mapping module 103, and a cognitive state recognition module 104. The module described in this invention can also be called a unit, referring to a series of computer program segments that can be executed by the processor of an electronic device and perform a fixed function, stored in the memory of the electronic device.

[0130] In this embodiment, the functions of each module / unit are as follows:

[0131] The multimodal signal image acquisition module 101 is used to acquire the initial EEG of the target user and the initial peripheral image set of the preset peripheral physiological signals, and to perform noise reduction and cleaning operations on the initial EEG and the initial peripheral image set according to the preset image preprocessing strategy to obtain the EEG and the peripheral image set.

[0132] The feature encoding module 102 is used to perform feature extraction operations on the electroencephalogram using a pre-trained recurrent neural network layer to obtain a first feature sequence, and to perform feature extraction operations on the peripheral image set to obtain a second feature sequence set.

[0133] The feature mapping module 103 is used to configure the self-attention weights of the first feature sequence based on each channel using the first feature mapping network in the pre-trained feature mapping layer to obtain a channel attention-enhanced feature sequence, and to perform a time-channel based two-dimensional feature mapping operation on the channel attention-enhanced feature sequence to obtain a first mapped feature sequence. It also uses the second feature mapping network in the feature mapping layer to configure the second feature sequence set based on multiple linear projection attention weights based on time steps to obtain a time attention-enhanced feature sequence set, and to perform a time-step based one-dimensional feature mapping operation on the time attention-enhanced feature sequence set to obtain a second mapped feature sequence set.

[0134] The cognitive state recognition module 104 is used to map the first mapping feature sequence and the second mapping feature sequence set to the target space using a pre-trained fusion and classification layer to obtain a multimodal fusion feature sequence set, and to perform fully connected classification judgment on the multimodal fusion feature sequence set to obtain the high and low order cognitive state recognition results of the target user.

[0135] In detail, each module in the cognitive state recognition device 100 based on multimodal signals described in this application embodiment adopts the same characteristics as described above during use. Figures 1 to 3 The method uses the same techniques as the cognitive state recognition method based on multimodal signals described above and can produce the same technical effects, so it will not be repeated here.

[0136] like Figure 5 The diagram shown is a structural schematic of an electronic device 1 that implements a cognitive state recognition method based on multimodal signals, according to an embodiment of the present invention.

[0137] The electronic device 1 may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13. It may also include a computer program stored in the memory 11 and capable of running on the processor 10, such as a cognitive state recognition program based on multimodal signals.

[0138] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device 1, connecting various components of the electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., executing a cognitive state recognition program based on multimodal signals) and calls data stored in the memory 11 to perform various functions of the electronic device and process data.

[0139] The memory 11 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of an electronic device, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. Furthermore, the memory 11 can include both internal and external storage units of the electronic device. The memory 11 can be used not only to store application software and various types of data installed on the electronic device, such as code for a cognitive state recognition program based on multimodal signals, but also to temporarily store data that has been output or will be output.

[0140] The communication bus 12 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0141] The communication interface 13 is used for communication between the electronic device 1 and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, Bluetooth interface, etc.), typically used to establish communication connections between the electronic device and other electronic devices. The user interface may be a display, an input unit (such as a keyboard), or, optionally, a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device and to display a visual user interface.

[0142] Figure 5 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 5 The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0143] For example, although not shown, the electronic device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0144] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0145] The cognitive state recognition program based on multimodal signals stored in the memory 11 of the electronic device 1 is a combination of multiple instructions. When run in the processor 10, it can achieve the following:

[0146] The initial EEG and the initial peripheral image set of the preset peripheral physiological signals of the target user are obtained, and the initial EEG and the initial peripheral image set are subjected to noise reduction and cleaning operations according to the preset image preprocessing strategy to obtain the EEG and peripheral image set.

[0147] Using a pre-trained recurrent neural network layer, feature extraction is performed on the electroencephalogram to obtain a first feature sequence, and feature extraction is performed on the peripheral image set to obtain a second feature sequence set;

[0148] Using the first feature mapping network in the pre-trained feature mapping layer, the first feature sequence is configured with self-attention weights based on each channel to obtain a channel attention-enhanced feature sequence, and the channel attention-enhanced feature sequence is subjected to a time-channel based two-dimensional feature mapping operation to obtain a first mapped feature sequence.

[0149] Using the second feature mapping network in the feature mapping layer, the second feature sequence set is configured with multiple linear projection attention weights based on time steps to obtain a time attention-enhanced feature sequence set, and a one-dimensional feature mapping operation based on time steps is performed on the time attention-enhanced feature sequence set to obtain a second mapped feature sequence set.

[0150] Using a pre-trained fusion and classification layer, the first mapping feature sequence and the second mapping feature sequence set are mapped to the target space to obtain a multimodal fusion feature sequence set. The multimodal fusion feature sequence set is then subjected to fully connected classification to obtain the high and low order cognitive state recognition results of the target user.

[0151] Specifically, the specific implementation method of the processor 10 for the above instructions can be referred to the description of the relevant steps in the corresponding embodiment of the accompanying drawings, and will not be repeated here.

[0152] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0153] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor of an electronic device, can perform the following:

[0154] The initial EEG and the initial peripheral image set of the preset peripheral physiological signals of the target user are obtained, and the initial EEG and the initial peripheral image set are subjected to noise reduction and cleaning operations according to the preset image preprocessing strategy to obtain the EEG and peripheral image set.

[0155] Using a pre-trained recurrent neural network layer, feature extraction is performed on the electroencephalogram to obtain a first feature sequence, and feature extraction is performed on the peripheral image set to obtain a second feature sequence set;

[0156] Using the first feature mapping network in the pre-trained feature mapping layer, the first feature sequence is configured with self-attention weights based on each channel to obtain a channel attention-enhanced feature sequence, and the channel attention-enhanced feature sequence is subjected to a time-channel based two-dimensional feature mapping operation to obtain a first mapped feature sequence.

[0157] Using the second feature mapping network in the feature mapping layer, the second feature sequence set is configured with multiple linear projection attention weights based on time steps to obtain a time attention-enhanced feature sequence set, and a one-dimensional feature mapping operation based on time steps is performed on the time attention-enhanced feature sequence set to obtain a second mapped feature sequence set.

[0158] Using a pre-trained fusion and classification layer, the first mapping feature sequence and the second mapping feature sequence set are mapped to the target space to obtain a multimodal fusion feature sequence set. The multimodal fusion feature sequence set is then subjected to fully connected classification to obtain the high and low order cognitive state recognition results of the target user.

[0159] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0160] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0161] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0162] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0163] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0164] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0165] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0166] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0167] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A cognitive state recognition method based on multimodal signals, characterized in that, The method includes: The initial EEG and the initial peripheral image set of the preset peripheral physiological signals of the target user are obtained, and the initial EEG and the initial peripheral image set are subjected to noise reduction and cleaning operations according to the preset image preprocessing strategy to obtain the EEG and peripheral image set. Using a pre-trained recurrent neural network layer, feature extraction is performed on the electroencephalogram to obtain a first feature sequence, and feature extraction is performed on the peripheral image set to obtain a second feature sequence set; Using the first feature mapping network in the pre-trained feature mapping layer, the first feature sequence is configured with self-attention weights based on each channel to obtain a channel attention-enhanced feature sequence. Then, a time-channel based two-dimensional feature mapping operation is performed on the channel attention-enhanced feature sequence to obtain a first mapped feature sequence. Using the second feature mapping network in the feature mapping layer, the second feature sequence set is configured with multiple linear projection attention weights based on time steps to obtain a time attention-enhanced feature sequence set, and a one-dimensional feature mapping operation based on time steps is performed on the time attention-enhanced feature sequence set to obtain a second mapped feature sequence set. Using a pre-trained fusion and classification layer, the first mapping feature sequence and the second mapping feature sequence set are mapped to the target space to obtain a multimodal fusion feature sequence set. The multimodal fusion feature sequence set is then subjected to fully connected classification to obtain the high and low order cognitive state recognition results of the target user.

2. The cognitive state recognition method based on multimodal signals as described in claim 1, characterized in that, The first feature mapping network in the pre-trained feature mapping layer is used to configure the self-attention weights of the first feature sequence based on each channel to obtain a channel attention-enhanced feature sequence. Then, a time-channel-based two-dimensional feature mapping operation is performed on the channel attention-enhanced feature sequence to obtain a first mapped feature sequence, including: The first feature sequence is transformed by the first multilayer perceptual layer in the first feature mapping network based on the self-attention of each channel to obtain the weight configuration feature sequence. The first feature sequence and the weight configuration feature sequence are then connected by a residual skip connection to obtain the channel attention enhanced feature sequence. Using the two-dimensional convolutional neural network in the first feature mapping network, an adaptive number of two-dimensional double-layer convolutional pooling operations are performed on the channel attention enhancement feature sequence to obtain a dimension-reduced enhancement feature sequence; Using the second multilayer perception layer in the first feature mapping network, the dimensionality reduction and enhancement feature sequence is mapped into a time attention feature sequence and a channel attention feature sequence according to the preset time-channel dimension, and the time attention feature sequence and the channel attention feature sequence are fused to obtain the fused feature result. The fused feature result is subjected to a fully convolutional operation using the fully convolutional network in the first feature mapping network to obtain the first mapping feature sequence.

3. The cognitive state recognition method based on multimodal signals as described in claim 2, characterized in that, The step of performing a nonlinear transformation based on self-attention for each channel on the first feature sequence using the first multilayer perceptron in the first feature mapping network to obtain a weighted feature sequence includes: The first feature sequence is subjected to average pooling to obtain a first average pooled feature sequence, and the first feature sequence is subjected to max pooling to obtain a first max pooled feature sequence. Using the first multilayer perceptron in the first feature mapping network, attention weights are calculated on the first average pooling feature sequence and the first max pooling feature sequence respectively to obtain the average attention sequence and the max attention sequence respectively. The average attention sequence and the maximum attention sequence are summed and averaged to obtain the attention fusion sequence. The attention fusion sequence is then output using the sigmoid activation function to obtain the weight configuration feature sequence.

4. The cognitive state recognition method based on multimodal signals as described in claim 1, characterized in that, The step of using the second feature mapping network in the feature mapping layer to configure the second feature sequence set with multiple linear projection attention weights based on time steps to obtain a time-attention-enhanced feature sequence set includes: The second feature sequence set is subjected to two linear transformations using the multilayer perceptron in the second feature mapping network to obtain the transformed feature result. The attention distribution is obtained by normalizing the transformed feature results using the softmax function. The second feature sequence set is weighted according to the attention distribution to obtain the time attention-enhanced feature sequence set.

5. The cognitive state recognition method based on multimodal signals as described in claim 1, characterized in that, Before performing feature extraction on the electroencephalogram using a pre-trained recurrent neural network layer, the method further includes: Obtain a cognitive state recognition model that includes a recurrent neural network layer, a feature mapping layer, and a fusion and classification layer, and obtain a training sample set that includes self-collected data and a DEAP physiological signal database; A target sample is extracted sequentially from the training sample set, and the cognitive state recognition model is used to perform forward network computation on the target sample to obtain the cognitive prediction result; Using the cross-entropy loss algorithm, the loss value between the true label corresponding to the target sample and the cognitive prediction result is calculated. Then, the loss value is minimized according to the gradient descent method to obtain the network model parameters when the loss value is minimized. Finally, the network model parameters are updated by performing a network inverse parameter update operation to obtain an updated cognitive state recognition model. Determine the convergence of the loss value; When the loss value fails to converge, return to the above steps of extracting a target sample from the training sample set in sequence, and use the new target sample to iteratively train the updated cognitive state recognition model. When the loss value converges, the training process stops, and the trained recurrent neural network layer, feature mapping layer, and fusion and classification layer are obtained.

6. The cognitive state recognition method based on multimodal signals as described in claim 1, characterized in that, The process of using a pre-trained recurrent neural network layer to perform feature extraction on the electroencephalogram (EEG) to obtain a first feature sequence includes: The long short-term memory network in the recurrent neural network layer is used to perform feature extraction on the electroencephalogram to obtain the feature extraction results; The feature extraction results are filtered for important information using a gating state filtering algorithm to obtain the first feature sequence.

7. The cognitive state recognition method based on multimodal signals as described in claim 1, characterized in that, The fusion and classification layer includes a pre-built joint representation framework, an intermodal attention encoder, a forward feedback attention encoder, and a two-layer fully connected classifier.

8. A cognitive state recognition device based on multimodal signals, characterized in that, The device includes: The multimodal signal image acquisition module is used to acquire the initial EEG of the target user and the initial peripheral image set of the preset peripheral physiological signals, and to perform noise reduction and cleaning operations on the initial EEG and the initial peripheral image set according to the preset image preprocessing strategy to obtain the EEG and peripheral image set. The feature encoding module is used to perform feature extraction operations on the electroencephalogram using a pre-trained recurrent neural network layer to obtain a first feature sequence, and to perform feature extraction operations on the peripheral image set to obtain a second feature sequence set. The feature mapping module is used to configure self-attention weights for each channel on the first feature sequence using the first feature mapping network in the pre-trained feature mapping layer to obtain a channel attention-enhanced feature sequence, and to perform a time-channel based two-dimensional feature mapping operation on the channel attention-enhanced feature sequence to obtain a first mapped feature sequence. It also uses the second feature mapping network in the feature mapping layer to configure multiple linear projection attention weights for the second feature sequence set based on time steps to obtain a time attention-enhanced feature sequence set, and to perform a time-step based one-dimensional feature mapping operation on the time attention-enhanced feature sequence set to obtain a second mapped feature sequence set. The cognitive state recognition module is used to map the first mapping feature sequence and the second mapping feature sequence set to the target space using a pre-trained fusion and classification layer to obtain a multimodal fusion feature sequence set, and to perform fully connected classification judgment on the multimodal fusion feature sequence set to obtain the high and low order cognitive state recognition results of the target user.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the cognitive state recognition method based on multimodal signals as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the cognitive state recognition method based on multimodal signals as described in any one of claims 1 to 7.

Images