An online learning state detection method based on multi-modal data fusion

Abstract

The application discloses an online learning state detection method based on multi-modal data fusion, which comprises the following steps: obtaining a face image and an electroencephalogram signal of a to-be-detected object; inputting the face image information and the electroencephalogram signal into a trained multi-modal fusion model to extract the current gaze direction and electroencephalogram emotional state of the to-be-detected object; the multi-modal fusion model comprises a first network model and a second network model, the first network model is obtained by training a plurality of face image samples with gaze direction labels, and the second network model is obtained by training a plurality of electroencephalogram wave samples with electroencephalogram emotional labels; and generating learning state data of the to-be-detected object at present based on the gaze direction and the electroencephalogram emotional state. The application realizes the complementation of the gaze information and the electroencephalogram signal, realizes real-time detection of the online learning state of a learner, provides objective support data for the next learning guidance, helps to reasonably plan learning content, improves learning efficiency, and realizes a virtuous circle.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of pattern recognition and signal processing, and more particularly to an online learning state detection method based on multi-modal data fusion. BACKGROUND

[0002] With the rapid development of education informatization, artificial intelligence technology has been increasingly applied in the education and teaching process. In recent years, the outbreak of the new coronavirus pneumonia has broken the traditional offline teaching mode, and many students use online learning to learn. However, the supervision scene is separated from the offline teaching, so that students cannot control their real-time state, cannot reasonably plan the learning content, and the learning efficiency is low, which becomes a problem to be solved. Therefore, it is of great significance to use a learning state detection method to assist learners in online learning, and to make students make targeted next learning planning according to the detection results, so as to improve the learning efficiency of online learning of learners.

[0003] The special mental and psychological state affecting the learning efficiency--mental fatigue, is a human state mainly manifested as subjective tiredness, cognitive function decline, and sleep tendency. Mental fatigue is a special fatigue mental state, mainly manifested as fatigue, reduced alertness, lack of motivation, decreased cognitive activity ability, and reduced learning and working efficiency.

[0004] Electroencephalogram (EEG) is a spontaneous neural activity, which is formed by the summation calculation of a large number of neuron synchronous post-synaptic potentials when the brain is active, and is generated in the central nervous system. The brain wave, also known as electroencephalogram, records the change of the electric wave when the brain is active, and is the overall reflection of the electrical physiological activity of the brain nerve cells on the cerebral cortex or scalp surface. It is an electrical signal generated by the brain and transmitted through the cortex, which contains rich physiological and psychological information. Studies have shown that if the physiological or psychological state of the human body changes, the electroencephalogram will also fluctuate. Therefore, it is theoretically feasible to use electroencephalogram data for state detection; and the electroencephalogram is implicit and has no deceptive appearance.

[0005] Gaze tracking technology helps to determine the gaze position of a person on a digital screen, feedback the blink frequency, and the pupil's response to different stimuli. Therefore, it helps to determine the gaze direction of a person, whether he is sleepy, and judge his focus, etc. The traditional learning state detection mainly relies on this gaze tracking technology, but its relying modalities are few, not objective enough, and the gaze external behavior has deceptive nature. Relying only on gaze information is not enough to accurately determine the learning state, resulting in low accuracy of the current learning state monitoring results. SUMMARY

[0006] In view of the improvement needs of the prior art, the present application provides an online learning state detection method based on multi-modal data fusion, which uses facial images and electroencephalogram signals for learning state detection at the same time, can realize information complementation of different modalities, and aims to improve the accuracy of the obtained learning state of a learner, thereby providing support basis for the next learning planning and improving learning efficiency.

[0007] To achieve the above-mentioned purpose, according to one aspect of the present application, an online learning state detection method based on multi-modal data fusion is provided, which comprises:

[0008] S1 obtaining facial images and electroencephalogram signals of a to-be-detected object;

[0009] S2 inputting the facial images and electroencephalogram signals into a trained multi-modal fusion model to extract the current gaze direction and electroencephalogram emotional state of the to-be-detected object;

[0010] The multi-modal fusion model comprises a first network model and a second network model; the first network model is trained by using a plurality of facial image samples with gaze direction labels, and the second network model is trained by using a plurality of electroencephalogram samples with electroencephalogram emotional labels.

[0011] The multi-modal fusion model fuses and classifies the gaze features output by the first network model and the electroencephalogram emotional features output by the second network model to obtain the current gaze direction and electroencephalogram emotional state of the to-be-detected object.

[0012] S3 generating learning state data of the to-be-detected object at present based on the gaze direction and electroencephalogram emotional state.

[0013] Further, in the above-mentioned online learning state detection method, an RGB-D camera is used to obtain the facial images of the to-be-detected object, and a Bluetooth electroencephalogram device is used to obtain the electroencephalogram signals of the to-be-detected object.

[0014] Further, in the above-mentioned online learning state detection method, before the facial image information and electroencephalogram signals are input into the trained multi-modal fusion model, the method further comprises:

[0015] The facial images are preprocessed by frame division and cropping operation;

[0016] The electroencephalogram signals are down-sampled and band-pass filtered by using an EEGLAB toolbox.

[0017] Further, in the above-mentioned online learning state detection method, the first network model comprises a first feature extraction layer and a second feature extraction layer sharing weights, and a feature reconstruction module and an encoding perception module.

[0018] The first feature extraction layer and the second feature extraction layer are configured to extract local features from the face image and generate a feature matrix;

[0019] The feature reconstruction module is configured to recover the image according to the feature matrix and reconstruct the original image from the feature space; and the feature reconstruction module is combined with the second feature extraction layer to form a contrastive reconstruction network.

[0020] The encoding perception module is configured to generate a gaze feature according to the feature matrix, and the encoding perception module includes a Transformer model and a double-layer MLP perception.

[0021] Further, in the online learning state detection method, the training process of the first network model includes:

[0022] Obtaining a face image sample with a gaze direction label;

[0023] Inputting the face image sample into the first network model to be trained to output a predicted gaze feature through the model;

[0024] Calculating a first loss function according to the gaze feature and the gaze direction label, and adjusting the model parameters of the first network model to be trained in a reverse direction until the first loss function is minimized, thereby obtaining a trained first network model.

[0025] Further, in the online learning state detection method, the first loss function is defined as the sum of loss functions of the second feature extraction layer, the feature reconstruction module and the encoding perception module.

[0026] The loss function of the feature reconstruction module is denoted as L re , which is defined as:

[0027]

[0028] wherein p is an input face image, is a reconstructed face image;

[0029] The second feature extraction layer and the feature reconstruction module are in contrast, and the contrastive loss is denoted as L ag , which is defined as:

[0030]

[0031] The encoding perception module uses an L1 loss as a gaze estimation loss function, denoted as L e , which is defined as:

[0032]

[0033] wherein e∈R 3 is a real gaze direction, to estimate the gaze direction;

[0034] The first loss function combines the gaze estimation loss function and the adversarial loss function, which is defined as:

[0035]

[0036] where T is an indicator function, M is an attention map, * represents element-wise multiplication, E represents mathematical expectation, β1 and β2 are hyperparameters; k is a self-defined threshold.

[0037] Further, in the above online learning state detection method, the training process of the second network model comprises:

[0038] obtaining electroencephalogram samples with state labels, the state labels being divided into focused state labels, wandering state labels and fatigue state labels;

[0039] inputting the electroencephalogram samples with state labels into the second network model to output predicted electroencephalogram emotional states through the model;

[0040] calculating a second loss function according to the electroencephalogram emotional states and the state labels, and adjusting the model parameters of the second network model to be trained in reverse until the second loss function is minimized, thereby obtaining the trained second network model.

[0041] Further, in the above online learning state detection method, the second loss function is represented as L CAP , which is defined as:

[0042]

[0043] where t i is a true value, is a predicted value obtained through the SCPNet network model.

[0044] Further, in the above online learning state detection method, the multi-modal fusion model further comprises an aggregation network and a classification network;

[0045] The aggregation network is used to aggregate the gaze features output by the first network model and the electroencephalogram emotional features output by the second network model to obtain aggregated features;

[0046] The classification network generates the current gaze direction and electroencephalogram emotional state of the to-be-detected object according to the aggregated features.

[0047] Further, in the above online learning state detection method, the current learning state data of the to-be-detected object is generated based on the gaze direction and the electroencephalogram emotional state, comprising:

[0048] input the gaze direction and the electroencephalogram emotion data into a learning state evaluator, and obtain the current explicit learning state of the to-be-detected object according to the gaze direction and a preset learning space region;

[0049] combine the explicit learning state and the electroencephalogram emotion state, and obtain the current learning state data of the to-be-detected object.

[0050] According to a second aspect of the present application, a computer device is also provided, which comprises at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit executes the steps of the online learning state detection method according to any one of the above aspects.

[0051] Overall, compared with the prior art, the above technical solutions conceived by the present application can achieve the following beneficial effects:

[0052] (1) The online learning state detection method based on multi-modal data fusion provided by the present application uses facial images and electroencephalogram signals simultaneously for learning state detection, adopts a novel multi-modal fusion model to accurately extract gaze features and electroencephalogram features, and fuses the gaze features and the electroencephalogram features, thereby overcoming the limitation of single detection dimension in the traditional detection method, improving the accuracy of detection, and having important significance for improving learning efficiency and assisting teaching.

[0053] (2) The present application provides timely feedback to the current learner according to the obtained data, so that the learner can know his / her current state in time and make adjustments; it is helpful for reasonably planning learning content, combining work with rest to improve learning efficiency, and realizing a virtuous cycle of learning and rest. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0055] Figure 1 A flowchart of the online learning state detection method based on multi-modal data fusion provided by the present embodiment is shown in the figure;

[0056] Figure 2 A data acquisition diagram in an online learning environment provided by the present embodiment is shown in the figure;

[0057] Figure 3 A structure diagram of the multi-modal fusion network provided by the present embodiment is shown in the figure;

[0058] Figure 4 A learning state judgment mechanism provided by the embodiment of the present application. DETAILED DESCRIPTION

[0059] The technical solutions in the embodiments of the present application will be clearly and completely described in combination with the drawings in the embodiments of the present application.

[0060] The terms "first", "second", and the like in the specification and claims of the present application and the above drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed or can optionally include other steps or units inherent to the process, method, product, or device.

[0061] Figure 1 A flowchart of an online learning state detection method based on multi-modal data fusion provided by the embodiment is shown in Figure 1 The method mainly includes the following steps:

[0062] Step one, obtaining the facial image (H pic_0 ) and the electroencephalogram (H EEG_0 ) of the object to be detected (H);

[0063] In the embodiment, the object to be detected (H) is an object that is using a learning device for online learning, Figure 2 A data acquisition diagram in an online learning environment provided by the embodiment is shown in Figure 2 In one specific example, the facial image (H pic_0 ) of the object to be detected is recorded by using an RGB-D camera, and the electroencephalogram of the object to be detected is extracted by using a Bluetooth electroencephalogram device, and the electroencephalogram (H EEG_0 ) corresponding to the facial image in time is extracted. The RGB-D camera can be integrated in the learning device, or an independent device can be used and erected on the learning device, or placed in other areas where the facial image and electroencephalogram data of the object to be detected can be collected. In this example, it is introduced according to the integration in the learning device.

[0064] In an optional embodiment, the facial image (H pic_0 ) and the electroencephalogram (H EEG_0 ) of the object to be detected are preprocessed according to requirements, specifically:

[0065] (1) Select the corresponding video, sample an image according to the set frequency (for example, every 20 frames), and crop each frame to detect the object face image (H pic_0 ) according to the same size, to obtain the preprocessed face image (H pic ).

[0066] (2) The electroencephalogram (H EEG_0 ) is down-sampled, band-pass filtered, etc., so as to improve the signal-to-noise ratio of the signal, remove artifacts, interference, noise as much as possible, and retain pure electroencephalogram; for electroencephalogram data, the main thing to remove is eye movement interference and eye muscle interference, to obtain the preprocessed electroencephalogram (H EEG ).

[0067] In one specific example, the EEGLAB toolbox is used to preprocess the electroencephalogram (H EEG_0 ). Since the power frequency interference is higher than 50hz, and some useless physiological noise is lower than 1hz, the sampling rate is reduced to 100hz, and the electroencephalogram signal is band-pass filtered at 1-50hz to remove artifacts. After processing the data, the method for removing the channels with obvious electrooculogram or electromyogram signal interference (i.e. bad leads) is as follows: the average value of the four electrode signals around the bad lead is used for processing. The transition period is removed, and three signals of about 10 minutes are obtained, which are respectively marked as concentration, distraction and fatigue.

[0068] Step two, input the face image information (H pic ) and the electroencephalogram (H EEG ) into the trained multi-modal fusion model (FSNet) to extract the current gaze direction (C e ) and the electroencephalogram emotional state (C EEG ) of the to-be-detected object (H);

[0069] Figure 3 is a network structure diagram of the multi-modal fusion model provided by the embodiment, as shown in Figure 3 , the multi-modal fusion model (FSNet) includes a first network model (FE 2 GNet) and a second network model (SCPNet); the first network model is trained by using a plurality of face image samples with gaze direction labels, and the second network model is trained by using a plurality of electroencephalogram samples with electroencephalogram emotional labels;

[0070] The multi-modal fusion model fuses and classifies the gaze features output by the first network model (FE 2 GNet) and the electroencephalogram emotional features output by the second network model (SCPNet) to obtain the current gaze direction and electroencephalogram emotional state of the to-be-detected object;

[0071] In this embodiment, the first network model is a convolutional transformation optimization network (FE 2 GNet) for extracting line-of-sight features of RGB image data.

[0072] The first network model is trained using a plurality of face image samples with line-of-sight labels, and mainly includes three network modules, namely two identical convolution modules sharing weights, a feature reconstruction module, and an encoding perception module. The following modules are described in detail as follows:

[0073] (1) The first convolution module and the second convolution module sharing weights: The main task is to extract image features. Given a face image H pic ∈R h×w×c , a CNN is used to process the face image and obtain a feature map f pic ∈R h×w×c . The CNN extracts local features from the image, and each feature contains information of a local region.

[0074] More specifically, given a face image H pic of 224x224x3, a convolutional layer of ResNet-18 is used to process the face image and obtain a feature map f pic . The convolutional layer generates 7x7x512 feature maps from the face image.

[0075] (2) Feature reconstruction module (RE-Moudle): The main task is to reconstruct the original picture from the feature space for image recovery. In addition, the ultimate task of the feature reconstruction module is to combine with one of the convolution modules (for example, the second convolution module) as an adversarial reconstruction network, and combine with other module loss functions to optimize the network for the purpose of minimizing the TOT loss function.

[0076] As shown in Figure 3 , the feature reconstruction module RE-Moudle includes an up-sampling block and a 1x1 convolutional layer to align the channels. A five-layer RE module is used, and the number of feature maps of each block is 256, 128, 64, 32, and 16 (from bottom to top), respectively. The number of feature maps of the last 1x1 convolutional layer is 3. More specifically, the adversarial loss (L ag ) is assigned to the second convolution module CNN and the feature reconstruction module. The feature reconstruction module can reconstruct the image and use the MSE loss (L re ) to optimize. The second convolution module CNN network organizes the reconstruction, where L ag =1-L reThe second convolutional module CNN and the feature reconstruction module RE-Module are antagonistic in reconstruction, i.e., the second convolutional module CNN will eventually remove general image information from the extracted features. In this way, the next step of gaze estimation is prepared.

[0077] (3) Encoding perception module: The main task is to obtain the gaze feature from the feature space, which mainly consists of a transformer and a double-layer MLP perception. First, use a 1x1 convolutional layer to scale the channel to obtain a 7x7x32 feature map. Input the feature map into a six-layer transformer and MLP to regress the gaze.

[0078] More specifically, the feature map is reconstructed into a 2D patchf p ∈R l×c , where l = h x w, thus obtaining a feature matrix of length l, each feature being c-dimensional. Input this image feature matrix into the Transformer encoder to obtain the output of the Transformer encoder, which is input into the MLP perception to obtain the result of the electroencephalogram emotion feature. In this embodiment, the hidden dimension of the double-layer MLP is set to 512, and 8 heads of self-attention mechanism are performed, and the Dropout is set to 0.1.

[0079] In addition, an additional tokenf token is added to the image feature matrix, where token is a learnable embedding with the same dimension as the feature vector, i.e., f token ∈R 1×c During training, token aggregates the features of other patches through self-attention mechanism, and finally outputs the gaze feature at the output of the transformer. The position information of each patch is encoded by using the position embedding method. A learnable embedding f pose ∈R (l+1)×c is created and added to the image feature matrix. In general, the final feature matrix is defined as:

[0080]

[0081] where [] represents the concatenation operation, and f ∈ R (l+1)×c .

[0082] The obtained feature matrix is input into the transformer, and the transformer processes the feature matrix to output a new feature matrix f out ∈R (l+1)×c . The first feature vector (corresponding to the position of f token ) is selected as the gaze representation, and the MLP is used to regress the gaze from the gaze representation. The whole process can be expressed as:

[0083]

[0084] where [0, :] means we select the first row of the feature matrix, and g represents the estimated gaze.

[0085] In an optional embodiment, the first network model is trained according to multiple sample images with gaze direction labels, and the training process includes:

[0086] obtaining face image samples with gaze direction labels;

[0087] inputting the face image samples into the first network model to be trained to output predicted gaze features through the model;

[0088] calculating a first loss function according to the gaze features and the gaze direction labels, and adjusting the model parameters of the first network model to be trained in a reverse direction until the first loss function is minimized, thereby obtaining the trained first network model.

[0089] In the training process, the total loss function of the first network model is defined as the sum of the loss functions of the second feature extraction layer, the feature reconstruction module and the encoding perception module. Specifically:

[0090] The feature reconstruction module is used to reconstruct the image, and the MSE loss is used, denoted as L re which is defined as:

[0091]

[0092] where p is the input face image, is the reconstructed face image.

[0093] The second convolutional module is in confrontation with the feature reconstruction module, and the confrontation loss is denoted as L ag which is defined as:

[0094]

[0095] The encoding perception module is used to recover the gaze from the feature space, and the L1 loss is used as the gaze estimation loss function, denoted as L e which is defined as:

[0096]

[0097] where e∈R 3 is the actual gaze direction, is the estimated gaze direction;

[0098] The total loss function combines the gaze estimation loss function and the confrontation loss function, and is defined as:

[0099]

[0100] where T is an indicator function, M is an attention map, * denotes element-wise multiplication, E[.] denotes mathematical expectation, β1 and β2 are hyperparameters, k is a threshold, and according to experience, β1 and β2 are set to 1 and k is set to 0.70. The adam is used for optimization, and the network parameters are optimized by minimizing the TOT function.

[0101] In this embodiment, the second network model is a routing conversion network (SCPNet) used to extract emotional features of the electroencephalogram (EEG). EEG The main steps are as follows:

[0102] First, the primary features of the electroencephalogram (EEG) are extracted using the primary capsule layer. EEG

[0103] In this embodiment, the primary capsule layer is a 32-channel convolutional capsule layer, each capsule is an 8D vector, and each primary capsule includes 8 convolutional units, each of which uses a 9*9 convolution kernel with a step size of 2.

[0104] Then, the output of the primary capsule layer is converted into a tensor as the input of the routing module, and the specific implementation of the routing algorithm is as follows:

[0105] (u ti ,u fi ) represents the i-th output of the l-1 layer capsule.

[0106] The feature attributes of the low-level capsule are encapsulated into a predicted time-frequency vector, denoted as The definition is as follows:

[0107]

[0108] The above formula can be arranged as:

[0109]

[0110] where N represents the number of input capsules, T and F represent the input time and frequency dimensions, respectively, K represents the kernel function, and N = 32 x h x w. A 2D convolution function is used to replace the multiplied weight s, and the same padding is used in the convolution, and the output size is the same as the input size.

[0111] The predicted capsules are weighted and summed with different weights to obtain (s tj ,s fj ), and the high-level input is obtained, and the formula is as follows:

[0112] ​

[0113] c itj With c ifj being the coupling coefficient defined as:

[0114]

[0115]

[0116] The output of the N capsules is denoted as S N defined as:

[0117]

[0118] where S N can be seen as N matrix vectors, each element s n has a size of TxF, and the matrix vector contains information of two dimensions, the information obtained in each matrix vector represents different "features";

[0119] The weight of the nth capsule of S is denoted as w n defined as:

[0120]

[0121] Generating appropriate weights for each capsule matrix vector, the capsule weight can be seen as a set of time-frequency descriptors, the statistics of which help to express the entire capsule layer, which is arranged as follows:

[0122]

[0123] Then, the received capsules are subjected to nonlinear activation and compression, which have two dimensions of time and frequency, denoted as (v tj ,v fj ), representing the probability of each element representing a certain attribute, and the formula is:

[0124]

[0125] where, is the total input of the jth capsule, and (v tj ,v fj ) is the total output of the jth high-level capsule after nonlinear activation and compression.

[0126] In the SCPNet network model, (c itj ,c ifj ), and (v tj ,v fj ) are iteratively updated according to formula 16 to extract global time-frequency information.

[0127] (c itj ,c ifj )←(c itj ,c ifj )+(c itj ,c ifj )(v tj ,v fj ) (16)

[0128] More specifically, Figure 3 wherein T i ×F i ×N i is the input size of the main capsule layer, and T0×F0×N0is the output size. The pooling layer is global average pooling, and f represents a Softmax function, and the classification electroencephalogram feature is output by the softmax function.

[0129] In an optional embodiment, the second network model is trained using a plurality of electroencephalogram samples with electroencephalogram emotion labels, and the training process includes (taking an electroencephalogram sample as an example):

[0130] obtaining electroencephalogram samples with focus state labels as focus samples, electroencephalogram samples with wandering state labels as wandering samples, and electroencephalogram samples with fatigue state labels as fatigue samples;

[0131] inputting all electroencephalogram samples with state labels into the second network model, taking cross entropy as a loss function, and adjusting the model parameters of the second network model to be trained in reverse until the loss function is minimized, to obtain a trained second network model, and the loss function is represented as L CAP , as shown in the following formula:

[0132]

[0133] wherein t i is a true value, is a predicted value obtained by the SCPNet network model.

[0134] In this embodiment, the multi-modal fusion model further includes an aggregation network and a classification network;

[0135] The aggregation network is connected to the output side of the first network model and the second network model respectively, and is used to aggregate the line-of-sight feature (F e ) output by the first network model and the electroencephalogram emotion feature (F EEG ) output by the second network model to obtain an aggregated feature;

[0136] In a specific example, the classification network can adopt a softmax classifier, which generates the current line-of-sight direction (Ce ) and electroencephalogram emotional state (C EEG ).

[0137] Step three, generating the current learning state data of the to-be-detected object based on the gaze direction (C e ) and electroencephalogram emotional state (C EEG ).

[0138] After obtaining the gaze direction (C e ) and electroencephalogram emotional state (C EEG ) of the to-be-detected object through the multi-modal fusion model, different ways can be taken to determine the learning state data of the to-be-detected object based on the gaze direction and electroencephalogram emotional state.

[0139] In one specific example, the gaze direction (C e ) and electroencephalogram emotional state (C EEG ) are input into a learning state evaluator, which is a pre-trained model, to calculate the current gaze landing point of the to-be-detected object according to the gaze direction and a preset learning space region, and to obtain the current learning state data of the to-be-detected object according to a preset standard, combining the gaze landing point position and the electroencephalogram emotional state.

[0140] The horizontal distance l (l can be obtained from the depth map) between the eyes of the to-be-detected object and the learning device, the left screen distance x l , and the right screen distance x r together determine the learning interval range (θ1, θ2), that is:

[0141]

[0142] If the gaze direction is detected to be within the pre-set learning space, the gaze landing point is in the learning zone, otherwise the gaze landing point is not in the learning zone; the gaze landing point and the electroencephalogram emotional state are combined as the current learning state of the learner.

[0143] Please refer to Figure 4 , the learning state of the learner is estimated according to the gaze landing point and the electroencephalogram emotional state, specifically including: initializing the learning state score to 0, setting the division thresholds of the full state, passive state and tired state to 1, 0 and -1 respectively, scoring the gaze landing point in the learning zone as 1 and scoring the gaze landing point not in the learning zone as 0; determining the electroencephalogram emotional state as focused as 1, determining the electroencephalogram emotional state as absent-minded as 0, and determining the electroencephalogram emotional state as tired as -1; calculating the sum of the gaze estimation score and the electroencephalogram state score as the fusion score; if the fusion score is 2, it is determined that the current state is the full state, the fusion score is judged, if the fusion score is 0 / 1, it is determined that the current state is the passive state, and if the fusion score is -1, it is determined that the current state is the tired state.

[0144] After obtaining the learning state result of the to-be-detected object, corresponding intervention measures can be provided when the state is a passive state or an exhausted state, such as issuing a voice warning to remind the learner to replace the learning content or appropriately move and relax the brain when the state is a passive state, and reminding the learner to take a break in time when the state is an exhausted state.

[0145] It should be noted that although the operations of the methods of the embodiments of the present specification are described in a specific order in the above-described embodiments, this does not require or imply that the operations must be performed in this specific order, or that all of the illustrated operations must be performed to achieve the desired result. Instead, the order of the steps depicted in the flowcharts can be changed. Additionally or alternatively, certain steps can be omitted, multiple steps can be combined into one step, or one step can be divided into multiple steps.

[0146] The embodiment also provides a computer device including at least one processor and at least one memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the online learning state monitoring method described above, which will not be described here; in the embodiment, the types of the processor and the memory are not specifically limited, for example: the processor can be a microprocessor, a digital information processor, a system on a programmable chip, etc.; the memory can be a volatile memory, a non-volatile memory, or a combination thereof, etc.

[0147] The computer device can also communicate with one or more external devices such as a keyboard, a pointing terminal, a display, etc., and also communicate with one or more terminals that enable a user to interact with the computer device, and / or communicate with any terminal that enables the computer device to communicate with one or more other computing terminals (such as a network card, a modem, etc.). Such communication can be performed through an input / output (I / O) interface. Moreover, the computer device can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) through a network adapter.

[0148] The application further provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the steps of the online learning state monitoring method. The computer readable storage medium can include, but is not limited to, any type of disk, including a floppy disk, an optical disk, a DVD, a CD-ROM, a micro drive, and a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, a magnetic card or an optical card, a nanosystem (including a molecular memory IC), or any type of medium or device suitable for storing instructions and / or data.

[0149] It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the application is not limited by the action sequence described, because according to the application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily required by the application.

[0150] In the above embodiments, the description of each embodiment is focused on, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0151] Those of ordinary skill in the art can understand that all or part of the steps of the various methods in the above embodiments can be completed by a program instructing related hardware, and the program can be stored in a computer readable memory, which can include a flash disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0152] The above is only exemplary embodiments of the disclosure, which cannot limit the scope of the disclosure. That is, any equivalent changes and modifications made according to the teachings of the disclosure are still within the scope of the disclosure. Those skilled in the art will easily think of the embodiments of the disclosure after considering the specification and practicing the disclosure herein. The application is intended to cover any variations, uses, or adaptive changes of the disclosure that follow the general principles of the disclosure and include common knowledge or conventional technical means in the art that are not recorded in the disclosure. The specification and examples are only considered as exemplary, and the scope and spirit of the disclosure are defined by the claims.

[0153] The technical features of the above embodiments can be combined arbitrarily, and in order to make the description simple, not all possible combinations of the technical features in the above embodiments are described, however, as long as the combination of the technical features does not exist contradictory, it should be considered that it is within the scope recorded by the specification.

[0154] It is to be understood that the above description is intended to be illustrative and not restrictive. Many other embodiments will be apparent to those of skill in the art upon reading and understanding the above description. The scope of the application should, therefore, be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

Claims

1. A method for online learning state detection based on multi-modal data fusion, characterized in that, The method comprises the steps of: S1 obtaining a face image and an electroencephalogram signal of a to-be-detected object; S2 inputting the face image and the electroencephalogram signal into a trained multi-modal fusion model to extract a current gaze direction and an electroencephalogram emotional state of the to-be-detected object; The multi-modal fusion model comprises a first network model, a second network model, an aggregation network and a classification network; the first network model is trained by using a plurality of face image samples with gaze direction labels, and the second network model is trained by using a plurality of electroencephalogram samples with electroencephalogram emotional labels; The first network model comprises a first feature extraction layer, a second feature extraction layer, a feature reconstruction module and an encoding perception module; The first feature extraction layer and the second feature extraction layer are used to extract local features from the face image to generate a feature matrix; The feature reconstruction module is used to recover an image according to the feature matrix to reconstruct an original image from a feature space; the feature reconstruction module and the second feature extraction layer form a generative adversarial network; The encoding perception module is used to generate a gaze feature according to the feature matrix, and the encoding perception module comprises a Transformer model and a double-layer MLP perceptual. The aggregation network is used to aggregate the gaze feature output by the first network model and the electroencephalogram emotional feature output by the second network model to obtain an aggregated feature; The classification network generates a current gaze direction and an electroencephalogram emotional state of the to-be-detected object according to the aggregated feature; S3 generating learning state data of the to-be-detected object according to the gaze direction and the electroencephalogram emotional state, specifically comprising: inputting the gaze direction and the electroencephalogram emotional data into a learning state evaluator to obtain an explicit learning state of the to-be-detected object according to the gaze direction and a preset learning space region; combining the explicit learning state and the electroencephalogram emotional state to obtain learning state data of the to-be-detected object.

2. The online learning state detection method of claim 1, wherein, Before inputting the face image information and the electroencephalogram signal into the trained multi-modal fusion model, the method further comprises: performing frame division and cropping on the face image for preprocessing; performing down-sampling and band-pass filtering on the electroencephalogram signal.

3. The online learning state detection method of claim 1, wherein, The training process of the first network model comprises: obtaining face image samples with gaze direction labels; inputting the face image samples into the first network model to be trained to output predicted gaze features; calculating a first loss function according to the gaze features and the gaze direction labels, adjusting the model parameters of the first network model to be trained in a reverse direction until the first loss function is minimized, and obtaining the trained first network model.

4. The online learning state detection method of claim 3, wherein, The first loss function is defined as the sum of loss functions of the second feature extraction layer, the feature reconstruction module and the encoding perception module; The loss function of the feature reconstruction module is denoted as which is defined as: wherein, p is the input face image, is the reconstructed face image; The second feature extraction layer is in confrontation with the feature reconstruction module, and the confrontation loss is denoted as which is defined as: =1- The encoding perception module uses L 1 loss as a line of sight estimation loss function, denoted as which is defined as: wherein, e ∈ R 3 is a true line of sight direction, ∈ R 3 is an estimated line of sight direction; The first loss function combines a gaze estimation loss function and an adversarial loss function, and is defined as: wherein, T is an indicator function, M is an attention map, denotes element-wise multiplication, denotes mathematical expectation, and is a hyper-parameter; k is a self-defined threshold value.

5. The online learning state detection method of any one of claims 1-4, wherein, The training process of the second network model comprises: obtaining electroencephalogram samples with state labels, wherein the state labels include a focused state label, a mind-wandering state label and a fatigue state label; inputting the electroencephalogram samples with state labels into the second network model to output predicted electroencephalogram emotional states; According to the brain electrical emotion state and the state label, a second loss function is calculated, model parameters of a second network model to be trained are adjusted reversely until the second loss function is minimized, and a trained second network model is obtained.

6. The online learning state detection method of claim 5, wherein, The second loss function is represented as which is defined as: wherein, is the true value, is the predicted value obtained by the SCPNet network model.

7. A computer device, comprising: The computer program product comprises at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit executes the steps of the online learning state detection method in any one of claims 1-6.

Images