Interactive learning optimization method and system based on brain-eye fusion and transfer learning

Abstract

The invention discloses an interactive learning optimization method and system based on brain-eye fusion and transfer learning. The method comprises the following steps that electroencephalogram data and eye movement data of different subjects are collected; preprocessing the electroencephalogram data and the eye movement data to obtain an electroencephalogram sample and an eye movement sample, and performing data pairing; automatically extracting an electroencephalogram feature vector and an eye movement feature vector in the electroencephalogram sample and the eye movement sample by using a convolutional neural network in combination with a multi-head attention structure, and fusing the electroencephalogram feature vector and the eye movement feature vector into a final feature vector through a full connection layer; aiming at data differences among different subjects, constructing a transfer learning model; and according to the pupil size and the residence time of the user in combination with the electroencephalogram data, retraining and identifying to obtain sentiment classification for different contents. On the basis of multi-modal data fusion and transfer learning, various physiological data such as electroencephalogram signals and visual signals are fused, and the limitation of traditional single signal analysis in a complex scene is overcome.

Description

Technical Field

[0001] The present application belongs to the field of machine learning, and in particular relates to an interactive learning optimization method and system based on brain-eye fusion and transfer learning. Background Art

[0002] In the current fields of education, medical diagnosis, brain-computer interface, cognitive research, etc., personalized learning, medical diagnosis, and cognitive status assessment have become the focus of researchers in various fields.

[0003] Taking the education sector as an example, teachers and educational institutions hope to provide more personalized teaching content and feedback based on the user's learning status and ability. However, existing learning monitoring and analysis tools mainly rely on a single data source (such as video recordings, test scores, etc.), which cannot fully and real-time reflect the real status of students in the learning process. This traditional single teaching model has the following limitations:

[0004] Lack of real-time analysis of cognitive status; students’ misunderstandings in learning are not easy to be discovered, and the lagging feedback mechanism cannot promptly correct students’ misunderstandings or distractions in the learning process, resulting in low learning efficiency, and this phenomenon may continue;

[0005] Poor adaptability across individuals: Everyone has different learning styles and cognitive abilities. Existing learning tools are difficult to adapt to individual differences and cannot be dynamically adjusted to meet the specific needs of different students.

[0006] Lack of integration of multimodal data; a single data source alone cannot reveal the comprehensive cognitive responses of students in different types of learning tasks. At the same time, there is a lack of the ability to effectively integrate multimodal data, which makes it impossible to capture the comprehensive learning status of students in complex learning scenarios, limiting the accuracy and effectiveness of personalized learning recommendations.

[0007] Electroencephalogram (EEG) and eye movement signals are two key physiological signals, each playing an important role in different fields. By recording the electrical activity of brain neurons, EEG can provide information about the dynamic state of the brain with high temporal resolution, which makes it widely used in education, medical diagnosis, brain-computer interface and cognitive science research. Although EEG can obtain internal activity data of the brain non-invasively, its signals are often complex and susceptible to noise interference, and the individual differences are obvious, which increases the difficulty of classification and analysis. In contrast, eye movement signals mainly reflect the individual's visual focus and line of sight movement trajectory. Recording these signals with an eye tracker can accurately track the user's visual behavior. Eye movement signals also have the advantage of being non-invasive and can reveal in detail the user's attention distribution when performing tasks, which makes them widely used in user experience analysis, psychological research and clinical diagnosis. Combining these two signals can give full play to their respective advantages and provide a more comprehensive analysis of individual cognitive states. For example, EEG signals reveal the internal activities of the brain, while eye movement signals provide information on external behavior and visual attention. The fusion of the two can significantly improve the accuracy of data analysis and provide strong support for real-time monitoring and feedback, which is particularly important for application scenarios such as personalized learning and smart medical care. Summary of the invention

[0008] In response to the problems existing in the prior art, this application proposes an interactive learning optimization method and system based on brain-eye fusion and transfer learning, which achieves more efficient and accurate emotion classification and state analysis, and can detect the user's state in the learning process in real time.

[0009] In a first aspect, the present application provides an interactive learning optimization method based on brain-eye fusion and transfer learning, comprising the following steps:

[0010] S1, collect EEG data and eye movement data of different subjects;

[0011] S2, preprocessing the EEG data and the eye movement data to obtain EEG samples and eye movement samples, which constitute training data; performing data pairing on the training data;

[0012] S3, using a deep neural network combined with a multi-head attention structure to automatically extract EEG feature vectors and eye movement feature vectors from EEG samples and eye movement samples, and merging the EEG feature vectors and eye movement feature vectors into the final feature vector through a fully connected layer;

[0013] S4, in view of the data distribution differences between different subjects, a transfer learning model is constructed, and the data distribution differences are reduced by the maximum mean difference criterion to obtain an adjusted deep neural network;

[0014] S5, uses the adjusted deep neural network to retrain the target data and identify the sentiment classification results for different contents.

[0015] Optionally, in step S1, the course is presented to the subject in a visual manner, and the EEG data and eye movement data of the subject while watching the course are collected.

[0016] Optionally, step S2 includes a preprocessing step of using a bandpass filter to extract frequency bands related to brain activity in the EEG data and / or using baseline drift correction to correct the EEG data.

[0017] Furthermore, step S2 may also include a preprocessing step of smoothing the eye movement data.

[0018] Furthermore, in step S2, the data pairing includes the following contents:

[0019] Mark the data for synchronization to ensure alignment between EEG data and eye movement data;

[0020] The subject provides labels for the EEG data and the eye movement data, and the number of the labels corresponds to the number of emotion types;

[0021] Construct positive and negative sample pairs for EEG data and eye movement data respectively. The construction method is as follows: Assuming there are M kinds of emotions, the same category label data are combined in pairs to form M kinds of positive sample pairs, and the different category label data are combined in pairs to form 1+2+…+(M-1) kinds of negative sample pairs;

[0022] Furthermore, in step S3, for each EEG sample and eye movement sample, it is first processed by a convolutional neural network to extract multi-scale features at different levels, and then the extracted multi-scale features are input into a multi-head attention mechanism, and the multi-scale features are weighted by a self-attention module to automatically adjust the weight of each feature in the global context to highlight the features most relevant to emotion classification; specifically, the following contents are included:

[0023] S30, extracts multi-scale features through multi-layer convolution and pooling operations:

[0024]

[0025]

[0026] Among them, F (l) (X) represents the feature map of the input sample X after the convolution operation at the lth layer; ReLU represents the activation function; W (l) Represents the convolution kernel weight matrix in the l-th layer convolution operation; represents the feature map after the maximum pooling of the lth layer; b (l)Represents the bias term of the l-th layer convolution operation; Maxpool represents the maximum pooling operation;

[0027] S31, the extracted multi-scale features are mapped to the query space, key space, and value space, and the calculation formula is as follows:

[0028] Q=W Q F multi ,K=W K F multi ,V=W V F multi

[0029] Among them, W Q , W K , W V are the linear transformation matrices of query, key and value respectively; Q represents the query vector, K represents the key vector, and V represents the value vector;

[0030] S32, calculate the self-attention weight and weighted output:

[0031]

[0032] Among them, d k is the dimension of the key vector, used to ensure numerical stability when computing dot products;

[0033] S33, each attention head performs the above operation independently, and then concatenates the outputs of all heads:

[0034] MultiHead(Q,K,V)=Concat(Head1,Head2,...,Head N )W O

[0035] Where N is the number of attention heads, h∈[1,N]; W O is the output linear transformation matrix; Head h represents the hth attention head.

[0036] Furthermore, after feature extraction, the EEG and eye movement feature vectors are obtained. Through the fully connected layer, the fused EEG and eye movement feature vectors are converted into the final feature vector:

[0037] F final =FC(MultiHead(Q,K,V))

[0038] F final is the final feature vector, which is input into the subsequent transfer learning model.

[0039] Furthermore, in step 4, the training data is defined as the source domain, and the data to be tested is defined as the target domain, and the difference between the feature distributions of the source domain and the target domain is reduced by domain adaptation; specifically, the following contents are included:

[0040] S41, input the data of the source domain and the target domain into the feature alignment module; let the source domain feature be represented as F s , the target domain feature is represented by F t , the goal of feature alignment is to make F s and F t The distribution in the feature space is close;

[0041] S42, measures the distribution difference between the two domains through the maximum mean difference criterion and minimizes this difference during training:

[0042]

[0043] Among them, n s and n t are the number of samples in the source domain and the target domain, i∈[1,n s ],j∈[1,n t ]; φ(·) is the characteristic mapping function, and H is the reproducing kernel Hilbert space.

[0044] Furthermore, in step 42, a domain adaptation network is used to further reduce the distribution difference between the source domain and the target domain features; specifically, the following contents are included:

[0045] The features of the source domain and the target domain are respectively input into the shared deep neural network, and the classification loss and domain adaptation loss of the source domain are optimized simultaneously through back propagation; the ultimate goal is to minimize the total loss function:

[0046] L total =L classification +λ·L MMD

[0047] Among them, L classification is the classification loss of the source domain; L MMD is the feature alignment loss; λ is the trade-off coefficient used to control the impact of the two.

[0048] In a second aspect, the present application provides a system for executing the interactive learning optimization method, comprising:

[0049] Brain-computer equipment, used to collect EEG data and eye movement data from different subjects;

[0050] A preprocessing module is used to preprocess the EEG data and the eye movement data to obtain EEG samples and eye movement samples; the EEG samples and the eye movement samples constitute training data; and the training data is paired;

[0051] The feature extraction and fusion module uses a deep neural network combined with a multi-head attention structure to automatically extract EEG feature vectors and eye movement feature vectors from EEG samples and eye movement samples, and fuses the EEG feature vectors and eye movement feature vectors into the final feature vector through a fully connected layer;

[0052] The transfer learning module measures the distribution difference between the source domain and the target domain by the maximum mean difference criterion, and obtains an adjusted deep neural network.

[0053] The classification and recognition module uses the adjusted deep neural network to retrain the test data based on the user's pupil size and dwell time combined with EEG data, obtains emotion classification for different content and outputs recognition results.

[0054] Compared with the existing methods, this application has the following beneficial effects:

[0055] (1) This application uses a multi-scale feature extraction method combined with a multi-head attention mechanism to perceive information more comprehensively and synthesize more accurate prediction results. By introducing information fusion and transfer learning technology, multiple physiological data such as EEG signals and visual signals are integrated, overcoming the limitations of traditional single signal analysis in complex scenarios;

[0056] (2) This application can analyze and classify the user's learning status more efficiently and accurately, thereby assisting educational work; at the same time, this application can maintain high accuracy across individual applications and provide real-time feedback to users, thereby improving the adaptability and application value of the system;

[0057] (3) This application can realize the integration of cross-domain physiological data, and thus be widely used in education, medical diagnosis, brain-computer interface, cognitive research and other fields, providing more efficient and intelligent solutions for personalized education, medical diagnosis, cognitive status assessment and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is the process of an interactive learning optimization method based on brain-eye fusion and transfer learning in one embodiment of the present application;

[0059] Figure 2 This is a schematic diagram of feature extraction combining CNN and attention mechanism in one embodiment of the present application;

[0060] Figure 3 This is a schematic diagram of feature extraction and multimodal fusion in one embodiment of the present application;

[0061] Figure 4 This is a schematic diagram of transfer learning in one embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings.

[0063] It should be clear that the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0064] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0065] It should be understood that the term "and / or" used in this article is only a description of the same field of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0066] In a first aspect, the present application provides an interactive learning optimization method based on brain-eye fusion and transfer learning, comprising the following steps:

[0067] S1, set up the brain-computer experiment paradigm, and the subjects wore the brain-computer device (integrated surface electrodes and eye tracker).

[0068] Used to collect EEG data and eye movement data;

[0069] S2, preprocessing the EEG data and eye movement data to obtain EEG samples and eye movement samples (training data), and pairing the training data;

[0070] S3, using a deep neural network combined with a multi-head attention structure to automatically extract EEG feature vectors and eye movement feature vectors from EEG samples and eye movement samples, and merging the EEG feature vectors and eye movement feature vectors into the final feature vector through a fully connected layer;

[0071] S4, based on the data differences between different subjects, a transfer learning model was constructed to reduce the data distribution differences through the maximum mean difference criterion and adjust the parameters of the deep neural network;

[0072] S5, for the target data to be tested, according to the user's pupil size, dwell time and EEG data, the adjusted deep neural network is used to retrain the target data to be tested, and the emotion classification for different contents is identified.

[0073] Optionally, in step S1, a PPT online course is made as a visual presentation paradigm, the online course video content is played to the subjects, and the brain-computer device is used to synchronously collect the EEG data and eye movement data of the subjects while watching the video.

[0074] Furthermore, in step S2, the EEG data and eye movement data collected in step S1 are preprocessed respectively, preferably using a 0.5 Hz to 75 Hz bandpass filter to retain the main frequency bands related to brain activity in the EEG signal, and baseline drift correction can be used to eliminate slow potential drift caused by poor electrode contact or patient head movement; at the same time, the eye movement signal is smoothed to reduce noise in the eye movement signal caused by rapid eye movement or device jitter.

[0075] Furthermore, the data are synchronously marked to ensure that the EEG data is aligned with the eye movement data;

[0076] The subject provides labels for the EEG data and the eye movement data, and the number of the labels corresponds to the number of emotion types;

[0077] Construct positive and negative sample pairs for EEG data and eye movement data respectively. The construction method is as follows: Assuming there are M kinds of emotions, the same category label data are combined in pairs to form M kinds of positive sample pairs, and the different category label data are combined in pairs to form 1+2+…+(M-1) kinds of negative sample pairs;

[0078] The positive and negative sample pairs of EEG data and eye movement data are input into Figure 3 In the model shown;

[0079] This example uses "interest, confusion, and surprise" as an example. The subject provides three labels: interest, confusion, and surprise. Samples with different labels are used to construct positive and negative sample pairs. The construction method is: the same category is combined into three positive sample pairs, and different categories are combined into three negative sample pairs; the same method is used to construct samples of EEG and eye movement signals.

[0080] Furthermore, in step S3, for each EEG sample and eye movement sample, it is first processed by a convolutional neural network (CNN) to extract multi-scale features at different levels, and then the extracted multi-scale features are input into a multi-head attention mechanism to perform weighted processing on the multi-scale features, and automatically adjust the weight of each feature in the global context to highlight the features most relevant to emotion classification; specifically, the following contents are included:

[0081] S30, through multi-layer convolution and pooling operations, extracts deeper features:

[0082]

[0083]

[0084] F (l) (X) represents the feature map of the input sample X after the convolution operation at the lth layer, which contains the features extracted by the convolution kernel at different time steps and spatial positions, and is used to represent the key patterns and information in the signal;

[0085] ReLU stands for Rectified Linear Unit, which is an activation function used to introduce nonlinearity. Its expression is as follows:

[0086] ReLU(x)=max(0,x)

[0087] In the feature extraction process, ReLU is used to enhance the feature representation ability of the convolutional layer, remove negative values, and make the network more stable;

[0088] W (l) Represents the convolution kernel weight matrix in the l-th layer convolution operation; the convolution kernel is a trainable parameter used to extract characteristic patterns in the input signal, such as waveforms in time series or spatial changes;

[0089] Represents the feature map after the lth layer of maximum pooling (MaxPooling); the pooling operation is used to reduce the size of the feature map, retain important feature information, reduce the amount of data and computational complexity, and improve the robustness of the feature.

[0090] b (l) Represents the bias term of the l-th layer convolution operation, which is a trainable parameter used to adjust the output value of the convolution operation and enhance the expressiveness of the network.

[0091] Maxpool represents the maximum pooling operation, which is used to slide a window on the feature map and take the maximum value in each window, retaining the significant features of the signal and removing unimportant details, thereby reducing the dimension and complexity of the feature map;

[0092] S31, the extracted multi-scale features are mapped to the query, key, and value spaces, and the calculation formula is as follows:

[0093] Q=W Q F multi ,K=W K F multi ,V=W V F multi

[0094] Among them, W Q , W K , W V are the linear transformation matrices for query, key, and value respectively;

[0095] Q (query, Query) represents the query vector, which is a multi-scale feature F multi The result obtained by linear transformation. The query vector is used to query other feature information to determine which features are important in the global context.

[0096] K(key, Key) represents the key vector, which is composed of multi-scale features F multi Obtained through linear transformation. The key vector is used to represent the identity of each feature and calculates the similarity with the query vector Q to determine the weight of the feature in the self-attention mechanism.

[0097] V (value, Value) represents the value vector, which is also composed of multi-scale features F multi Obtained through linear transformation. The value vector contains the information content of the feature, and is used for weighted summation after the weight is calculated to generate the final output feature;

[0098] S32, calculate the self-attention weight and weighted output:

[0099]

[0100] Among them, d k is the dimension of the key vector, used to ensure numerical stability when computing dot products;

[0101] S33, in the multi-head attention mechanism, each attention head performs the above operation independently, and then concatenates the outputs of all heads:

[0102] MultiHead(Q,K,V)=Concat(Head1,Head2,...,Head N )W O

[0103] Where N is the number of attention heads, h∈[1,N]; W O is the output linear transformation matrix; Head h represents the hth attention head;

[0104] In the multi-head attention mechanism, the automatic adjustment of weights is achieved through the self-attention mechanism; the self-attention mechanism dynamically calculates the importance of each feature based on the relationship between feature vectors (i.e. similarity or correlation), and then automatically assigns different weights to them;

[0105] like Figure 2 The figure shows the feature extraction process mentioned above. As the extraction goes deeper, features at different levels will be extracted. These levels are the features extracted by the convolution kernel at different time steps and spatial positions. The extraction results may be local-global information, which may map the low-frequency-high-frequency part of the signal.

[0106] After feature extraction, the EEG and eye movement feature vectors are obtained. Through the fully connected layer, the fused EEG and eye movement feature vectors are converted into the final feature vector:

[0107] F final =FC(MultiHead(Q,K,V))

[0108] F final is the final feature vector, which is input into the subsequent transfer learning model;

[0109] like Figure 3 Shown is the multimodal fusion process.

[0110] Furthermore, in step 4, the training data is defined as the source domain, and the target data to be tested is defined as the target domain; the difference between the feature distribution of the source domain and the target domain is reduced by domain adaptation; specifically, the following contents are included:

[0111] S41, input the data of the source domain and the target domain into the feature alignment module; let the source domain feature be represented as F s , the target domain feature is represented by F t , the goal of feature alignment is to make F s and F t The distribution in the feature space is close;

[0112] S42, the distribution difference between the two domains is measured by the Maximum Mean Discrepancy (MMD) criterion, and this difference is minimized during training:

[0113]

[0114] Among them, n s and n t are the number of samples in the source domain and the target domain, i∈[1,n s ],j∈[1,n t ]; φ(·) is the eigenmapping function, and H is the reproducing kernel Hilbert space (RKHS).

[0115] Preferably, step 42 uses a domain adaptation network to further reduce the distribution difference between the source domain and the target domain features; specifically includes the following contents:

[0116] The features of the source domain and the target domain are respectively input into the shared deep neural network, and the classification loss and domain adaptation loss of the source domain are optimized simultaneously through back propagation; the ultimate goal is to minimize the total loss function:

[0117] L total =L classification +λ·LMMD

[0118] Among them, L classification is the classification loss of the source domain; L MMD is the feature alignment loss; λ is the trade-off coefficient used to control the influence of the two;

[0119] like Figure 4 The figure shows the process in which the transfer learning model measures the distribution difference between the source and target domain features by using the maximum mean difference criterion and adopts a domain adaptation network to further narrow the distribution difference between the source and target domain features.

[0120] The above transfer learning is mainly used to adjust the difference between the source domain and the target domain, modify the previous deep model parameters, reduce the difference in data distribution of different subjects (data is distributed), and also adjust the deep model so that the model can adapt well when facing data from different subjects. After reducing the difference in distribution, the target domain is used for training. At this time, the training module is not the transfer learning module, but the adjusted deep model.

[0121] In a second aspect, the present application provides a system for executing the interactive learning optimization method, comprising:

[0122] Brain-computer equipment, used to collect EEG data and eye movement data from different subjects;

[0123] A preprocessing module is used to preprocess the EEG data and eye movement data to obtain EEG samples and eye movement samples to form training data, and perform data pairing on the training data;

[0124] The feature extraction and fusion module uses a deep neural network combined with a multi-head attention structure to automatically extract EEG feature vectors and eye movement feature vectors from EEG samples and eye movement samples, and fuses the EEG feature vectors and eye movement feature vectors into the final feature vector through a fully connected layer;

[0125] The transfer learning module measures the distribution difference between the source domain and the target domain by the maximum mean difference criterion, and obtains an adjusted deep neural network.

[0126] The classification and recognition module uses the adjusted deep neural network to retrain the target data according to the user's pupil size and dwell time combined with EEG data, obtains emotion classification for different contents and outputs recognition results.

[0127] In a third aspect, the present application further provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the method in the above embodiment when executed by a processor.

[0128] For example, the above can be achieved Figure 1 The steps of the interactive learning optimization method shown in .

[0129] Computer readable storage media can be tangible devices that can hold and store instructions used by instruction execution devices. Computer readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the above.

[0130] The computer program instructions used to perform the operations of the present application may be source code or object code written in any combination of one or more programming languages, and the computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server.

[0131] In summary, this embodiment adopts a multimodal feature extraction method of convolutional neural network and attention mechanism to enhance the model's ability to recognize complex emotional states. In addition, this embodiment achieves the alignment of the feature distribution of the source domain and the target domain by introducing domain adaptation technology, thereby significantly improving the generalization ability and classification accuracy of the model in different data domains. And through the maximum mean difference (MMD) minimization method, the distribution difference between the source domain and the target domain is effectively reduced, so that the model can also maintain efficient emotion classification performance in the new domain environment. After domain adaptation adjustment, the model is retrained on the target domain, which not only ensures the accuracy of the classification results, but also simplifies the adaptation process of the model in the new data environment. Finally, under the condition of multimodal data, the interactive learning optimization system that can automatically adjust and adapt to different data domains can be used to detect students' learning status, assist education work, and can be extended to various complex application scenarios such as medical diagnosis and cognitive status assessment. The system has significant practicality and reliability.

[0132] The above embodiments are only for illustrating the technical concept and features of the present application, and their purpose is to enable technicians familiar with the technical field to understand the content of the present application and implement it accordingly, and they cannot be used to limit the protection scope of the present application. For ordinary technicians in the technical field, they can make several improvements and modifications without departing from the principles of the present application, and these improvements and modifications should also be regarded as the protection scope of the present application.

Claims

1. An interactive learning optimization method based on brain-eye fusion and transfer learning, characterized in that: The steps include: S1, collect EEG data and eye movement data of different subjects; S2, preprocessing the EEG data and the eye movement data to obtain EEG samples and eye movement samples, which constitute training data; performing data pairing on the training data; S3, using a deep neural network combined with a multi-head attention structure to automatically extract EEG feature vectors and eye movement feature vectors from EEG samples and eye movement samples, and merging the EEG feature vectors and eye movement feature vectors into the final feature vector through a fully connected layer; S4, in view of the data distribution differences between different subjects, a transfer learning model is constructed, and the data distribution differences are reduced by the maximum mean difference criterion to obtain an adjusted deep neural network; S5, uses the adjusted deep neural network to retrain the target data and identify the sentiment classification results for different contents.

2. The interactive learning optimization method according to claim 1, characterized in that: In step S1, the course is presented to the subject in a visual manner, and the EEG data and eye movement data of the subject while watching the course are collected.

3. The interactive learning optimization method according to claim 1, characterized in that: Step S2 includes a preprocessing step of using a bandpass filter to extract the frequency band related to brain activity in the EEG data and / or using a baseline drift to correct the EEG data.

4. The interactive learning optimization method according to claim 1, characterized in that: Step S2 includes a preprocessing step of smoothing the eye movement data.

5. The interactive learning optimization method according to claim 1, characterized in that: In step S2, the data pairing includes the following contents: Mark the data for synchronization to ensure alignment between EEG data and eye movement data; The subject provides labels for the EEG data and the eye movement data, and the number of the labels corresponds to the number of emotion types; Positive and negative sample pairs are constructed for EEG data and eye movement data respectively. The construction method is as follows: assuming there are M kinds of emotions, the label data of the same category are combined in pairs to form M kinds of positive sample pairs, and the label data of different categories are combined in pairs to form 1+2+…+(M-1) kinds of negative sample pairs.

6. The interactive learning optimization method according to claim 1, characterized in that: In step S3, for each EEG sample and eye movement sample, it is first processed by a convolutional neural network to extract multi-scale features at different levels, and then the extracted multi-scale features are input into the multi-head attention mechanism. The multi-scale features are weighted by the self-attention module, and the weight of each feature in the global context is automatically adjusted to highlight the features most relevant to emotion classification. Specifically, the following contents are included: S30, extracts multi-scale features through multi-layer convolution and pooling operations: Among them, F (l) (X) represents the feature map of the input sample X after the convolution operation at the lth layer; ReLU represents the activation function; W (l) Represents the convolution kernel weight matrix in the l-th layer convolution operation; represents the feature map after the maximum pooling of the lth layer; b (l) Represents the bias term of the l-th layer convolution operation; Maxpool represents the maximum pooling operation; S31, the extracted multi-scale features are mapped to the query space, key space, and value space, and the calculation formula is as follows: Q=W Q F multi ,K=W K F multi ,V=W V F multi Among them, W Q , W K , W V are the linear transformation matrices of query, key and value respectively; Q represents the query vector, K represents the key vector, and V represents the value vector; S32, calculate the self-attention weight and weighted output: Among them, d k is the dimension of the key vector, used to ensure numerical stability when computing dot products; S33, each attention head performs the above operation independently, and then concatenates the outputs of all heads: MultiHead(Q,K,V)=Concat(Head1,Head2,...,Head N )W O Where N is the number of attention heads, h∈[1,N]; W O is the output linear transformation matrix; Head h represents the hth attention head.

7. The interactive learning optimization method according to claim 6, characterized in that: After feature extraction, the EEG and eye movement feature vectors are obtained. Through the fully connected layer, the fused EEG and eye movement feature vectors are converted into the final feature vector: F final =FC(MultiHead(Q,K,V)) F final is the final feature vector, which is input into the subsequent transfer learning model.

8. The interactive learning optimization method according to claim 1, characterized in that: In step 4, the training data is defined as the source domain and the data to be tested is defined as the target domain. The difference between the feature distribution of the source domain and the target domain is reduced by domain adaptation. Specifically, the following contents are included: S41, input the data of the source domain and the target domain into the feature alignment module; let the source domain feature be represented as F s , the target domain feature is represented by F t , the goal of feature alignment is to make F s and F t The distribution in the feature space is close; S42, measures the distribution difference between the two domains through the maximum mean difference criterion and minimizes this difference during training: Among them, n s and n t are the number of samples in the source domain and the target domain, i∈[1,n s ],j∈[1,n t ]; φ(·) is the characteristic mapping function, and H is the reproducing kernel Hilbert space.

9. The interactive learning optimization method according to claim 8, characterized in that: In step 42, a domain adaptation network is used to further reduce the distribution difference between the source domain and the target domain features; specifically, the following contents are included: The features of the source domain and the target domain are respectively input into the shared deep neural network, and the classification loss and domain adaptation loss of the source domain are optimized simultaneously through back propagation; the ultimate goal is to minimize the total loss function: L total =L classification +λ·L MMD Among them, L classification is the classification loss of the source domain; L MMD is the feature alignment loss; λ is the trade-off coefficient used to control the impact of the two.

10. A system for executing the interactive learning optimization method according to any one of claims 1 to 9, characterized in that: include: Brain-computer equipment, used to collect EEG data and eye movement data from different subjects; A preprocessing module, used for preprocessing EEG data and eye movement data to obtain EEG samples and eye movement samples; the EEG samples and eye movement samples constitute training data; Perform data pairing on training data; The feature extraction and fusion module uses a deep neural network combined with a multi-head attention structure to automatically extract EEG feature vectors and eye movement feature vectors from EEG samples and eye movement samples, and fuses the EEG feature vectors and eye movement feature vectors into the final feature vector through a fully connected layer; The transfer learning module measures the distribution difference between the source domain and the target domain by the maximum mean difference criterion, and obtains an adjusted deep neural network. The classification and recognition module uses the adjusted deep neural network to retrain the test data based on the user's pupil size and dwell time combined with EEG data, obtains emotion classification for different content and outputs recognition results.

Images