Learning effect prediction method based on electroencephalogram near-infrared bimodal signals

Abstract

The invention discloses a learning effect prediction method based on electroencephalogram near-infrared bimodal signals, relates to the technical field of information, and solves the problems that in the prior art, evaluation lacks comprehensive supervision, evaluation standards are single, and learning conditions of students cannot be accurately reflected. Pre-processed data are obtained; performing score evaluation on the preprocessed data by using the trained score evaluation model to obtain an evaluation score; making learning suggestions according to the evaluation results; according to the method, space-time-frequency spectrum information fusion feature extraction is realized, and the learning condition of students is accurately reflected.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a learning effect prediction method based on electroencephalogram (EEG) and near-infrared (NIR) dual-modal signals. Background Art

[0002] With the advancement of education reform, the education model has undergone tremendous changes, among which a "wall-less" education model is particularly prominent in the reform. However, the emerging education form also brings new challenges. For example, teachers have switched from traditional face-to-face teaching to hybrid teaching, which has led to the problem of difficulty in supervising students' learning status and results. In addition, students' self-disciplined learning ability has also become a challenge. They need to adapt to new learning methods and manage their learning progress by themselves.

[0003] At present, the evaluation of online learning mainly relies on simple learning behavior data such as the length of time students watch videos and test scores. However, this kind of evaluation lacks comprehensive supervision and has a single evaluation standard, which cannot accurately reflect the students' learning situation; the data type is single and the mining degree is limited, making it difficult to effectively supervise and evaluate students; the evaluation is often formed after the learning process is over, which has a lag, making it difficult to achieve real-time evaluation and early intervention in the learning process. Summary of the invention

[0004] The present invention provides a learning effect prediction method based on EEG near-infrared dual-modal signals, which solves the problems in the prior art that the evaluation lacks comprehensive supervision, the evaluation standard is single, and the learning situation of students cannot be accurately reflected. It realizes the feature extraction of space-time-spectrum information fusion and accurately reflects the learning situation of students.

[0005] In a first aspect, the present invention provides a learning effect prediction method based on EEG near-infrared dual-modal signals, the method comprising:

[0006] Acquiring basic data, and preprocessing the basic data to obtain preprocessed data;

[0007] The pre-processed data is evaluated using the trained performance evaluation model to obtain an evaluation score; wherein the performance evaluation model includes: an EEG data processing branch, an fNIRS data processing branch, an information interaction module and a performance regression module; wherein the EEG data processing branch and the fNIRS data processing branch are independent of each other; the EEG data processing branch and the fNIRS data processing branch interact with the information interaction module; the performance regression module obtains data from the EEG data processing branch and the fNIRS data processing branch to perform performance evaluation;

[0008] Make study recommendations based on said assessment results.

[0009] In combination with the first aspect, in a possible implementation manner, preprocessing the basic data to obtain preprocessed data includes:

[0010] Screening and classifying the basic data to obtain EEG sampling data and fNIRS sampling data;

[0011] Re-screening reference points for the EEG sampling data, and re-referencing the EEG sampling data according to the reference points to obtain re-referenced EEG sampling data;

[0012] Using a bandpass filter to perform bandpass filtering on the re-referenced EEG sampling data to obtain pre-processed EEG sampling data;

[0013] removing motion artifacts from the fNIRS sampling data, and performing band-pass filtering on the fNIRS sampling data from which the motion artifacts have been removed using a band-pass filter to obtain pre-processed fNIRS sampling data;

[0014] The preprocessed EEG sampling data and the preprocessed fNIRS sampling data are respectively segmented, and the segmented EEG sampling data and the segmented fNIRS sampling data are respectively baseline-removed to obtain preprocessed data.

[0015] In combination with the first aspect, in a possible implementation manner, before preprocessing the basic data, the method further includes: screening the basic data by using a questionnaire survey method.

[0016] In combination with the first aspect, in a possible implementation, the EEG data processing branch includes: a first single-layer convolutional codec, a first convolutional multi-head self-attention layer, and a Fourier transformed post-calculation multi-head self-attention layer connected in sequence;

[0017] The fNIRS data processing branch includes: a second single-layer convolutional codec, a second convolutional multi-head self-attention layer, and a third convolutional multi-head self-attention layer connected in sequence;

[0018] The information interaction module includes: a first information interaction layer and a second information interaction layer that are independent of each other; wherein the first information interaction layer interacts with the EEG data processing branch and the fNIRS data processing branch, and the second information interaction layer interacts with the EEG data processing branch and the fNIRS data processing branch;

[0019] The performance regression module includes: a multi-layer perception layer.

[0020] In combination with the first aspect, in a possible implementation,

[0021] The first single-layer convolutional codec is used to obtain first reconstructed EEG data based on the preprocessed data;

[0022] The first convolutional multi-head self-attention layer is used to extract the first neural electrical data feature in the first reconstructed EEG data;

[0023] The second single-layer convolutional codec is used to obtain second reconstructed fNIRS data according to the preprocessed data;

[0024] The second convolutional multi-head self-attention layer is used to extract the first blood flow data feature in the second reconstructed fNIRS data;

[0025] The first information interaction layer is used to calculate the weight of each feature in the first neural electrical data feature according to the first blood flow data feature and the first neural electrical data feature, and use the weight to weight the first neural electrical data feature to obtain first weighted feature data, and calculate the weight of each feature in the first blood flow data feature, and use the weight to weight the first blood flow data feature to obtain second weighted feature data;

[0026] The multi-head self-attention layer is calculated after the Fourier transform to convert the first weight feature data to obtain a second neural electrical data feature;

[0027] The third convolutional multi-head self-attention layer is used to extract features from the second weight feature data to obtain second blood flow data features;

[0028] The multi-layer perception layer is used to obtain an evaluation score based on the second neural electrical data characteristics and the second blood flow data characteristics.

[0029] In combination with the first aspect, in a possible implementation manner, the loss function used when training the performance evaluation model is expressed as:

[0030] loss = W id ·loss id +W grade ·loss grade ;

[0031] Among them, W id represents the trainable weights to determine the impact of identity token loss on loss; loss id represents the identity token loss, where the identity token is a classification cross loss entropy; W grade Represents the trainable weights to determine the impact of performance regression loss on loss; loss grade Represents the loss function Huber loss for score regression.

[0032] In a second aspect, the present invention provides a learning effect prediction device based on EEG near-infrared dual-modal signals, the device comprising:

[0033] A preprocessing module, used for preprocessing basic data to obtain preprocessed data;

[0034] An evaluation module, used to use the trained performance evaluation model to perform performance evaluation on the pre-processed data to obtain an evaluation score; wherein the evaluation module includes: an EEG data processing branch, an fNIRS data processing branch, an information interaction module and a performance regression module; wherein the EEG data processing branch and the fNIRS data processing branch are independent of each other; the EEG data processing branch and the fNIRS data processing branch interact with the information interaction module;

[0035] The output module is used to make learning suggestions based on the evaluation results and output them.

[0036] In a third aspect, the present invention provides a learning effect prediction server based on EEG near-infrared dual-modal signals, the server comprising a memory and a processor;

[0037] The memory is used to store computer executable instructions;

[0038] The processor is used to execute the computer executable instructions to implement a learning effect prediction method based on EEG near-infrared dual-modal signals.

[0039] One or more technical solutions provided in the present invention have at least the following technical effects or advantages:

[0040] The present invention adopts a trained performance evaluation model and uses the performance evaluation model to replace the signal preprocessing part with higher time cost, thereby improving the real-time performance of the overall method. The performance evaluation model is then used to extract the spatiotemporal spectral features of the EEG signal and the spatiotemporal features of the near-infrared signal, which effectively solves the problems of high time cost and inaccurate prediction, thereby achieving better utilization of the time laws existing during data collection, which helps to improve the prediction accuracy of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A flowchart of the steps of a learning effect prediction method based on EEG near-infrared dual-modal signals provided in an embodiment of the present invention;

[0042] Figure 2 A schematic diagram of data preprocessing provided by an embodiment of the present invention;

[0043] Figure 3 A schematic diagram of the performance evaluation model structure provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] The present invention provides a learning effect prediction method based on EEG near-infrared dual-modal signals, such as Figure 1 As shown, the method includes the following steps S101 to S103.

[0046] S101, acquiring basic data, and preprocessing the basic data to obtain preprocessed data.

[0047] Specifically, in step S101, the basic data is preprocessed to obtain preprocessed data, such as Figure 2 As shown, steps S1011 to S1015 are included.

[0048] S1011, screening and classifying the basic data to obtain EEG sampling data and fNIRS sampling data.

[0049] S1012, re-screening reference points for the EEG sampling data, and re-referencing the EEG sampling data according to the reference points to obtain re-referenced EEG sampling data.

[0050] S1013, using a bandpass filter to perform bandpass filtering on the re-referenced EEG sampling data to obtain pre-processed EEG sampling data.

[0051] S1014, removing motion artifacts from the fNIRS sampling data, and performing band-pass filtering on the fNIRS sampling data from which the motion artifacts have been removed using a band-pass filter to obtain pre-processed fNIRS sampling data.

[0052] S1015, segmenting the preprocessed EEG sampling data and the preprocessed fNIRS sampling data, and removing the baseline of the segmented EEG sampling data and the segmented fNIRS sampling data to obtain preprocessed data.

[0053] Specifically, before preprocessing the basic data, the method further includes: screening the basic data by using a questionnaire survey method.

[0054] Exemplarily, designing an EEG-fNIRS experiment and analyzing experimental data; including designing an fNIRS-EEG experiment, processing and analyzing experimental data, designing pre-test and post-test questionnaires, and constructing a deep model. Among them:

[0055] EEG-fNIRS experimental design: In order to obtain the changes in students' neural blood oxygen in the state of no prior knowledge when learning multidisciplinary rich knowledge, we first selected two subjects from the four major subject types according to the Bigland subject classification to screen the learning videos. Subsequently, we designed a pre-test questionnaire to be completed before the students started the experiment to ensure that the students had no prior knowledge of the subjects to be learned. We edited the learning videos into short videos of about 3 minutes, each video containing several knowledge points, and designed corresponding learning questions. The learning videos and questions were integrated into a program through Psychtoolbox. Students participating in the experiment need to wear EEG near-infrared signal acquisition equipment throughout the process and perform rest, study, and answer questions according to the program requirements. Each experiment includes 4 subjects, a total of 8 video learning contents and 80 questions, and the entire experimental process takes about 45 minutes. The experimental program will record the students' choices for each question, and the EEG near-infrared acquisition device will record the neural blood oxygen data during the experiment.

[0056] EEG experimental data processing: The EEG data obtained from the experiment needs to be preprocessed using Python, including re-referencing, bandpass filtering, data segmentation, and baseline removal. The preprocessing of EEG data often includes ICA, which is extremely time-consuming. Here we remove this step to improve the real-time efficiency of the model. In the following step, we will use the deep learnable preprocessing module instead.

[0057] fNIRS experimental data processing: The near-infrared data obtained from the experiment needs to be preprocessed using Python, including motion artifact removal, bandpass filtering, and baseline removal. Here we also build a deep learnable preprocessing module in the deep model for further preprocessing.

[0058] Contents of pre-test and post-test questionnaires: In order to ensure that the questions are clear and avoid using vague or ambiguous expressions, and also to take into account the learners' time cost, we designed a concise questionnaire. The questionnaire contains basic information of the students, such as age, gender, educational background, etc., and further asks the learners whether they have understood or learned the knowledge related to the course in order to assess their previous knowledge level. Next, we use the key concepts, basic principles, and related topics previously learned in the covered courses to test the learners' understanding of the relevant concepts before they start learning. Based on the questionnaire results, we excluded students who had too high an understanding of the subject to avoid the situation where the grades caused by prior knowledge were inconsistent with the neural blood oxygen signal. The post-test questionnaire covers the knowledge points and their applications in the learning video to examine the students' understanding of the learning video.

[0059] Step S101 is based on the student's own EEG-fNIRS data. Compared with the current learning assessment methods that only rely on simple dimensional online behavioral data or only use neural information to predict grades, this method is more multidimensional and personalized. It can detect students' brain responses, evaluate learning effects, predict learning results, and provide reasonable learning suggestions. In addition, this method can also analyze the responses of brain areas in complex tasks, which is of great significance for a deep understanding of brain mechanisms. The method of using questionnaires to evaluate prior knowledge before the start of the course provides a basis for students to learn new knowledge points. During the learning process, learning effects are predicted through the source of neural blood oxygen data, and phased learning suggestions are put forward. In addition, the present invention can also provide new insights for a deep understanding of brain mechanisms.

[0060] The EEG-fNIRS experimental design was used, which included video materials. Compared with the traditional simple paradigm experiment, this design presents audio and images at the same time, which can better simulate the real learning scene, thus improving the applicability of educational assessment.

[0061] S102, using the trained performance evaluation model to perform performance evaluation on the preprocessed data to obtain an evaluation performance. The performance regression module obtains data from the EEG data processing branch and the fNIRS data processing branch to perform performance evaluation.

[0062] Specifically, in step S102, the performance evaluation model is as follows: Figure 3 As shown, it includes: an EEG data processing branch, an fNIRS data processing branch, an information interaction module and a performance regression module; wherein the EEG data processing branch and the fNIRS data processing branch are independent of each other; the EEG data processing branch and the fNIRS data processing branch interact with the information interaction module.

[0063] The EEG data processing branch includes: a first single-layer convolutional codec, a first convolutional multi-head self-attention layer, and a multi-head self-attention layer calculated after Fourier transformation, which are connected in sequence.

[0064] The fNIRS data processing branch includes: a second single-layer convolutional encoder-decoder, a second convolutional multi-head self-attention layer, and a third convolutional multi-head self-attention layer connected sequentially.

[0065] The information interaction module includes: a first information interaction layer and a second information interaction layer which are independent of each other;

[0066] The score regression module includes: multi-layer perception layer.

[0067] Specifically, in step S102, the functions of each branch point of the performance evaluation model specifically include:

[0068] (1) A first single-layer convolutional codec, used to obtain first reconstructed EEG data based on the preprocessed data.

[0069] (2) A first convolutional multi-head self-attention layer, used to extract the first neural electrical data features in the first reconstructed EEG data.

[0070] (3) A second single-layer convolutional encoder / decoder, used to obtain second reconstructed fNIRS data based on the preprocessed data.

[0071] (4) A second convolutional multi-head self-attention layer, used to extract the first blood flow data features in the second reconstructed fNIRS data.

[0072] (5) A first information interaction layer, which is used to calculate the weight of each feature in the first neural electrical data feature based on the first blood flow data feature and the first neural electrical data feature, and use the weight to weight the first neural electrical data feature to obtain first weighted feature data, and calculate the weight of each feature in the first blood flow data feature, and use the weight to weight the first blood flow data feature to obtain second weighted feature data.

[0073] (6) After Fourier transform, the multi-head self-attention layer is calculated to transform the first weight feature data to obtain the second neural electrical data feature. Here, the multi-head self-attention layer is calculated after Fourier transform to transform the first weight feature data. First, the input data is converted to the frequency domain through Fourier transform. The purpose of this step is to convert the data from the time domain or space domain to the frequency domain in order to better capture the periodicity or frequency characteristics in the data, and then apply the multi-head self-attention mechanism to process the transformed data. The multi-head self-attention mechanism calculates the relationship between different parts of the input data in parallel through multiple attention heads (multiple self-attention sub-layers). The purpose of this step is to capture the important information in the frequency domain features through the mechanism of the multi-head self-attention layer, and combine the results of multiple attention heads to generate the final feature representation.

[0074] (7) A third convolutional multi-head self-attention layer is used to extract features from the second weighted feature data to obtain second blood flow data features.

[0075] (8) A multi-layer perception layer, used to obtain an evaluation score based on the second neural electrical data characteristics and the second blood flow data characteristics.

[0076] Exemplarily, the encoder parts of the first single-layer convolutional codec and the second single-layer convolutional codec are responsible for converting the input data into a representation in a latent space, which contains key features of the input data and can better perform data processing and analysis in the latent space.

[0077] The decoder part of the first single-layer convolutional encoder-decoder and the second single-layer convolutional encoder-decoder is responsible for decoding the encoded representation in the latent space into reconstructed data in the original data space. The decoder is usually also a deep neural network with a structure opposite to that of the encoder. It gradually decodes the encoded representation in the latent space into reconstructed data similar to the original input data through a series of layers. The reconstructed data generated by the decoder has similar structure and characteristics to the original input data, but may be slightly different. These differences are considered to be noise in the original data.

[0078] Due to the different characteristics of the preprocessed data (EEG-fNIRS), different feature extraction modules need to be designed according to their characteristics.

[0079] Since EEG signals have a wide spectrum range, ranging from low frequency (such as delta waves) to high frequency (such as gamma waves), building a deep spectrum extraction module helps to extract effective information from them. The first convolutional multi-head self-attention layer is used to extract time and space domain features from the signal, including the dynamic changes of time series data and the spatial relationship between different channels, to obtain the first neural electrical data feature.

[0080] The spectrum information of near-infrared signals is weaker than that of EEG signals, so there is no need for a deep spectrum extraction module, and only a second convolutional multi-head self-attention layer is needed to capture the temporal and spatial information in the signal. The second convolutional multi-head self-attention layer is used to capture the temporal and spatial information in the signal to obtain the first blood flow data feature.

[0081] In the feature extraction process, the first neural electrical data feature and the first blood flow data feature interact through the first information interaction layer. The specific interaction process includes:

[0082] The first neural electrical data feature and the first blood flow data feature are processed in parallel by their respective neural networks to continuously extract various features. However, data of different modalities have complementary information. EEG signals and near-infrared signals can provide different aspects of brain activity information. The information interaction module can promote the fusion of cross-modal information and make full use of the information of the two modalities to improve the overall feature representation ability.

[0083] Specifically, in step S102, the loss function used in the training performance evaluation model is expressed as:

[0084] loss = W id ·loss id +W grade ·loss grade ;

[0085] Among them, W id represents the trainable weights to determine the impact of identity token loss on loss; lossid represents the identity token loss, where the identity token is a classification cross loss entropy; W grade Represents the trainable weights to determine the impact of performance regression loss on loss; loss grade Represents the loss function Huber loss for score regression.

[0086] Step S102, by improving the real-time performance of data preprocessing, first perform simple preprocessing, then build an encoder and decoder for further preprocessing. Next, select a feature extraction neural network suitable for each type to extract data features. This method can better utilize the time regularity of data collection and help improve the prediction accuracy of the network.

[0087] S103, making learning suggestions based on the evaluation results.

[0088] For example, the behavioral data and personalized portraits are combined for evaluation and prediction: the personalized portrait can intuitively describe the student's learning characteristics, learning experience, etc., and can reflect the student's learning ability to a certain extent. The behavioral data records the student's daily learning trajectory. The academic performance predicted by this part of the data can reflect the student's performance and level of involvement in learning a certain course. If a student's behavioral data predicts a low grade, but his personal portrait shows that he has good learning ability, the student can be reminded to invest more energy in the course.

[0089] Step S103, using the students' neural blood flow information to predict their grades, giving personalized evaluation results, and providing students with appropriate learning suggestions, thereby promoting students to improve their learning enthusiasm, take the initiative to learn, and improve their learning outcomes.

[0090] The present invention also provides a learning effect prediction device based on EEG near-infrared dual-modal signals, which comprises: a preprocessing module, an evaluation module and an output module.

[0091] The preprocessing module is used to preprocess the basic data to obtain preprocessed data.

[0092] The evaluation module is used to use the trained performance evaluation model to perform performance evaluation on the preprocessed data to obtain evaluation results.

[0093] The output module is used to make learning suggestions based on the evaluation results and output the suggestions.

[0094] Some modules in the apparatus of the present invention may be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform specific tasks or implement specific abstract data types. The present invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0095] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary hardware. Based on such an understanding, the technical solution of the present invention can be essentially or partly contributed to the prior art in the form of a software product, or it can be reflected in the implementation process of data migration. The computer software product can be stored in a storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention or certain parts of the embodiments.

[0096] The present invention provides a learning effect prediction server based on EEG near-infrared dual-modal signals, the server comprising a memory and a processor; the memory is used to store computer executable instructions;

[0097] The processor is used for executing computer executable instructions to realize a learning effect prediction method based on EEG near-infrared dual-modal signals.

[0098] Although the present invention provides method operation steps such as embodiments or flow charts, more or fewer operation steps may be included based on conventional or non-creative labor. The order of steps listed in this embodiment is only one way of executing the order of many steps and does not represent the only execution order. When the device or client product in practice is executed, it can be executed in the order of the method shown in this embodiment or the accompanying drawings or in parallel (for example, in a parallel processor or multi-threaded processing environment).

[0099] The devices or modules described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above devices are described in various modules according to their functions. When implementing the present invention, the functions of each module can be implemented in the same or multiple software and / or hardware. Of course, the module that implements a certain function can also be implemented by combining multiple sub-modules or sub-units.

[0100] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the present invention.

Claims

1. A learning effect prediction method based on EEG near-infrared dual-modal signals, characterized in that: include: Acquiring basic data, and preprocessing the basic data to obtain preprocessed data; The pre-processed data is evaluated using the trained performance evaluation model to obtain an evaluation score; wherein the performance evaluation model includes: an EEG data processing branch, an fNIRS data processing branch, an information interaction module and a performance regression module; wherein the EEG data processing branch and the fNIRS data processing branch are independent of each other; the EEG data processing branch and the fNIRS data processing branch interact with the information interaction module; the performance regression module obtains data from the EEG data processing branch and the fNIRS data processing branch to perform performance evaluation; Make study recommendations based on said assessment results.

2. The learning effect prediction method based on EEG near-infrared dual-modal signals according to claim 1 is characterized in that: The preprocessing of the basic data to obtain preprocessed data includes: Screening and classifying the basic data to obtain EEG sampling data and fNIRS sampling data; Re-screening reference points for the EEG sampling data, and re-referencing the EEG sampling data according to the reference points to obtain re-referenced EEG sampling data; Using a bandpass filter to perform bandpass filtering on the re-referenced EEG sampling data to obtain pre-processed EEG sampling data; removing motion artifacts from the fNIRS sampling data, and performing band-pass filtering on the fNIRS sampling data from which the motion artifacts have been removed using a band-pass filter to obtain pre-processed fNIRS sampling data; The preprocessed EEG sampling data and the preprocessed fNIRS sampling data are respectively segmented, and the segmented EEG sampling data and the segmented fNIRS sampling data are respectively baseline-removed to obtain preprocessed data.

3. The learning effect prediction method based on EEG near-infrared dual-modal signals according to claim 1 is characterized in that: Before preprocessing the basic data, the method further includes: screening the basic data by using a questionnaire survey method.

4. The learning effect prediction method based on EEG near-infrared dual-modal signals according to claim 1 is characterized in that: The EEG data processing branch includes: a first single-layer convolutional codec, a first convolutional multi-head self-attention layer, and a Fourier transformed post-calculation multi-head self-attention layer connected in sequence; The fNIRS data processing branch includes: a second single-layer convolutional codec, a second convolutional multi-head self-attention layer, and a third convolutional multi-head self-attention layer connected in sequence; The information interaction module includes: a first information interaction layer and a second information interaction layer that are independent of each other; wherein the first information interaction layer interacts with the EEG data processing branch and the fNIRS data processing branch, and the second information interaction layer interacts with the EEG data processing branch and the fNIRS data processing branch; The performance regression module includes: a multi-layer perception layer.

5. The learning effect prediction method based on EEG near-infrared dual-modal signals according to claim 4 is characterized in that: The first single-layer convolutional codec is used to obtain first reconstructed EEG data based on the preprocessed data; The first convolutional multi-head self-attention layer is used to extract the first neural electrical data feature in the first reconstructed EEG data; The second single-layer convolutional codec is used to obtain second reconstructed fNIRS data according to the preprocessed data; The second convolutional multi-head self-attention layer is used to extract the first blood flow data feature in the second reconstructed fNIRS data; The first information interaction layer is used to calculate the weight of each feature in the first neural electrical data feature according to the first blood flow data feature and the first neural electrical data feature, and use the weight to weight the first neural electrical data feature to obtain first weighted feature data, and calculate the weight of each feature in the first blood flow data feature, and use the weight to weight the first blood flow data feature to obtain second weighted feature data; The multi-head self-attention layer is calculated after the Fourier transform to convert the first weight feature data to obtain a second neural electrical data feature; The third convolutional multi-head self-attention layer is used to extract features from the second weight feature data to obtain second blood flow data features; The multi-layer perception layer is used to obtain an evaluation score based on the second neural electrical data characteristics and the second blood flow data characteristics.

6. The learning effect prediction method based on EEG near-infrared dual-modal signals according to claim 1 is characterized in that: The loss function used when training the performance evaluation model is expressed as: loss=W id ·loss id +W grade ·loss grade ; Among them, W id represents the trainable weights to determine the impact of identity token loss on loss; loss id represents the identity token loss, where the identity token is a classification cross loss entropy; W grade Represents the trainable weights to determine the impact of performance regression loss on loss; loss grade Represents the loss function Huber loss for score regression.

7. A learning effect prediction device based on EEG near-infrared dual-modal signals, characterized in that: include: A preprocessing module, used for preprocessing basic data to obtain preprocessed data; An evaluation module, used to perform performance evaluation on the preprocessed data using the trained performance evaluation model to obtain an evaluation score; The output module is used to make learning suggestions based on the evaluation results and output them.

8. A learning effect prediction server based on EEG near-infrared dual-modal signals, characterized in that: including memory and processor; The memory is used to store computer executable instructions; The processor is used to execute the computer executable instructions to implement the learning effect prediction method based on EEG near-infrared dual-modal signals as described in any one of claims 1-6.

Images