Method and device for identifying bilingual knowledge learning result based on physiological signals

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a method and device for recognizing a bilingual knowledge learning result based on physiological signals. The method comprises the following steps: collecting physiological signal data of a learner; the behavior data acquisition subsystem is used for acquiring behavior data of the learner based on a preset stimulation program on the behavior data acquisition subsystem; stimulation presentation is carried out; electroencephalogram feature data, eye movement feature data and behavior feature data are obtained, the feature data are input into a preset classifier model, and different weights are distributed to different output results; modeling a mapping relation between the physiological signal data and the behavior data through a secondary discriminant analysis algorithm QDA; and based on the feature data and the mapping relation modeling result, evaluating a language knowledge learning result of the learner. According to the invention, the accuracy of language knowledge and ability level testing can be improved, and the adjustment of a student learning plan and a classroom teaching scheme can be scientifically guided.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence, and particularly to a method and device for identifying the learning results of second language knowledge based on physiological signals. Background Art

[0002] In the process of learning a second language (L2), identifying the learning results of knowledge and further evaluating the level of students' language learning ability play a decisive role in adjusting the learning progress and designing a personalized learning plan. The traditional judgment of students' knowledge learning results in second language teaching is mainly led by teachers, who combine the learning status in class and the results of after-class exercises or tests to obtain the evaluation results of learning quality in a specific period. Further, the judgment of students' comprehensive second language learning level or ability is also mainly evaluated through a unified examination.

[0003] Language learning is directly related to brain cognitive activities, which means that the physiological signals reflecting brain activities are closely related to the learning results of language and the ability of second language acquisition. As long as the human brain is not dead, it will continuously generate brain waves, that is, electroencephalogram (EEG) signals. Electroencephalogram (EEG) is an electrical signal graph obtained by amplifying and recording the bioelectric potential of the brain from the scalp through a precise electronic instrument, which is the spontaneous and rhythmic electrical activity performance of brain cell groups recorded by electrodes. By synchronously recording the stimulus sequence during the process of recording the original EEG, and then performing multiple superposition averaging analysis based on time-locking and / or phase-locking on the potential changes caused by multiple similar stimuli in the EEG signal in the brain, the evoked EEG signal of the stimulus can be obtained, forming a data representation of the brain's response to the stimulus. Since the time accuracy of EEG is very high (it can be accurate to milliseconds), it is particularly suitable for observing the time process of the brain in information processing. EEG technology is an objective method to reflect the brain's advanced thinking activities and has been widely used in the field of cognitive function research. An eye tracker and eye movement technology can reveal the distribution of attention of an individual during the information processing process by precisely tracking the movement trajectory of the eyes. For example, the duration of fixation, the path of saccade, the frequency of regression, etc. all reflect the degree of attention of the brain to different information. In the cognitive field, eye movement technology is widely used to study language processing because it can reflect in real time how an individual processes language information in tasks such as reading and listening comprehension. Combining electroencephalogram technology (EEG), eye movement technology can provide more abundant spatio-temporal data to help researchers understand the process of language cognition more deeply. Electroencephalogram captures the bioelectric activity of the brain during language processing, while eye movement data reveals the instantaneous changes in attention. The combination of the two enables us to study the dynamic mechanism of language understanding more comprehensively, especially the real-time response of language learning, which involves complex language processing tasks such as word meaning and semantic processing.

[0004] Machine learning is an important branch of artificial intelligence. It automatically learns laws and patterns from data through calculations to make predictions or decisions, rather than relying on programmers to manually write rules to process data and problems. In machine learning, the system can self-adjust according to the statistical laws in historical data to generate a predictive model. In data analysis, feature classification is a core task, and machine learning has shown significant advantages, especially when dealing with complex signal data. For example, in electroencephalogram (EEG) data analysis, EEG signals that reflect brain activity are usually accompanied by a lot of noise and variation, making traditional manual feature extraction and analysis very difficult. Machine learning algorithms, such as support vector machines (SVM), deep learning, and convolutional neural networks (CNN), can automatically learn to extract useful features from raw EEG data and classify them based on these features. Through machine learning, researchers can transform EEG data into meaningful patterns for a variety of applications such as brain-computer interfaces (BCI), epileptic seizure prediction, and emotion recognition. The power of machine learning algorithms lies in their ability to handle the nonlinearity, temporal sequence, and high dimensionality of data, and automatically identify complex feature combinations, thereby providing higher accuracy and efficiency than traditional methods. In particular, the application of deep learning technology has further promoted the intelligent development of EEG data analysis, making research and application in this field more accurate and efficient.

[0005] The method that teachers still widely use to evaluate learners' language learning and ability by combining classroom performance and specific test questions is to infer their learning results and learning ability based on their behavioral performance. This approach lacks scientific explanations and is accidental. It is difficult to identify the learners' true psychological state during the learning and testing process. Teachers' observations are limited, and the evaluation standards are subjective and inconsistent. The accuracy still needs to be improved. Inappropriate interactions between teachers in the classroom and the presence of examiners in the examination room may even make learners feel uncomfortable and unable to perform at their normal level. In addition, the existing evaluation standards are based on the needs of language application scenarios. Questions and scales are set based on knowledge learning and language application ability in various dimensions of listening, speaking, reading, writing and translation. The establishment of standards and the selection of test questions are all extremely uncertain, which increases evaluation errors.

[0006] In order to evaluate the scientificity and accuracy of the research, language cognition research at home and abroad has gradually turned to research based on brain cognition technology, hoping to find differences in physiological signals that can indicate differences in language learning in the process of brain language cognition. However, due to the complexity of the human brain's language cognition process and the technical difficulty of physiological signal processing, especially the traditional data analysis technology is almost unable to process the physiological signals of the learning process in natural scenes, most of the current research based on brain cognition technology is developing in the direction of abstract experimental stimulation or even directly using artificial language, which loses the intuitive reflection of the real learning process.

[0007] With the development of the information age, great changes have taken place in people's lives. The learning scenario of language learning - especially second language learning - has changed from traditional classroom learning to online video teaching, showing the characteristics of diversification, short time, and fragmentation. The traditional test mode cannot provide the most immediate and crucial inspiration and guidance for language learning plans and teaching adjustments. The current language evaluation still lacks a mode transformation that keeps up with the changes in learning methods. Summary of the Invention

[0008] In order to solve the technical problems that the traditional test mode in the prior art cannot provide the most immediate and crucial inspiration and guidance for language learning plans and teaching adjustments, and the current language evaluation still lacks a mode transformation that keeps up with the changes in learning methods, the embodiments of the present invention provide a method and device for recognizing the learning results of second language knowledge based on physiological signals. The technical solutions are as follows:

[0009] On the one hand, a device for recognizing the learning results of second language knowledge based on physiological signals is provided, characterized in that the device includes:

[0010] A physiological signal data acquisition system for acquiring physiological signal data of learners; wherein, the signal data acquisition includes: an electroencephalogram acquisition subsystem and an eye movement acquisition subsystem;

[0011] A total acquisition control system for acquiring the behavior data of learners based on a stimulation program preset on the behavior data acquisition subsystem; and presenting the stimulation;

[0012] An evaluation system for respectively obtaining electroencephalogram feature data, eye movement feature data, and behavior feature data, inputting the feature data into a preset classifier model respectively, and assigning different weights to different output results; modeling the mapping relationship between physiological signal data and behavior data through the quadratic discriminant analysis algorithm QDA; and evaluating the language knowledge learning results of learners based on the feature data and the modeling results of the mapping relationship.

[0013] Optionally, the electroencephalogram acquisition subsystem includes an electroencephalogram cap, an amplifier, and an electroencephalogram signal acquisition terminal; for acquiring the electroencephalogram data of learners;

[0014] The eye movement acquisition subsystem includes an eye movement camera and an eye movement signal acquisition terminal;

[0015] The electroencephalogram acquisition subsystem, the eye movement acquisition subsystem, and the behavior data acquisition subsystem are under the same local area network.

[0016] Optionally, the electroencephalogram acquisition subsystem includes:

[0017] The EEG cap is for the learner to wear; the EEG cap, amplifier, and EEG signal acquisition terminal are communicatively connected in sequence;

[0018] The EEG signal data is collected through the EEG cap; the amplifier receives the EEG signal data from the EEG cap and the behavior code data issued by the behavior data acquisition subsystem; the EEG signal acquisition terminal receives the behavior code data and the EEG signal data and encodes and records them in a data file.

[0019] Optionally, the eye movement acquisition subsystem includes:

[0020] The movement calibration program in the eye movement acquisition subsystem is used to adapt and adjust the eye movement camera to the eyes of the learner;

[0021] The eye movement camera collects high-precision eye movement signals and transmits the eye movement signals to the acquisition terminal, and the eye movement signal terminal controls the acquisition behavior of the eye tracker.

[0022] Optionally, based on the stimulation program preset on the behavior data acquisition subsystem, the behavior data of the learner is collected; and stimulation presentation is performed, including:

[0023] Based on the stimulation program preset on the behavior data acquisition subsystem, the evaluation process of the learner is timed with high precision to obtain the total time of each link, the reaction time of the learner in the test part, the test score and accuracy rate of the learner;

[0024] And the learning and test content is presented on the terminal screen.

[0025] Optionally, obtaining EEG feature data includes:

[0026] Obtaining EEG signals;

[0027] Preprocessing the EEG signals to obtain EEG signals with noise interference removed from the original EEG signals;

[0028] Performing dimensionality reduction and simplification calculations on the processed EEG signals to extract the feature quantities of the preprocessed EEG signals;

[0029] Classifying the extracted feature quantities x EEG =[x1,x2,…,x p T .

[0030] Optionally, eye movement feature data includes:

[0031] Indicators reflecting eye changes x EM =[x p+1 ,x p+2 ,…,x q ​​T Among them, the eye movement indicators for studying mental workload can be divided into four categories: fixation indicators, saccade indicators, blink indicators, and pupil indicators.

[0032] Optionally, a mapping relationship between physiological signal data and behavioral data is modeled by the quadratic discriminant analysis algorithm QDA, including:

[0033] For a certain category of behavioral label y: y = k, the conditional probability of its feature x satisfies a Gaussian distribution:

[0034]

[0035] where: μ k is the mean vector of category k, and Σ k is the covariance matrix of category k;

[0036] Calculate the discriminant function for each category and select the category with the maximum value; for category k, the discriminant function is:

[0037]

[0038] where: p(y = k) is the prior probability of category k, indicating the occurrence probability of this category in the data; |Σ k | is the determinant of the covariance matrix Σ k , μ k is the mean vector of category k, and Σ k -1 is the inverse of the covariance matrix;

[0039] The input x is assigned to the category with the maximum discriminant function value That is:

[0040]

[0041] On the other hand, a method for recognizing the learning results of second language knowledge based on physiological signals is provided. This method is applied to a device for recognizing the learning results of second language knowledge based on physiological signals, and this method includes:

[0042] S1. Collect physiological signal data of the learner; among them, signal data collection includes: an electroencephalogram collection subsystem and an eye movement collection subsystem;

[0043] S2. Based on the stimulation program preset on the behavioral data collection subsystem, collect the behavioral data of the learner; and perform stimulation presentation;

[0044] S3. Obtain electroencephalogram feature data, eye movement feature data, and behavioral feature data respectively, input the feature data into a preset classifier model respectively, and assign different weights to different output results;

[0045] S4. Model the mapping relationship between physiological signal data and behavioral data through the Quadratic Discriminant Analysis (QDA) algorithm;

[0046] S5. Evaluate the learning results of the learner's language knowledge based on the feature data and the modeling results of the mapping relationship.

[0047] On the other hand, provided is a device for recognizing second language knowledge learning results based on physiological signals. The device for recognizing second language knowledge learning results based on physiological signals includes: a processor; a memory, on which computer-readable instructions are stored. When the computer-readable instructions are executed by the processor, any one of the methods in the above method for recognizing second language knowledge learning results based on physiological signals is implemented.

[0048] On the other hand, provided is a computer-readable storage medium, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement any one of the methods in the above method for recognizing second language knowledge learning results based on physiological signals.

[0049] The beneficial effects brought by the technical solutions provided in the embodiments of the present invention at least include:

[0050] In the embodiments of the present invention, for the model that evaluates the language level and knowledge learning results of learners based on short-term second language learning physiological signals, through artificial intelligence technology, analyze the real reactions in the short-term learning scenarios of the videos synchronously recorded by the physiological signal acquisition device, identify the learning status and results of students, and then assist in adjusting the second language learning plan and designing personalized teaching plans. The video online learning scenario conforms to the current mainstream second language learning scenario, and can also avoid the mutual emotional influence between the teaching teacher and students, and the examination room examiner and candidates, which may lead to deviation of the test results. The acquisition process is almost imperceptible, which can reduce the tension and anxiety of users and truly reflect the real level and performance of learners. Description of the Drawings

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0052] Figure 1 It is a block diagram of the device for recognizing second language knowledge learning results based on physiological signals provided in the embodiments of the present invention;

[0053] Figure 2 It is a schematic flowchart of the method for recognizing second language knowledge learning results based on physiological signals provided in the embodiments of the present invention;

[0054] Figure 3 Schematic structural diagram of the electronic device provided by the embodiment of the present invention. Detailed implementation manners

[0055] The technical solutions in the present invention will be described below with reference to the accompanying drawings.

[0056] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to represent examples, illustrations or explanations. Any embodiment or design solution described as an "example" in the present invention should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Exactly speaking, the use of the word "example" is intended to present concepts in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one of the two.

[0057] In the embodiments of the present invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0058] To make the technical problems, technical solutions and advantages to be solved by the present invention clearer, the following will be described in detail with reference to the accompanying drawings and specific embodiments.

[0059] Figure 1 It is a block diagram of a device for recognizing the learning result of second language knowledge based on physiological signals shown according to an exemplary embodiment. The device 100 is used for a method of recognizing the learning result of second language knowledge based on physiological signals. Refer to Figure 1 , the device includes a physiological signal data acquisition system 110, an overall acquisition control system 120, and an evaluation system 130. Among them:

[0060] The physiological signal data acquisition system 110 is used to acquire physiological signal data of learners; among them, the signal data acquisition includes: an electroencephalogram acquisition subsystem 111 and an eye movement acquisition subsystem 112;

[0061] The overall acquisition control system 120 is used to acquire the behavior data of learners based on a stimulation program preset on the behavior data acquisition subsystem; and perform stimulation presentation;

[0062] The evaluation system 130 is used to respectively obtain electroencephalogram feature data, eye movement feature data, and behavior feature data, input the feature data into a preset classifier model respectively, and assign different weights to different output results; model the mapping relationship between physiological signal data and behavior data through the quadratic discriminant analysis algorithm QDA; and evaluate the language knowledge learning result of learners based on the feature data and the modeling result of the mapping relationship.

[0063] In a feasible implementation manner, the object of the present invention is to keep up with the transformation of the popular methods of second language learning in the information age, use artificial intelligence technology to analyze electroencephalogram (EEG) and eye movement signals that are closely related to the brain's language cognitive process, establish a new evaluation model for identifying the learning results of second language learners' knowledge and the level of learners' language learning ability, explore scientific indicators indicating human brain language cognition, improve the accuracy of discriminating the knowledge learning results of learners after language learning and evaluating learning ability, and further provide guidance for the adjustment of personal learning plans and the design of personalized teaching plans.

[0064] In the embodiments of the present invention, the present invention focuses on the language learning scenario, and emphasizes identifying the learning results of a certain language knowledge or the cognitive state (or acquisition state) of a certain morpheme, which determines the features extracted by the data processing part - the EEG focuses on the power features that centrally represent cognitive activities, the eye movement reflects the attention distribution, and the behavioral data represents the learning effect of knowledge, such as whether the learning is correct, and the degree of certainty of the learner's own mastery of knowledge after learning.

[0065] The data acquisition scheme of the present invention effectively limits the irrelevant influences in the EEG and the specific research scenario, and whether it is reasonable more directly determines whether the recognition results of the algorithm model part are scientific and effective.

[0066] The simulation design of the present invention is aimed at the online short video learning scenario. The key points are short time and the environment is mainly the learner himself, without being interfered by communication, and can play the real level; and in order to make up for the defect of the small amount of data collected in short-time data acquisition, efforts are made in the richness of data acquisition, using multi-modal data, combining EEG, eye movement, and behavioral data, and giving comprehensive and accurate results for knowledge learning judgment.

[0067] In a feasible implementation manner, the experiment adopts the natural paradigm to simulate the real online learning scenario. The test scheme is as follows: Select a language that the learner has never learned as the learning object, learn two introductory courses, and the interval between the two learnings is one week to reduce the deviation of the language learning ability performance caused by the strength of the learner's memory ability. The learning adopts the video teaching method. During the process of the experiment, keep only the learner himself in the environment to avoid environmental factors interfering with the test and let the learner play the most real learning level. The test questions are set comprehensively in multiple aspects of listening, speaking, reading, and writing, including various forms such as pure text, pure audio, pure pictures, text + audio, text + pictures, etc., and collect the stimulus responses in the learning process in all-round way. The test questions are set comprehensively in multiple aspects of listening, speaking, reading, and writing, including various forms such as pure text, pure audio, pure pictures, text + audio, text + pictures, etc., and collect the stimulus responses in the learning process in all-round way.

[0068] In a feasible implementation manner, the EEG acquisition subsystem 111 includes an EEG cap, an amplifier, and an EEG signal acquisition terminal; it is used to acquire the EEG data of the learner;

[0069] The eye movement acquisition subsystem 112 includes an eye movement camera and an eye movement signal acquisition terminal;

[0070] The electroencephalogram (EEG) acquisition subsystem 111, the eye movement acquisition subsystem 112, and the behavior data acquisition subsystem are on the same local area network.

[0071] In a feasible implementation, the EEG acquisition subsystem 111 includes:

[0072] The EEG cap is for the learner to wear; the EEG cap, the amplifier, and the EEG signal acquisition terminal are communicatively connected in sequence;

[0073] The EEG signal data is collected through the EEG cap; the amplifier receives the EEG signal data from the EEG cap and the behavior code data issued by the behavior data acquisition subsystem; the EEG signal acquisition terminal receives the behavior code data and the EEG signal data and encodes and records them into a data file.

[0074] In a feasible implementation, the eye movement acquisition subsystem 112 includes:

[0075] Through the motion calibration program in the eye movement acquisition subsystem 112, the eye movement camera and the eye conditions of the learner are adaptively adjusted;

[0076] The eye movement camera collects high-precision eye movement signals and transmits the eye movement signals to the acquisition terminal, and the eye movement signal terminal controls the acquisition behavior of the eye tracker.

[0077] In a feasible implementation, based on the stimulation program preset on the behavior data acquisition subsystem, the behavior data of the learner is collected; and stimulation presentation is performed, including:

[0078] Based on the stimulation program preset on the behavior data acquisition subsystem, the evaluation process of the learner is accurately timed, and the total time of each link, the reaction time of the learner in the test part, the test score and accuracy rate of the learner are obtained;

[0079] And the learning and test content is presented on the terminal screen.

[0080] In a feasible implementation, the total acquisition control system 120 is responsible for stimulation presentation and accurate timing, and synchronizes EEG, eye movement, and behavior data. The three parts are connected through a switch on the same local area network to achieve message transmission and event synchronization among the three.

[0081] In a feasible implementation, the collection of physiological data during the learning process includes the following steps:

[0082] 1. Debugging of software and hardware devices before the learning experiment

[0083] (1) Device synchronization: The test host (the main tester) connects the EEG acquisition device, the eye movement camera, and the behavioral data acquisition host under the same local area network to ensure that they can communicate with each other;

[0084] (2) EEG device operation test: The main tester wears an EEG cap for the learner and ensures the comfort of the subject. Then connect the EEG cap to the amplifier and the amplifier to the EEG acquisition terminal, and turn on the acquisition software of the EEG acquisition terminal. After the acquisition software is set up, check whether the EEG cap and the corresponding electrodes can be used normally and collect stable EEG signals;

[0085] (3) Eye movement camera debugging: The main tester adjusts the sitting posture of the subject and the auxiliary bracket to adapt to the subject's height, making the subject's line of sight level with the center of the screen. Turn on the eye movement control system and run the camera calibration program to make the camera adapt to the subject's eye conditions;

[0086] (4) Explain the relevant precautions to the subject. After the subject makes a little preparation, start the test.

[0087] 2. Complete each test link in sequence

[0088] (1) The main tester turns on the corresponding stimulation program on the behavioral data acquisition terminal, and the subject only needs to operate according to the test requirements presented in the stimulation program, such as pressing keys, watching, or memorizing, until the end of the entire test process. Here, it is agreed with the subjects participating in the experiment that during the test process, answer according to the real situation. When the answer is uncertain, you can directly skip this question to avoid generating blindly answered data.

[0089] (2) After the subject's learning and testing are both completed, check whether the data of the three terminals are collected completely, turn off the equipment, remove the EEG cap from the subject, ask the subject about the learning situation, and make a record.

[0090] In a feasible implementation manner, the mapping relationship modeling between physiological signals and behaviors is completed through machine learning algorithms. After preprocessing, the EEG segments of the test part are extracted from the EEG signals, and the eye movement time periods during the test are aligned in time. Eye movement analysis and EEG analysis are performed within each time period. Among them, alignment includes: collecting the data of the entire experimental test process, segmenting it during the preprocessing stage, that is, slicing by time, to ensure that the time window of each segment of eye movement data is the same as that of the EEG.

[0091] In a feasible implementation manner, EEG analysis is to directly record the spontaneous EEG signals of the operator during the execution of the memory task, and then process and analyze them to evaluate the cognitive state of the brain. Obtain EEG feature data, including:

[0092] Obtain EEG signals;

[0093] Preprocess the electroencephalogram (EEG) signal to obtain an EEG signal with noise interference removed from the original EEG signal;

[0094] Perform dimensionality reduction and simplification calculations on the processed EEG signal to extract the characteristic quantities of the preprocessed EEG signal;

[0095] Classify the extracted characteristic quantities x EEG =[x1,x2,…,x p T .

[0096] In a feasible implementation manner, during specific analysis, the EEG signal mainly undergoes time-domain analysis and frequency-domain analysis. Time-domain analysis mainly identifies the waveform by analyzing the geometric characteristics of the EEG waveform, and common indicators such as amplitude, maximum amplitude, median amplitude, standard deviation, variance, kurtosis, etc. are used. Power spectrum estimation is an important algorithm in the frequency-domain analysis of EEG signals. The power of the EEG signal is a relationship graph between brain power and time, from which the distribution of each rhythm (α, β, δ, θ, etc.) in the EEG signal can be directly observed, including the total energy value of each frequency band of Total Power, the percentage of the energy value of this band in the total energy value of all bands of Power Percent, and the average energy value of this band of Average Power.

[0097] In a feasible implementation manner, the eye movement characteristic data includes:

[0098] Eye movement analysis is to analyze the eye movement data of the operator during the test. The eye movement indicators specifically refer to the indicators reflecting the changes in the eyeball x EM =[x p+1 ,x p+2 ,…,x q T ; Among them, the eye movement indicators used to study mental workload can be divided into four categories: fixation indicators, saccade indicators, blink indicators, and pupil indicators. Among them, fixation time, fixation count, saccade frequency, and pupil size indicators are all the most effective mental workload measurement indicators.

[0099] Among them, the input feature is a total vector, including two major categories: eye movement and EEG. 1 to p are p EEG features, and from p + 1 to p + q are q eye movement features, that is, X EEG is a p*1 vector, and X EM is a q*1-dimensional vector.

[0100] In a feasible implementation manner, a batch of randomly selected learners can be first subjected to learning tests, and the correct / wrong results of the test questions of this batch of learners and the language ability results X of traditional subjective evaluation indicators behavio ​​As a label, physiological data during the test process is input into a machine learning model for model training to obtain a pre-trained model. Then, the newly acquired physiological data of the learner is input into the pre-trained model. Based on the trained model, the evaluation result of the learner can be obtained according to the physiological data.

[0101] In a feasible implementation, QDA (Quadratic Discriminant Analysis) algorithm is selected for machine learning to complete the modeling between physiological signals and behaviors. The quadratic discriminant analysis algorithm is a classification method based on Bayesian theory, used to establish a mapping model between different categories (behavior labels in this example). QDA assumes that the features of each category follow a Gaussian distribution, but each category has an independent covariance matrix. The goal of the model is to calculate the discriminant function for each category, and then classify the input signal according to these functions. The collected physiological signals include electroencephalogram signal features (such as total power, power spectral entropy, power in each frequency band) x EEG =[x1,x2,…,x p T and eye movement signal features (such as fixation, saccade, blink, and pupil metrics) x EM =[x p+1 ,x p+2 ,…,x q T as input features, where p and q are the dimensions of the features. The QDA model first extracts these features and combines them into a vector x = [x EEG ,x EM , and then classifies them through the discriminant function according to these features, and further predicts the behavior label y (correct / wrong results of test questions, language ability results of traditional subjective evaluation indicators, etc.) of the learner.

[0102] In a feasible implementation, this application involves and screens machine learning algorithms, and selects the QDA algorithm through experimental conclusions:

[0103] 1. Since QDA has the feature of dimensionality reduction, and physiological signals cannot be collected on a large scale, and the short-term learning scenario targeted by the present invention further limits the amount of data that can be collected. Moreover, the data collected by both high-density electroencephalogram acquisition devices and close eye trackers itself contains many-dimensional features, which leads to the potential problem of dimensionality disaster. Through experimental calculation, it is found that the dimensionality reduction feature of QDA well matches the high feature dimensionality of the data:

[0104]

[0105]

[0106] ​​2. In principle, QDA is based on the central limit theorem. Assuming that all feature distributions satisfy the Gaussian distribution, EEG and eye movement both have short-term temporal correlations and long-term irrelevance, which is more in line with the event process of the central limit theorem.

[0107] In a feasible implementation, a mapping relationship between physiological signal data and behavioral data is modeled through the quadratic discriminant analysis algorithm QDA, including:

[0108] For a certain class of behavior label y: y = k, the conditional probability of its feature x satisfies the Gaussian distribution:

[0109]

[0110] where: μ k is the mean vector of class k, and Σ k is the covariance matrix of class k;

[0111] Calculate the discriminant function for each class and select the class with the maximum value; for class k, the discriminant function is:

[0112]

[0113] where: p(y = k) is the prior probability of class k, indicating the occurrence probability of this class in the data, which can usually be estimated by the class conditional frequency of the training data; |Σ k | is the determinant of the covariance matrix Σ k , μ k is the mean vector of class k, and Σ k -1 is the inverse of the covariance matrix;

[0114] The input x is assigned to the class with the maximum discriminant function value That is:

[0115]

[0116] In the embodiments of the present invention, a model for evaluating the language level and knowledge learning results of learners based on short-term second language learning physiological signals is proposed. By analyzing the real reactions in the short-term learning scenario of the video synchronously recorded by the physiological signal acquisition device through artificial intelligence technology, the learning status and results of students are identified, and then the adjustment of the second language learning plan and the design of personalized teaching plans are assisted. The video online learning scenario conforms to the current mainstream second language learning scenario, and can also avoid the mutual emotional influence between the teaching teacher and students, and the examination room examiner and candidates, resulting in the deviation of the test results. The acquisition process is almost imperceptible, which can reduce the tension and anxiety of users and more truly reflect the real level and performance of learners.

[0117] The present invention can establish a correlation model between physiological signals and second language learning cognitive outcomes based on the analysis of electroencephalogram (EEG) and eye movement signals during two video-based second language learning and testing processes spaced one week apart. According to this model, learners with different abilities will generate different EEG and eye movement signal responses when in different learning states. Representative features are extracted from the physiological data to determine the cognitive state of language learners after learning, improve the accuracy of language knowledge and ability level tests, and then scientifically guide the adjustment of students' learning plans and classroom teaching programs.

[0118] An embodiment of the present invention provides a method for identifying second language knowledge learning outcomes based on physiological signals. This method can be implemented by a device for identifying second language knowledge learning outcomes based on physiological signals, which can be a terminal or a server. As Figure 2 shown in the flowchart of the method for identifying second language knowledge learning outcomes based on physiological signals, as Figure 2 shown, the method for identifying second language knowledge learning outcomes proposed by the present invention, the processing flow of this method can include the following steps:

[0119] S1. Collect physiological signal data of the learner; among them, signal data collection includes: electroencephalogram (EEG) signal data collection and eye movement signal data collection;

[0120] S2. Based on the stimulation program preset on the behavior data collection subsystem, collect the behavior data of the learner; and perform stimulation presentation;

[0121] S3. Obtain electroencephalogram (EEG) feature data, eye movement feature data, and behavior feature data respectively, input the feature data into a preset classifier model respectively, and assign different weights to different output results;

[0122] S4. Model the mapping relationship between physiological signal data and behavior data through the quadratic discriminant analysis (QDA) algorithm;

[0123] S5. Evaluate the second language knowledge learning outcomes of the learner based on the feature data and the modeling results of the mapping relationship.

[0124] Optionally, collecting physiological signal data of the learner includes: collecting electroencephalogram (EEG) data of the learner and collecting eye movement data of the learner;

[0125] Optionally, collecting electroencephalogram (EEG) data of the learner includes:

[0126] Collect electroencephalogram (EEG) signal data through an EEG cap; an amplifier receives the electroencephalogram (EEG) signal data of the EEG cap and the behavior code data issued by the behavior data collection subsystem; the electroencephalogram (EEG) signal acquisition terminal receives the behavior code data and the electroencephalogram (EEG) signal data, and encodes and records them into a data file.

[0127] Optionally, collect the eye movement data of the learner, including:

[0128] Through the motion calibration program in the eye movement acquisition subsystem, adapt and adjust the eye movement camera to the eye conditions of the learner;

[0129] Collect high-precision eye movement signals through the eye movement camera, transmit the eye movement signals to the acquisition terminal, and the eye movement signal terminal controls the acquisition behavior of the eye tracker.

[0130] Optionally, based on the stimulation program preset on the behavior data acquisition subsystem, collect the behavior data of the learner; and perform stimulation presentation, including:

[0131] Based on the stimulation program preset on the behavior data acquisition subsystem, perform high-precision timing on the evaluation process of the learner, obtain the total time of each link, the reaction time of the learner in the test part, as well as the test score and accuracy rate of the learner;

[0132] And present the learning and test content on the terminal screen.

[0133] Optionally, obtain electroencephalogram feature data, including:

[0134] Obtain electroencephalogram (EEG) signals;

[0135] Perform preprocessing on the electroencephalogram (EEG) signals to obtain electroencephalogram (EEG) signals with noise interference removed from the original electroencephalogram signals;

[0136] Perform dimensionality reduction and simplification calculations on the processed electroencephalogram (EEG) signals to extract the feature quantities of the preprocessed electroencephalogram (EEG) signals;

[0137] Classify the extracted feature quantities x EEG =[x1,x2,…,x p T .

[0138] Optionally, eye movement feature data, including:

[0139] Indicators x reflecting eye changes EM =[x p+1 ,x p+2 ,…,x q T ; Among them, the eye movement indicators for studying mental workload can be divided into four categories: fixation indicators, saccade indicators, blink indicators, and pupil indicators.

[0140] Optionally, model the mapping relationship between physiological signal data and behavior data through the quadratic discriminant analysis algorithm QDA, including:

[0141] For a certain category of behavior label y: y = k, the conditional probability of its feature x satisfies the Gaussian distribution:​​

[0142]

[0143] where: μ k is the mean vector of class k, and Σ k is the covariance matrix of class k;

[0144] Calculate the discriminant function for each class and select the class with the maximum value; for class k, the discriminant function is:

[0145]

[0146] where: p(y = k) is the prior probability of class k, representing the occurrence probability of this class in the data; |Σ k | is the determinant of the covariance matrix Σ k , μ k is the mean vector of class k, and Σ k -1 is the inverse of the covariance matrix;

[0147] The input x is assigned to the class with the maximum discriminant function value That is:

[0148]

[0149] In an embodiment of the present invention, a model for evaluating the language level and knowledge learning results of learners based on short-term second language learning physiological signals is provided. By analyzing the real reactions in the short-term learning scenario of the video synchronously recorded by the physiological signal acquisition device through artificial intelligence technology, the learning status and results of students are identified, and then the adjustment of the second language learning plan and the design of personalized teaching plans are assisted. The video online learning scenario conforms to the current mainstream second language learning scenario, and can also avoid the mutual emotional influence between the teaching teacher and students, and the examination room examiner and candidates, resulting in the deviation of the test results. The acquisition process is almost imperceptible, which can reduce the tension and anxiety of users and relatively truly reflect the real level and performance of learners.

[0150] Figure 3 is a schematic structural diagram of a device for identifying second language knowledge learning results based on physiological signals provided by an embodiment of the present invention. As Figure 3 shown, the device for identifying second language knowledge learning results based on physiological signals may include the above-mentioned Figure 1 device for identifying second language knowledge learning results based on physiological signals shown. Optionally, the device 410 for identifying second language knowledge learning results based on physiological signals may include a first processor 2001.

[0151] Optionally, the device 410 for identifying second language knowledge learning results based on physiological signals may further include a memory 2002 and a transceiver 2003.

[0152] Among them, the first processor 2001, the memory 2002, and the transceiver 2003 can be connected through a communication bus, for example.

[0153] The following combines Figure 3 to specifically introduce each component of the device 410 for recognizing the learning results of second language knowledge based on physiological signals:

[0154] Among them, the first processor 2001 is the control center of the device 410 for recognizing the learning results of second language knowledge based on physiological signals, and can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or can be an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, such as: one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs).

[0155] Optionally, the first processor 2001 can execute various functions of the device 410 for recognizing the learning results of second language knowledge based on physiological signals by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0156] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 3 the CPU0 and CPU1 shown in

[0157] In a specific implementation, as an embodiment, the device 410 for recognizing the learning results of second language knowledge based on physiological signals may also include multiple processors, such as Figure 3 the first processor 2001 and the second processor 2004 shown in

[0158] Among them, the memory 2002 is used to store software programs for implementing the solution of the present invention and is controlled by the first processor 2001 for execution. The specific implementation manner can refer to the above method embodiments and will not be elaborated here.

[0159] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or may be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM), or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, or any other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently and be coupled to the first processor 2001 through an interface circuit ( Figure 3 not shown) of the device 410 for recognizing the learning results of second language knowledge based on physiological signals. The embodiments of the present invention do not make specific limitations on this.

[0160] The transceiver 2003 is used for communicating with a network device or communicating with a terminal device.

[0161] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 3 not separately shown). Among them, the receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0162] Optionally, the transceiver 2003 may be integrated with the first processor 2001 or may exist independently and be coupled to the first processor 2001 through an interface circuit ( Figure 3 not shown) of the device 410 for recognizing the learning results of second language knowledge based on physiological signals. The embodiments of the present invention do not make specific limitations on this.

[0163] It should be noted that Figure 3 the structure of the device 410 for recognizing the learning results of second language knowledge based on physiological signals shown does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0164] In addition, the technical effects of the device 410 for recognizing the learning results of second language knowledge based on physiological signals may refer to the technical effects of the method for recognizing the learning results of second language knowledge based on physiological signals described in the above method embodiments, and will not be elaborated here.

[0165] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0166] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM) or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM) and direct rambus RAM (DR RAM).

[0167] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware, or any combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable sensors. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that contains one or more collections of available media. The available media can be magnetic media (such as floppy disks, hard disks, magnetic tapes), optical media (such as DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0168] It should be understood that the term "and / or" in this document is merely a description of the association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. Here, A and B can be singular or plural. In addition, the character " / " in this document generally represents an "or" relationship between the associated objects before and after, but it may also represent an "and / or" relationship, which can be specifically understood with reference to the context.

[0169] It should be understood that in various embodiments of the present invention, the magnitudes of the serial numbers of the above processes do not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0170] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed in this document can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0171] The unit described as a separation component may or may not be physically separated. The component shown as a unit may or may not be a physical unit, that is, it may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0172] In addition, each functional unit in various embodiments of the present invention may be integrated into a processing unit, may exist physically separately for each unit, or two or more units may be integrated into one unit.

[0173] If the above function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention.

[0174] As described above, only the specific implementation manners of the present invention are provided, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A device for identifying second language knowledge learning results based on physiological signals, characterized in that: The device comprises: The physiological signal data acquisition system is used to collect physiological signal data of learners; wherein the physiological signal data acquisition system includes: an electroencephalogram acquisition subsystem and an eye movement acquisition subsystem; The overall collection control system is used to collect the learner's behavior data and present the stimulation based on the stimulation program preset on the behavior data collection subsystem; The evaluation system is used to obtain EEG feature data, eye movement feature data and behavioral feature data respectively, input the feature data into a preset classifier model respectively, and assign different weights to different output results; model the mapping relationship between physiological signal data and behavioral data through a quadratic discriminant analysis algorithm QDA; and evaluate the learner's language knowledge learning results based on the feature data and the mapping relationship modeling results.

2. The method for identifying second language knowledge learning results based on physiological signals according to claim 1, characterized in that: The EEG acquisition subsystem includes an EEG cap, an amplifier and an EEG signal acquisition terminal; Used to collect learners’ EEG data; The eye movement acquisition subsystem includes an eye movement camera and an eye movement signal acquisition terminal; The EEG acquisition subsystem, eye movement acquisition subsystem and behavior data acquisition subsystem are in the same local area network.

3. The method for identifying second language knowledge learning results based on physiological signals according to claim 2, characterized in that: The EEG acquisition subsystem comprises: The EEG cap is used to be worn by the learner; the EEG cap, the amplifier and the EEG signal acquisition terminal are sequentially connected for communication; The EEG cap collects EEG signal data; the amplifier receives the EEG signal data from the EEG cap and the behavior code data issued by the behavior data collection subsystem; the EEG signal collection terminal receives the behavior code data and EEG signal data, and encodes and records them into a data file.

4. The method for identifying second language knowledge learning results based on physiological signals according to claim 3, characterized in that: The eye movement acquisition subsystem comprises: Adapting the eye-movement camera to the learner's eye condition through a motion calibration program in the eye-movement acquisition subsystem; The eye movement camera collects high-precision eye movement signals and transmits them to the acquisition terminal, which controls the acquisition behavior of the eye tracker.

5. The method for identifying second language knowledge learning results based on physiological signals according to claim 4, characterized in that: The behavior data of the learner is collected based on the stimulation program preset on the behavior data collection subsystem; And conduct stimulus presentation, including: Based on the preset stimulation program on the behavior data acquisition subsystem, the learner's evaluation process is timed with high precision to obtain the total time of each link, the learner's reaction time in the test part, and the learner's test score and accuracy; And present the learning and testing content on the terminal screen.

6. The method for identifying second language knowledge learning results based on physiological signals according to claim 5, characterized in that: The step of obtaining EEG characteristic data includes: Obtain EEG signals; Preprocessing the EEG signal to obtain an EEG signal that removes noise interference from the original EEG signal; Perform dimensionality reduction and simplification calculation on the processed EEG signal to extract the feature quantity of the preprocessed EEG signal; Classify the extracted features x EEG =[x1,x2,…,x p ] T , where T represents transpose.

7. The method for identifying second language knowledge learning results based on physiological signals according to claim 6, characterized in that: The eye movement feature data includes: Indicators reflecting eye changes EM =[x p+1 ,x p+2 ,…,x q ] T ; Among them, the eye movement indicators used to study psychological load can be divided into four categories: gaze indicators, scan indicators, blink indicators and pupil indicators.

8. The method for identifying second language knowledge learning results based on physiological signals according to claim 7, characterized in that: The mapping relationship between physiological signal data and behavioral data is modeled by using a quadratic discriminant analysis algorithm QDA, including: For a category of a behavior label y: y = k, the conditional probability of its feature x satisfies the Gaussian distribution: Where: μ k is the mean vector of class k, Σ k is the covariance matrix of class k; Calculate the discriminant function for each category and select the category with the maximum value; for category k, the discriminant function is: Where: p(y=k) is the prior probability of category k, indicating the probability of occurrence of this category in the data; |Σ k ∣ is the covariance matrix Σ k The determinant of μ k is the mean vector of class k, Σ k -1 is the inverse of the covariance matrix; The input x is assigned to the class with the largest discriminant function value. Right now:

9. A method for identifying second language knowledge learning results based on physiological signals, wherein the method for identifying second language knowledge learning results based on physiological signals is used to implement the device for identifying second language knowledge learning results based on physiological signals as claimed in any one of claims 1 to 8, characterized in that: The method comprises: S1. Collect physiological signal data from learners; wherein the signal data collection includes: EEG signal data collection and eye movement signal data collection; S2, based on the stimulation program preset on the behavior data collection subsystem, collect the behavior data of the learner; and present the stimulation; S3, respectively obtaining EEG feature data, eye movement feature data and behavioral feature data, respectively inputting the feature data into a preset classifier model, and assigning different weights to different output results; S4, modeling the mapping relationship between physiological signal data and behavioral data through the quadratic discriminant analysis algorithm QDA; S5. Evaluate the learner's language knowledge learning results based on the feature data and the mapping relationship modeling results.

10. A device for identifying second language knowledge learning results based on physiological signals, the device for identifying second language knowledge learning results based on physiological signals comprising: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, any one of the methods for identifying second language knowledge learning results based on physiological signals as recited in claim 9 is implemented.

Images