Multimodal learning level mining method, system and medium under small sample conditions

Abstract

The present invention belongs to the field of personalized learning technology, and discloses a multimodal learning level mining method, system and medium under small sample conditions. The method includes: extracting the content features, attribute features and test performance features of students, introducing BERT and VGG‑16 pre-training models to represent the image and text features, and fusing the image and text features based on tensors to obtain heterogeneous fusion semantic features of test questions; combining the heterogeneous fusion semantic features of test questions with the test performance features of students, estimating the learning state vector based on a bidirectional recurrent neural network and the attention weights between questions; using the heterogeneous fusion semantic features of test questions and the historical weighted learning state vector to predict the test performance of students and analyze and predict the student knowledge mastery matrix. The method disclosed in the present invention is conducive to improving the model's prediction accuracy performance in student test performance and learning mastery, optimizing the convergence rate and training rate of the model, and helping the implementation of smart education from a technical level.

Description

Technical Field

[0001] The present invention belongs to the field of personalized learning technology, and in particular relates to a multimodal learning level mining method, system and medium under small sample conditions. Background Art

[0002] With the development of technologies like artificial intelligence and cloud computing, smart education is becoming a new paradigm for educational informatization in a modern technological environment, providing an opportunity to achieve "teaching students in accordance with their aptitude." "China's Education Modernization 2035" emphasizes the importance of integrating large-scale education with personalized development as a key strategic task. Therefore, addressing learners' personalized learning and development needs has become a new topic in the new generation of education. Because students' learning levels are influenced by their current learning stage and evolve over time, dynamically tracking changes in their knowledge status is crucial for delivering personalized online education.

[0003] The learning proficiency mining task aims to model the time-series evolution of a student's knowledge state based on their past learning behaviors in a specific learning task, and to predict their performance on the next test. Students continuously leave their personalized learning trajectories while using online learning platforms. Learning proficiency mining methods can be used to analyze their learning progress and even leverage existing learning data to predict their performance on other unanswered test questions using a small sample size. This provides students and teachers with more personalized and reliable teaching plans.

[0004] Currently, the more mainstream learning level mining methods can be technically divided into three categories: probability-based learning level mining methods, logical function-based learning level mining methods, and deep learning-based learning level mining methods. Among them, the first two learning level mining methods are represented by Bayesian-based learning level mining and performance factor analysis-based learning level mining methods, respectively. Their prediction results need to be improved in most data sets. Deep learning-based learning level mining methods, on the other hand, are based on deep learning algorithms and introduce recurrent neural networks to dynamically track state changes, thereby achieving more accurate results. However, because deep learning-based learning level mining methods only model a single knowledge point of the test question, they ignore other content features of the test question and the influence of attention. Their modeling of students' knowledge mastery status is also incomplete. Although the stability and accuracy of their prediction results are better than those of the other two learning level mining methods, they still have certain problems.

[0005] Since the learning level mining method based on deep learning performs better than the other two types of learning level mining methods in terms of accuracy when processing most problems, the following discussion of the existing technology focuses on analyzing the learning level mining method based on deep learning, and calls it the traditional learning level mining method.

[0006] Through the above analysis, the problems and defects of the existing technology are as follows:

[0007] (1) Traditional learning level mining methods only model questions from the perspective of knowledge concepts, but ignore the semantic information contained in the questions from text and images, and their performance still needs to be improved;

[0008] (2) Traditional learning level mining methods ignore the impact of history questions on the construction of current learning status, and the modeling of students' knowledge mastery status is rough and lacks explanatory power;

[0009] (3) The deep algorithms in traditional learning level mining methods rely on a large amount of training data, but it is difficult to achieve accurate prediction under small sample conditions. Summary of the Invention

[0010] In response to the problems existing in the prior art, the present invention provides a multimodal learning level mining method, system and medium under small sample conditions.

[0011] The present invention is implemented as follows: a multimodal learning level mining method under small sample conditions, the multimodal learning level mining method under small sample conditions comprising:

[0012] The content features, attribute features of the test questions and the test performance features of the students are extracted, and the BERT and VGG-16 pre-trained models are introduced to represent the semantic features of the test text and image. The image and text features are fused based on the tensor fusion method to obtain the heterogeneous fusion semantic features of the test questions; the heterogeneous fusion semantic features of the test questions are combined with the test performance features of the students, and the learning state vector is estimated based on a bidirectional recurrent neural network and the attention weights between questions; the student's test performance is predicted using the heterogeneous fusion semantic features of the test questions and the historical weighted learning state vector, and the training parameters are used to analyze and predict the student's knowledge mastery matrix.

[0013] Furthermore, the content features, attribute features, and test performance features of the test questions are arranged in chronological order as model input; the content features of the test questions include the text semantic features and image semantic features of the test questions, the attribute features are the knowledge points tested by the test questions, and the test performance features are the score vectors corresponding to the students' answers to the test questions;

[0014] Using the BERT pre-training model to train the text portion of the content features to obtain a text semantic feature vector representation of the test question;

[0015] The VGG-16 pre-trained model is used to train the image part of the content feature to obtain an image semantic feature vector representation of the test question.

[0016] Furthermore, the tensor-based fusion method first expands the dimensions of the text semantic features and image semantic features of the test questions by 1, and then calculates the Cartesian product of the two to obtain the heterogeneous fused semantic features zt of the test questions:

[0017]

[0018] Where z vt is the text semantic feature of the test question answered by the student at time t, z lt is the image semantic feature of the test question answered by the student at time t.

[0019] Furthermore, the specific process of expressing the text semantic feature vector of the test question includes:

[0020] The text part of the content features of the test questions is preprocessed. After the text part of each test question is decomposed into word sequences using a word segmentation tool, a [CLS] tag is added to the head as the beginning of the question stem sentence, and a [SEP] tag is added at the end of each sentence to mark the end of the sentence or the interval between sequences; the word sequence is converted into a corresponding index sequence, and the maximum test question word sequence length is used as the standard. The sequence with insufficient length is padded with [0] to maintain the uniformity of the input sequence length; the network structure of the BERT pre-trained model is fine-tuned, retaining the original first ten layers of the encoder network, adding a fully connected layer and a Tanh activation layer thereafter, and only training the last two layers of the encoder layer, the pooling layer, and the newly added fully connected layer and Tanh activation layer;

[0021] The word index sequence of uniform length is used as the input of the fine-tuned BERT pre-trained model to complete the vector representation of the text semantic features of the test questions;

[0022] The specific process of representing the image semantic feature vector of the test question includes:

[0023] The image part of the content features of the test questions is preprocessed and the image size is unified to 128*128; the architecture of the VGG-16 pre-trained model is fine-tuned, retaining the original thirteen convolution layers with 3*3 convolution kernels and five maximum pooling layers with 2*2 filter kernels, and freezing their parameters during training. The last three fully connected layers are modified into two fully connected layers and one softmax activation layer; finally, the test question images are input into the fine-tuned VGG-16 pre-trained model.

[0024] Furthermore, the specific process of estimating the learning state vector includes:

[0025] According to the characteristics of students' test performance, the test item-response pair (z t , r t ) characterizes the students’ learning behavior. For the correct or incorrect answers, the heterogeneous fusion semantic features of the students’ answers and the students’ test performance features are spliced ​​in different orders. The spliced ​​student behavior vector x t The calculation formula is:

[0026]

[0027] Where r t Indicates the student's test performance characteristics; 0 is the same as z t All-zero vectors of the same dimension; Represents the operation of vector splicing;

[0028] A student learning state tracking model is constructed based on a bidirectional gated recurrent unit network, with the student behavior vector x t As input, the student learning status tracking model is specifically expressed as:

[0029]

[0030] Where Z t With R t are the gate values ​​of the update gate and the reset gate, W Z ,W R ,W H is the weight matrix, b Z ,b R ,b H is the corresponding bias term, σ(x) and Tanh(x) represent the activation functions respectively; represents the candidate hidden state of the gated recurrent unit network at time t, H t represents the student's learning knowledge state vector at time t;

[0031] Construct a static memory matrix M to store the information of K knowledge concepts i;

[0032] The attribute features of each question collected are represented as a one-hot vector v t , embed the attribute features of the test question through the linear layer, and calculate the influence weight of each knowledge concept:

[0033]

[0034] Where W k represents the weight matrix, k t is the low-dimensional vector after embedding representation, represents the influence weight of knowledge concept i in the knowledge weight matrix at time t;

[0035] Based on the attention mechanism, the cosine similarity of the test questions is used to measure the influence of the knowledge state at the historical moment on the knowledge state at the current moment T. The similarity weight β between the test questions answered at moment T and the historical answers is calculated as follows:

[0036]

[0037] Where z T represents the heterogeneous fusion semantic features of the students’ answers to the test questions at time T, H i′ is the student's knowledge state vector on free concept i, which contains historical answer information;

[0038] Combine the knowledge state vector and the influence weight to obtain the historical weighted learning state vector s T :

[0039]

[0040] Where K represents the number of all knowledge concepts.

[0041] Furthermore, the specific process of analyzing and predicting the student's knowledge mastery matrix using training parameters includes:

[0042] (1) Combining the heterogeneous fusion semantic features of the student's next test question with the student's historically weighted learning state vector to predict the student's test performance;

[0043] (2) The negative log-likelihood function of the student's performance on the test questions in the learning sequence is used as the objective function:

[0044]

[0045] In the formula, for time t, is the test performance predicted by the model;

[0046] (3) Using the Adam optimization method to minimize the objective function and train the model;

[0047] (4) Using the knowledge state vector, predict the student's knowledge mastery of each knowledge point.

[0048] Furthermore, the specific process of (1) includes:

[0049] Concatenate the heterogeneous fusion semantic feature vector of the student's answer to the test question at time T+1 and the student's historical weighted learning state vector to obtain the aggregated learning vector of the answer question:

[0050]

[0051] Construct a linear layer and an activation layer, take the student's aggregated learning vector at the next moment as input, and predict the student's performance on the corresponding test question:

[0052]

[0053] Where W1, W2 are weight matrices, b1, b2 are corresponding bias parameters, Relu(x) and σ(x) are activation functions, and y T+1 represents the intermediate variable for predicting the performance at the next moment, Indicates the student test performance predicted by the model at the next moment;

[0054] The specific process of (4) is as follows:

[0055] Construct a mask vector with the same dimension as the student's knowledge state vector and all zeros, and concatenate it with the student's knowledge state vector based on the attention mechanism:

[0056]

[0057] The concatenated vectors are sequentially input into the linear layer and activation layer, keeping the original parameters unchanged, and outputting the student’s knowledge mastery vector for the i-th knowledge point:

[0058]

[0059] in, Represents the intermediate variables for mastery prediction at time T, and the parameters {W1, W2, W1, W2} remain the same as those in (1) and do not require training.

[0060] Another object of the present invention is to provide a multimodal learning level mining system under small sample conditions, the multimodal learning level mining system under small sample conditions comprising:

[0061] The test question fusion semantic representation module based on transfer learning is used to collect the content features, attribute features and student test performance features from the student test question interaction sequence. Under the condition of limited training samples, the transfer learning strategy is used to freeze and fine-tune the BERT and VGG-16 pre-trained model structures. The pre-processed content features of the test questions in the text and image channels are used as input, and the bimodal semantic features obtained by deep learning representation are fused based on tensor methods to obtain a heterogeneous fusion semantic representation of the test questions.

[0062] Intelligent Knowledge State Dynamic Tracking Model Construction Module: This module is used to characterize students' learning behaviors based on their test performance and the fused semantics of the test questions. Using this as input, a bidirectional gated recurrent unit network is introduced to construct a student-oriented intelligent knowledge state dynamic tracking model.

[0063] Attention-based student learning status prediction module: used to calculate the impact of historical test questions on the current state construction using cosine similarity, and to predict students' test performance and mastery of knowledge concepts by combining historical weighted learning state vectors and test question fusion features.

[0064] Another object of the present invention is to provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the multimodal learning level mining method under small sample conditions.

[0065] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the multimodal learning level mining method under small sample conditions.

[0066] Another object of the present invention is to provide an information data processing terminal, which is used to implement the multimodal learning level mining system under the small sample condition.

[0067] In combination with the above technical solutions and the technical problems solved, please analyze the advantages and positive effects of the technical solutions to be protected by the present invention from the following aspects:

[0068] First, in view of the technical problems existing in the above-mentioned prior art and the difficulty of solving these problems, this paper closely combines the technical solutions to be protected by the present invention and the results and data during the research and development process, and analyzes in detail and in depth how the technical solutions of the present invention solve the technical problems and some creative technical effects brought about by solving the problems. The specific description is as follows:

[0069] (1) Traditional learning level mining methods only model questions from the perspective of knowledge concepts, but ignore the semantic information contained in the text and images between questions, and their performance still needs to be improved. This invention fully utilizes the text and image information of the questions interacted by students, designs a feature extraction, representation and fusion method, and better represents the learning resources;

[0070] (2) Traditional learning level mining methods ignore the impact of past questions on the construction of the current learning state, and the modeling of students' knowledge mastery status is rough and lacks explanatory power. This invention uses the attention mechanism to calculate the cosine similarity between the semantic features of the test questions, considers the impact of past learning questions on the construction of the current knowledge state, enhances students' historical state information, and improves the prediction effect;

[0071] (3) The deep algorithms in traditional learning level mining methods rely on a large amount of training data, but it is difficult to achieve accurate prediction under small sample conditions. Therefore, how to use limited training data to ensure good prediction performance under small sample conditions is an urgent problem that needs to be solved. This paper introduces the idea of ​​transfer learning and the strategy of fine-tuning. With the help of pre-trained models BERT and VGG-16, it solves the problem of implementing learning level mining under small sample conditions, and is more widely used on current online education platforms.

[0072] Second, considering the technical solution as a whole or from the perspective of the product, the technical effects and advantages of the technical solution to be protected by the present invention are described in detail as follows:

[0073] The multimodal learning level mining method under small sample conditions proposed in the present invention is significantly superior to traditional learning level mining methods in predicting students' answer responses. Experiments show that in terms of AUC, Loss, etc., the method proposed in the present invention is more effective than other traditional learning level mining methods. In addition, the present invention performs deep representation learning on the students' learning state matrix and quantifies the students' knowledge mastery, which is conducive to visual learning analysis.

[0074] Third, as auxiliary evidence for the inventiveness of the claims of the present invention, it is also reflected in the following important aspects:

[0075] (1) The expected benefits and commercial value of the technical solution of the present invention after transformation are:

[0076] The multimodal learning level mining method under small sample conditions proposed in this invention can better quantify and predict students' learning knowledge status, facilitate the implementation of smart education, and has huge commercial value.

[0077] (2) The technical solution of the present invention fills the technical gap in the industry at home and abroad:

[0078] In order to ensure good prediction performance using limited training data under small sample conditions, the present invention uses the BERT pre-trained language model and the VGG-16 pre-trained convolutional neural network to extract features from the text and image information of the test questions, respectively. Then, based on the tensor-based method, heterogeneous fusion semantic features of the test questions are constructed, and the changes in students' knowledge status are tracked in combination with their test performance, thereby achieving accurate prediction of future test answer results under small sample conditions.

[0079] (3) Whether the technical solution of the present invention solves the technical problems that people have been eager to solve but have not been able to solve successfully:

[0080] Traditional learning level mining methods capture students' knowledge mastery status in a coarse-grained manner and cannot solve the long-term dependency problem of long sequences. This invention fully considers the connection between knowledge points between different questions and the impact of historical test questions on the construction of current knowledge status. It uses Bi-GRU to dynamically model students' learning status during the learning process and introduces an attention mechanism to complete the prediction of students' knowledge mastery, improve the reliability of analysis results, and help students formulate the next learning plan more targetedly.

[0081] (4) Whether the technical solution of the present invention overcomes technical prejudice:

[0082] Traditional learning level mining methods only model questions from the perspective of knowledge concepts, but ignore the semantic information contained in the questions from text and images. This invention represents the semantic information of the test text and image channels as features and integrates them, thereby improving the structural construction of the prediction model and obtaining more accurate analysis results. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 This is a flow chart of a multimodal learning level mining method under small sample conditions provided by an embodiment of the present invention;

[0084] Figure 2 Schematic diagram of a multimodal learning level mining method under small sample conditions provided by an embodiment of the present invention;

[0085] Figure 3 Schematic diagram of the structure of a multimodal learning level mining system under small sample conditions provided by an embodiment of the present invention:

[0086] In the figure, 1. The test question fusion semantic representation module based on transfer learning, 2. The intelligent knowledge state dynamic tracking model construction module, 3. The student learning status prediction module based on the attention mechanism;

[0087] Figure 4 2 is a schematic diagram comparing the experimental results of AUC on the TIMSS2007 dataset provided by an embodiment of the present invention;

[0088] Figure 5 3 is a schematic diagram showing comparison of experimental results on the TIMSS2007 dataset in Loss provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0089] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0090] 1. Explanatory Examples In order to enable those skilled in the art to fully understand how to implement the present invention, this section provides an illustrative example that expands upon the technical solutions of the claims.

[0091] like Figure 1 As shown, the multimodal learning level mining method under small sample conditions provided by the embodiment of the present invention includes the following steps:

[0092] S101: Extract the content features, attribute features, and performance features of the test questions in the student test question interaction sequence. Based on the transfer learning strategy and fine-tuning ideas, the BERT and VGG-16 pre-trained models are introduced to represent the semantic features of the test question text and image modalities, respectively. The image and text features are fused using a tensor-based fusion method to obtain heterogeneous fused semantic features of the test questions.

[0093] S102: Combining the heterogeneous fusion semantic features of the test questions with the student's test performance characteristics, based on a bidirectional recurrent neural network, tracks the changes in the student's learning state and completes the estimation of the learning state vector by introducing the attention weights between questions;

[0094] S103, using the fused semantic features of the current test question and the student's historical weighted learning state vector, predict the student's answer score, and use the training parameters to analyze and predict the student's knowledge mastery matrix.

[0095] The principle diagram of the multimodal learning level mining method under small sample conditions provided by the embodiment of the present invention is as follows Figure 2 shown.

[0096] like Figure 3 As shown, the multimodal learning level mining system under small sample conditions provided by the embodiment of the present invention includes:

[0097] The transfer learning-based test question fusion semantic representation module 1 is used to collect the content features, attribute features of the test questions and the test question performance features from the student test question interaction sequence. Under the condition of limited training samples, the transfer learning strategy is used to freeze and fine-tune the BERT and VGG-16 pre-trained model structures. The pre-processed content features of the test questions in the text and image channels are used as input, and the bimodal semantic features obtained by deep learning representation are fused based on the tensor method to obtain a heterogeneous fusion semantic representation of the test questions.

[0098] Intelligent knowledge state dynamic tracking model construction module 2 is used to combine students' test performance and the fusion semantics of the test questions to represent students' learning behaviors in different answers. Based on this, a bidirectional gated recurrent unit network is introduced to build a student-oriented intelligent knowledge state dynamic tracking model.

[0099] The student learning status prediction module 3 based on the attention mechanism is used to calculate the impact of historical test questions on the current state construction using cosine similarity, and to predict students' test performance and mastery of knowledge concepts by combining the historical weighted learning state vector and test question fusion features.

[0100] Table 2 shows the symbols involved in the embodiments of the present invention.

[0101] Table 2 Symbols involved in the embodiments of the present invention

[0102]

[0103]

[0104] 2. Application Examples: In order to demonstrate the creativity and technical value of the technical solution of the present invention, this section provides application examples of the claimed technical solution on specific products or related technologies.

[0105] Example 1

[0106] The multimodal learning level mining method under small sample conditions provided by the embodiment of the present invention specifically includes:

[0107] (1) Extract the content features, attribute features and performance features of the test questions in the interactive sequence of student test questions. Based on the transfer learning strategy and fine-tuning ideas, the BERT and VGG-16 pre-trained models are introduced to represent the semantic features of the two modalities of test question text and image, respectively. The tensor-based fusion method is used to fuse the text and image features to obtain the heterogeneous fusion semantic features of the test questions.

[0108] (2) Combining the heterogeneous fusion semantic features of test questions with the characteristics of students' test performance, based on a bidirectional recurrent neural network, the process of students' learning state changes is tracked, and by introducing the attention weights between questions, the learning state vector is estimated;

[0109] (3) The fusion semantic features of the current test questions and the students’ historical weighted learning state vector are used to predict the students’ answer scores, and the training parameters are used to analyze and predict the students’ knowledge mastery matrix.

[0110] The first step provided in the embodiment of the present invention specifically includes:

[0111] (1.1) Collecting test content characteristics, test attribute characteristics, and student test performance characteristics from the student test interaction sequence:

[0112] We selected the interaction data of mathematics test questions for fourth-grade primary school students in the 2007 IEA Mathematics and Science Test (TIMSS) to form the dataset TIMSS2007. The number of interactions, number of knowledge points, number of test questions, and number of students are shown in Table 3 below.

[0113] Table 3. Relevant information of the TIMSS2007 dataset

[0114] Related information Specific number of data sets Number of interaction bars between students and exercises 6334 Number of knowledge points 9 Number of test questions 58 Number of students 779

[0115] The student performance characteristics, test content characteristics, and attribute characteristics of the test questions are selected from the TIMSS2007 dataset. The selected student-related characteristics and test question-related characteristics are shown in Table 4 below.

[0116] Table 4. Features selected in the TIMSS2007 dataset

[0117]

[0118]

[0119] (1.2) For the text part of the test question content features, use the BERT pre-trained model to obtain the text semantic feature vector representation of the test question:

[0120] Preprocess the test text content. Use a word segmentation tool to break down the text content of each test question into word sequences. Add a [CLS] tag at the beginning of the question stem sentence, and add a [SEP] tag at the end of each sentence to mark the end of the sentence or the interval between sequences.

[0121] Convert the word sequence into the corresponding index sequence, and use the maximum test word sequence length as the standard, and add [0] to the end of the sequence that is insufficient in length to keep the input sequence length uniform;

[0122] Fine-tune the original BERT network structure, retaining the original first ten encoder layers and adding fully connected layers and Tanh activation layers of the corresponding dimensions. Based on the fine-tuning concept, freeze the parameters of the BERT model in layers 1-10 during deep learning representation, and only train the last two encoding layers, the pooling layer, and the newly added layers. This allows the source dataset to better adapt to the sparser personalized dataset and improves the model's generalization ability.

[0123] A word index sequence of uniform length is passed through the network as input, thereby achieving deep learning representation of the text semantic features of the test questions.

[0124] (1.3) VGG-16 pre-trained model, obtain the image semantic feature vector representation of the test question:

[0125] Preprocess the test image and unify the image size to 128*128;

[0126] Fine-tune the VGG-16 architecture, retaining the original thirteen convolutional layers with 3x3 convolution kernels and five max-pooling layers with 2x2 filter kernels, freezing their parameters during training, and redefining the last three fully connected layers to two fully connected layers and one softmax activation layer to capture image feature information.

[0127] The test image is input into the fine-tuned VGG-16 architecture to achieve deep learning representation of the image semantic features of the test question.

[0128] (1.4) Based on the tensor method, we first expand the dimensions of the two-modal semantic features of the test questions by 1 and then calculate the outer product of the two to achieve semantic feature fusion, thereby obtaining a square-order heterogeneous fusion semantic feature vector of the test questions:

[0129]

[0130] Among them, for the test questions answered by students at time t, z vt is the semantic feature of the question in the text channel, z lt is the semantic feature of the test question on the image channel, and the Cartesian product is taken to obtain the heterogeneous fusion semantic feature vector z of the test question answered by the student. t .

[0131] Step 2 provided in this embodiment of the present invention specifically includes:

[0132] (2.1) Based on the students’ test performance characteristics, use the test-response pairs (z t , r t) characterizes students' learning behavior. In order to distinguish the degree to which different students' answer performances affect their knowledge status, the heterogeneous fusion semantic features of students' answers and their test performance characteristics are spliced ​​in different orders for correct or incorrect answers:

[0133]

[0134] z t is the heterogeneous fusion semantic feature of the student's answer to the test question at time t; r t is the student’s actual test performance; 0 is a t All-zero vectors of the same dimension; Represents the operation of vector splicing; the student behavior vector x obtained by splicing t Contains information on both the semantics of the test questions and the performance of the answers;

[0135] (2.2) Based on the bidirectional gated recurrent unit network, a student learning state tracking model is constructed, with the student behavior vector x t As input, for knowledge concept i, the hidden state at each moment is used to represent the student's dynamically changing learning knowledge state on this concept. The model can be specifically expressed as follows:

[0136]

[0137] Among them, Z t With R t are the gate values ​​of the update gate and the reset gate, W Z ,W R ,W H is the weight matrix, b Z ,b R ,b H is the corresponding bias term, σ(x) and Tanh are activation functions, represents the candidate hidden state of the gated recurrent unit network at time t, H t represents the student's learning knowledge state vector at time t;

[0138] (2.3) Construct a static memory matrix M to store the information of K knowledge concepts;

[0139] (2.4) Represent the attribute features of each question collected in step 1 as a one-hot vector v t , embed it through the linear layer and calculate the influence weight of each knowledge concept:

[0140]

[0141] Among them, W k is the corresponding weight matrix, k tis the low-dimensional vector after embedding representation, is the influence weight of concept i in the knowledge weight matrix at time t;

[0142] (2.5) Based on the attention mechanism, the similarity of the test content is used to measure the impact of the knowledge state at the historical moment on the knowledge state at the current moment:

[0143]

[0144] Among them, β t Use cosine similarity to measure the similarity weight between the questions answered at time T and the questions answered in the past, z T represents the heterogeneous fusion semantic features of the students’ answers to the test questions at time T, H i′ is the student’s knowledge state on concept i that contains historical answer information;

[0145] (2.6) Combine the knowledge state vector containing historical answer information with the concept influence weight to obtain the historical weighted learning state vector that combines historical answer information on all concepts:

[0146]

[0147] Step three provided in this embodiment of the present invention specifically includes:

[0148] (3.1) Combining the heterogeneous fusion semantic features of the student's next test question with the student's historically weighted learning state vector, the student's test performance is predicted:

[0149] Concatenate the heterogeneous fusion semantic feature vector of the student's answer to the test question at time T+1 and the student's historical weighted learning state vector to obtain the aggregated learning vector of the answer question:

[0150]

[0151] Construct a linear layer and an activation layer, take the student's aggregated learning vector at the next moment as input, and predict the student's performance on the corresponding test question:

[0152]

[0153] Where W1, W2 are weight matrices, b1, b2 are corresponding bias parameters. Relu(x) and σ(x) are both activation functions. T+1 Represents the intermediate variable for predicting performance at the next moment. That is, the student's test performance at the next moment predicted by the model.

[0154] (3.2) The negative log-likelihood function of the student’s test performance on the learning sequence is used as the objective function:

[0155]

[0156] For time t, r t For students' real test performance, is the test performance predicted by the model;

[0157] (3.3) Use Adam optimization to minimize the defined objective function and train the model;

[0158] (3.4) Using the student’s knowledge state vector containing historical answer information, predict the student’s knowledge mastery of each knowledge point:

[0159] Construct a mask vector with the same dimension as the student's knowledge state vector and all zeros, and concatenate it with the student's knowledge state vector based on the attention mechanism:

[0160]

[0161] The concatenated vectors are sequentially input into the linear layer and activation layer constructed in step (1), keeping the original parameters unchanged, and outputting the student's knowledge mastery level vector for the i-th knowledge point:

[0162]

[0163] in, Represents the intermediate variables for mastery prediction at time T, {W1, W2, W1, W2}, which remain the same as the parameters in step (1) and do not require training.

[0164] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware portion can be implemented using dedicated logic; the software portion can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. Those skilled in the art will appreciate that the above-mentioned devices and methods can be implemented using computer-executable instructions and / or contained in processor control code, for example, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, or can be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0165] 3. Evidence of the effects of the embodiments: The embodiments of the present invention have achieved some positive effects during the development or use process, and indeed have great advantages over the existing technology. The following content describes them with reference to the data, charts, etc. of the experimental process.

[0166] In the embodiment of the present invention, the main development environment includes: Windows 10, GTX 1080Ti, Pytorch 1.6.0, Python 3.7, and the specific hyperparameter settings of the model are shown in Table 4 below.

[0167] Table 4 Experimental model hyperparameter settings

[0168]

[0169] This paper compares the multimodal learning level mining method under small sample conditions with the traditional learning level mining method, comparing the area under the curve (AUC) and the binary cross entropy loss (Loss). The AUC evaluation index is an evaluation metric for measuring the quality of a binary classification model. It can accurately describe the overall performance of the model under the premise of an imbalance of positive and negative samples within the sample. An AUC value of 0.5 represents a randomly obtainable score, and the closer the score is to 1, the more accurate the prediction result. The Loss value can, to a certain extent, measure the deviation between the measured value and the true value. The smaller the value, the more accurate the prediction result.

[0170] The present invention compares the multimodal learning level mining method under small sample conditions with the traditional learning level mining method. In order to achieve fair comparison, the corresponding hyperparameters of the same modules in the two models are set to the same. The AUC and Loss comparison results of the multimodal learning level mining method under small sample conditions and the traditional learning level mining method on the data set TIMSS2007 are shown in Table 1. The schematic diagram of the model training process on the data set TIMSS2007 is shown in Figure 4 、 Figure 5 shown.

[0171] Table 1 Comparison of experimental results of different methods

[0172]

[0173] Experimental results show that the proposed multimodal learning level mining method under small sample conditions improves the AUC by 15.2% and reduces the loss by 0.027 for the optimal epoch on the TIMSS2007 dataset. This demonstrates that the proposed method considers incorporating features related to the multimodal semantic content of test questions and adopts a transfer learning strategy and fine-tuning method. Under the condition of limited training samples, it uses a pre-trained model to extract features and perform tensor-based fusion to better construct the semantic representation of test questions. Furthermore, by combining the heterogeneous fused semantic features of test questions with the characteristics of student test question performance, a bidirectional recurrent neural network model is constructed to track the evolution of students' learning states and estimate the learning state vector by introducing attention weights between questions. Finally, the fused semantic features of the current test question and the student's historical weighted learning state vector are used to predict the student's answer score and knowledge mastery vector, improving the model's accuracy in predicting student responses and achieving results superior to those of other traditional learning level mining methods. Experiments show that in terms of AUC and Loss, the multimodal learning level mining method under small sample conditions proposed in the present invention is more effective than the traditional learning level mining method. In short, the present invention has the best experimental effect.

[0174] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with this technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered by the scope of protection of the present invention.

Claims

1. A multimodal learning level mining method under small sample conditions, characterized by: The multimodal learning level mining method under small sample conditions includes: The content and attribute features of the test questions, as well as the student's test performance characteristics, were extracted. The BERT and VGG-16 pre-trained models were introduced to represent the semantic features of the test question text and image. The image and text features were fused using a tensor fusion method to obtain the test question's heterogeneous fusion semantic features. The heterogeneous fusion semantic features of the test questions were combined with the student's test performance characteristics. The learning state vector was estimated based on a bidirectional recurrent neural network and the attention weights between questions. The student's test performance was predicted using the heterogeneous fusion semantic features of the test questions and the historically weighted learning state vector. The training parameters were then used to analyze and predict the student's knowledge mastery matrix. The specific process of expressing the text semantic feature vector of the test question includes: The text part of the content features of the test questions is preprocessed. After the text part of each test question is decomposed into word sequences using a word segmentation tool, a [CLS] tag is added to the head as the beginning of the question stem sentence, and a [SEP] tag is added at the end of each sentence to mark the end of the sentence or the interval between sequences; the word sequence is converted into a corresponding index sequence, and the maximum test question word sequence length is used as the standard. The sequence with insufficient length is padded with [0] to maintain the uniformity of the input sequence length; the network structure of the BERT pre-trained model is fine-tuned, retaining the original first ten layers of the encoder network, adding a fully connected layer and a Tanh activation layer thereafter, and only training the last two layers of the encoder layer, the pooling layer, and the newly added fully connected layer and Tanh activation layer; The word index sequence of uniform length is used as the input of the fine-tuned BERT pre-trained model to complete the vector representation of the text semantic features of the test questions; The specific process of representing the image semantic feature vector of the test question includes: The image part of the content features of the test questions is preprocessed and the image size is unified to 128*128; the architecture of the VGG-16 pre-trained model is fine-tuned, retaining the original thirteen convolution layers with 3*3 convolution kernels and five maximum pooling layers with 2*2 filter kernels, and freezing their parameters during training. The last three fully connected layers are modified into two fully connected layers and one softmax activation layer; finally, the test question images are input into the fine-tuned VGG-16 pre-trained model.

2. The multimodal learning level mining method under small sample conditions according to claim 1 is characterized in that: The content features, attribute features, and student performance features of the test questions are arranged in chronological order as model input; the content features of the test questions include text semantic features and image semantic features of the test questions, the attribute features are the knowledge points tested by the test questions, and the performance features of the test questions are the score vectors corresponding to the students' answers to the test questions; Using the BERT pre-training model to train the text portion of the content features to obtain a text semantic feature vector representation of the test question; The VGG-16 pre-trained model is used to train the image part of the content feature to obtain an image semantic feature vector representation of the test question.

3. The multimodal learning level mining method under small sample conditions according to claim 1 is characterized in that: The tensor-based fusion method first expands the dimensions of the text semantic features and image semantic features of the test questions by 1, and then calculates the Cartesian product of the two to obtain the heterogeneous fusion semantic features z of the test questions. t : Where z vt is the text semantic feature of the test question answered by the student at time t, z lt is the image semantic feature of the test question answered by the student at time t.

4. The multimodal learning level mining method under small sample conditions according to claim 1 is characterized in that: The specific process of estimating the learning state vector includes: According to the characteristics of students' test performance, the test item-response pair (z t ,r t ) characterizes the students’ learning behavior. For the correct or incorrect answers, the heterogeneous fusion semantic features of the students’ answers and the students’ test performance features are spliced ​​in different orders. The spliced ​​student behavior vector x t The calculation formula is: Where r t Indicates the student's test performance characteristics; 0 is the same as z t All-zero vectors of the same dimension; Represents the operation of vector splicing; A student learning state tracking model is constructed based on a bidirectional gated recurrent unit network, with the student behavior vector x t As input, the student learning status tracking model is specifically expressed as: Where Z t With R t are the gate values ​​of the update gate and the reset gate, W Z ,W R ,W H is the weight matrix, b Z ,b R ,b H is the corresponding bias term, σ(x) and Tanh(x) represent the activation functions respectively; represents the candidate hidden state of the gated recurrent unit network at time t, H t represents the student's learning knowledge state vector at time t; Construct a static memory matrix M to store the information of K knowledge concepts i; The attribute features of each question collected are represented as a one-hot vector v t , embed the attribute features of the test question through the linear layer, and calculate the influence weight of each knowledge concept: Where W k represents the weight matrix, k t is the low-dimensional vector after embedding representation, represents the influence weight of knowledge concept i in the knowledge weight matrix at time t; Based on the attention mechanism, the cosine similarity of the test content is used to calculate the similarity weight β between the test questions answered at time T and the historical test questions. t , the formula is: Where z T represents the heterogeneous fusion semantic features of the students’ answers to the test questions at time T, H i′ is the student's knowledge state vector on free concept i, which contains historical answer information; Combine the knowledge state vector and the influence weight to obtain the historical weighted learning state vector s T : Where K represents the number of all knowledge concepts.

5. The multimodal learning level mining method under small sample conditions according to claim 1 is characterized in that: The specific process of analyzing and predicting the student's knowledge mastery matrix using training parameters includes: (1) Combining the heterogeneous fusion semantic features of the student's next test question with the student's historically weighted learning state vector to predict the student's test performance; (2) The negative log-likelihood function of the student's performance on the test questions in the learning sequence is used as the objective function: In the formula, for time t, is the test performance predicted by the model; (3) Using the Adam optimization method to minimize the objective function and train the model; (4) Using the knowledge state vector, predict the student's knowledge mastery of each knowledge point.

6. The multimodal learning level mining method under small sample conditions according to claim 5 is characterized in that: The specific process of (1) includes: Concatenate the heterogeneous fusion semantic feature vector of the student's answer to the test question at time T+1 and the student's historical weighted learning state vector to obtain the aggregated learning vector of the answer question: Construct a linear layer and an activation layer, take the student's aggregated learning vector at the next moment as input, and predict the student's performance on the corresponding test question: Where W1, W2 are weight matrices, b1, b2 are corresponding bias parameters, Relu(x) and σ(x) are activation functions, and y T+1 represents the intermediate variable for predicting the performance at the next moment, Indicates the student test performance predicted by the model at the next moment; The specific process of (4) is as follows: Construct a mask vector with the same dimension as the student's knowledge state vector and all zeros, and concatenate it with the student's knowledge state vector based on the attention mechanism: The concatenated vectors are sequentially input into the linear layer and activation layer, keeping the original parameters unchanged, and outputting the student’s knowledge mastery vector for the i-th knowledge point: in, Denotes the intermediate variables for mastery prediction at time T. The parameters {W1, W2, W1, W2} remain the same as those in (1) and do not require training.

7. A multimodal learning level mining system under small sample conditions implementing the multimodal learning level mining method under small sample conditions as described in any one of claims 1 to 6, characterized in that: The multimodal learning level mining system under small sample conditions includes: The transfer learning-based test question fusion semantic representation module is used to collect test question content features, attribute features, and student test question performance features from student test question interaction sequences. Under the condition of limited training samples, the transfer learning strategy is used to freeze and fine-tune the BERT and VGG-16 pre-trained model structures. The pre-processed content features of the test questions in both text and image channels are used as input, and the bimodal semantic features obtained by deep learning representation are fused based on tensor methods to obtain a heterogeneous fused semantic representation of the test questions. The intelligent knowledge state dynamic tracking model construction module is used to characterize students' learning behaviors based on their test performance and the fused semantics of the test questions. Using this as input, a bidirectional gated recurrent unit network is introduced to construct a student-oriented intelligent knowledge state dynamic tracking model. The student learning status prediction module based on the attention mechanism is used to calculate the impact of historical test questions on the current state construction using cosine similarity, and to predict students' test performance and mastery of knowledge concepts by combining the historical weighted learning state vector and test question fusion features.

8. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the multimodal learning level mining method under small sample conditions according to any one of claims 1 to 6.

9. An information data processing terminal, characterized in that: The information data processing terminal is used to implement the multimodal learning level mining system under small sample conditions as described in claim 7.

Images