Intelligent multi-label annotation method and system for difficulty-knowledge point-solving ideas

Abstract

The present invention belongs to the field of personalized learning technology, and discloses a method and system for marking test questions based on difficulty, knowledge points and problem-solving ideas based on multiple labels, and constructs a marking system covering difficulty, knowledge points and problem-solving ideas. By collecting test question data, text information such as question stems, answers, and analysis is collected, and test questions are labeled based on difficulty, knowledge points and problem-solving ideas to form a data set. In order to process texts containing mathematical formulas, the system performs special preprocessing and uses TF-IDF and Word2vec technology to embed the text. After fusion, these representations provide rich features for the test questions. A deep neural network model is further used to classify the difficulty of the test questions, and multi-label classification of knowledge points and problem-solving ideas is realized. The method of the present invention accurately predicts the difficulty, knowledge point level and problem-solving ideas of the test questions, and is particularly suitable for online education platforms. It can greatly reduce the cost of manual marking and has high commercial value.

Description

Technical Field

[0001] The present invention belongs to the field of personalized learning technology, and in particular relates to a method and system for marking test questions based on difficulty, knowledge points and problem-solving ideas based on multiple labels. Background Art

[0002] Currently, driven by the development and advancement of technologies such as cloud computing and artificial intelligence, smart education is gradually evolving into a new paradigm for educational informatization in today's technological landscape, providing tremendous opportunities for tailored teaching to individual students. Personalized test question recommendation, a crucial research topic in online learning reform, leverages information such as learners' individual characteristics, subject level, learning history, and cognitive style to more accurately match and recommend test questions, thereby increasing learners' engagement and knowledge mastery. Generally, a personalized test question recommendation system consists of three parts: learner modeling, test question modeling, and a recommendation algorithm. Test question modeling involves semantic representation and knowledge analysis of test questions, requiring comprehensive modeling from multiple perspectives, including associated knowledge points, difficulty, and discrimination.

[0003] Currently, the more mainstream test text classification methods used in test question modeling can be divided into two types: convolutional neural network-based test text classification methods and recurrent neural network-based test text classification methods. The former uses convolutional layers with convolutional kernels of different sizes to better capture local features at different levels of the test text, and utilizes pooling layers to extract more important features. The weight sharing method of the fully connected layer improves model robustness, thereby effectively modeling the test questions. However, when applied to multi-label tasks, the accuracy and stability of these two test text classification methods still have significant room for improvement.

[0004] Since the test text classification method based on convolutional neural network performs better than the test text classification method based on recurrent neural network in the task faced by the present invention, the following discussion of the prior art focuses on analyzing the test text classification method based on convolutional neural network, and calls it the traditional test text classification method.

[0005] Through the above analysis, the problems and defects of the existing technology are as follows:

[0006] (1) The deep algorithms in traditional test text classification methods require a large amount of training data and are difficult to apply to small and medium-sized test data;

[0007] (2) Traditional test text classification methods have difficulty handling a large number of categories in multi-label tasks, resulting in low model accuracy;

[0008] (3) Traditional test question text classification methods only consider using one of the one-hot, TF-IDF or word vector methods when embedding test questions, and do not fully utilize the text information of the test questions to model the test questions. Summary of the Invention

[0009] In response to the problems existing in the prior art, the present invention provides a method and system for marking test questions based on difficulty, knowledge points and problem-solving ideas based on multiple labels.

[0010] The present invention is implemented as follows: a multi-label-based difficulty-knowledge point-solution question annotation system, wherein the multi-label-based difficulty-knowledge point-solution question annotation method includes:

[0011] Test data is collected, and text information including question stems, standard answers, and analyses is collected. Test questions are annotated based on their difficulty, knowledge points, and problem-solving ideas. A dataset is constructed with test text as features and difficulty, knowledge points, and problem-solving ideas as labels. Test texts containing formulas are preprocessed and embedded using TF-IDF and Word2vec methods, respectively, and fused to obtain multi-level test text embedding features. A test annotation model based on deep neural networks is constructed to perform multi-classification predictions on the difficulty of test questions and implement multi-label classification of test knowledge points and problem-solving ideas.

[0012] Furthermore, the difficulty labels of the test questions include three levels: easy, medium, and difficult, represented by discrete values ​​1, 5, and 9. The knowledge points of the test questions correspond to a pre-organized knowledge concept system and are divided into four levels. The knowledge points between the levels have a one-to-many parent-child dependency relationship. The solution ideas of the test questions are standard method prompts for solving the problems using the underlying knowledge points.

[0013] Using the test text as a feature and difficulty, knowledge points, and problem-solving ideas as labels, we filter out some test questions that lack problem-solving ideas or knowledge point annotations, and only use the test questions with problem-solving idea annotations to construct the dataset, and divide it into training and test sets.

[0014] Furthermore, the specific process of preprocessing the test text containing the formula includes:

[0015] Initially preprocess the text, using regular expressions to remove hyperlinks from the test text and replace three or more consecutive spaces with the character \t. Then, use a search-and-replace algorithm to modify the latex and html characters in the test text to their specific Chinese meanings.

[0016] Perform word segmentation on the test text and remove stop words.

[0017] Furthermore, the process of using TF-IDF and Word2vec methods to perform sentence embedding representation and fusing them to obtain multi-level test text embedding features includes:

[0018] Based on the TF-IDF algorithm, training is performed on the text of the training set to obtain the sentence embedding representation x based on word frequency for the training set and the test set respectively. tf-idf :

[0019]

[0020] Where N d，t is the total number of times word t appears in document d, N d is the total number of times all words appear in document d, M is the total number of all documents, M t Indicates the number of documents in which word t appears;

[0021] The CBOW algorithm in the word vector model Word2vec is used to process the test questions. The target word is predicted according to the context of the word to obtain the word-level embedding representation of the training set and the test set test questions. Then, the average pooling operation is performed and normalized to obtain the sentence embedding representation x based on the word vector for the training set and the test set respectively. w2v ;

[0022] Use vector concatenation to tf-idf and x w2v Perform feature fusion to obtain multi-level test text embedding features:

[0023]

[0024] Furthermore, the process of constructing a test question annotation model based on a deep neural network and performing multi-classification prediction on the difficulty of the test questions includes:

[0025] Design a deep neural network consisting of two hidden layers, where the number of hidden units is 100 and the output of each hidden layer is transformed by the activation function φ:

[0026] H diff =φ(W2·φ(W1·x text +b1))+b2

[0027]

[0028] Where x text is the multi-level test text embedding feature as the input of the deep neural network, W1, W2 are the weight matrices of the hidden layer, b1, b2 are the corresponding bias parameters, H diff is the intermediate representation after passing through two hidden layers, W d is the weight matrix of the output layer, b dis the corresponding bias parameter, the activation function φ is set to ReLU, and This is the predicted difficulty value.

[0029] Furthermore, the process of implementing multi-label classification of test knowledge points and problem-solving ideas includes:

[0030] Find spaces to separate multiple knowledge points and multiple question types in the test questions, forming a list of all knowledge points and all solution ideas for all test questions. Then, filter out knowledge points and solution ideas that appear less than or equal to 1 time, as well as their unique associations. Use a one-hot method to further annotate the knowledge points and solution ideas corresponding to each test question. Then, use the Label Powerset method to convert each label combination in the training data into a class for prediction.

[0031] The design consists of two hidden layers with 100 hidden units, and uses multi-level test text embedding features x text For input, the knowledge points and problem-solving ideas of the pre-test questions are:

[0032]

[0033]

[0034] Where W3, W4, W5, W6 are the weight matrices of the hidden layers, b3, b4, b5, b6 are the corresponding bias parameters, and W k ,W t is the weight matrix of the output layer, b k ,b t is the corresponding bias parameter, the activation function φ is ReLU, The predicted values ​​of the third-level knowledge points and problem-solving ideas of the test questions;

[0035] After obtaining the predicted value of the third-level knowledge point, reverse search is performed according to the knowledge point directory tree to determine its corresponding first-level and second-level knowledge points.

[0036] Another object of the present invention is to provide a multi-label-based difficulty-knowledge point-solution question annotation system, the multi-label-based difficulty-knowledge point-solution question annotation system comprising:

[0037] The test question text embedding fusion representation module 1 is used to represent the preprocessed test question text based on word frequency and word vector methods, obtain sentence-level test question embedding representation, and perform feature fusion to obtain multi-level test question text embedding features;

[0038] The test question difficulty multi-classification prediction module 2 is used to construct a deep neural network by embedding features of multi-level test question texts to predict the difficulty level parameters of the test questions;

[0039] The test question multi-level knowledge point prediction module 3 is used to construct a deep neural network by embedding features of multi-level test question texts, predict the third-level knowledge points tested by the test questions, and thus trace back to the first-level and second-level knowledge points;

[0040] The test question solution prediction module 4 is used to construct a deep neural network through multi-level test question text embedding features to predict the solution label of the test question.

[0041] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0042] First, we will address the technical problems existing in the above-mentioned prior art and provide some creative technical effects after solving the problems. The details are as follows:

[0043] (1) The deep algorithms in traditional test text classification methods require a large amount of training data and are difficult to apply to small and medium-sized test data. This invention only designs two hidden layers and an activation layer to construct a deep neural network, which can overcome the dependence on large amounts of training data and maintain better accuracy performance;

[0044] (2) Traditional test text classification methods have difficulty handling a large number of categories in multi-label tasks, resulting in low model accuracy. Based on the LP algorithm, this paper transforms the multi-label task into a multi-classification problem, simplifies the problem complexity, and pre-screens categories with very low frequency to improve the accuracy of model prediction;

[0045] (3) Traditional test question text classification methods only consider using one of the one-hot, TF-IDF, or word vector methods when embedding test questions, and do not fully utilize the text information of the test questions to model the test questions. The present invention uses the word frequency method and the word vector method to perform sentence embedding representation on the test questions, and then fuses the vectors to obtain multi-level test question text embedding features to fully utilize more effective text features and improve prediction performance.

[0046] Second, the multi-label-based difficulty-knowledge point-solution idea test question annotation method proposed in the present invention is significantly superior to the traditional test question text classification method in predicting the difficulty, knowledge points, and solution ideas of the test questions. Experiments on indicators such as accuracy and recall rate show that the method proposed in the present invention is more effective than the traditional test question text classification method.

[0047] Third, as auxiliary evidence for the inventiveness of the claims of the present invention, it is also reflected in the following important aspects:

[0048] (1) The expected benefits and commercial value of the technical solution of the present invention after transformation are:

[0049] The multi-label-based difficulty-knowledge point-problem-solving idea test question annotation method proposed in the present invention can better predict the test question difficulty parameters, knowledge points at various levels, and problem-solving ideas. It is suitable for many scenarios under the online education platform, can effectively save the cost of manually annotating knowledge points and question types, and has huge commercial value.

[0050] (2) The technical solution of the present invention fills the technical gap in the industry at home and abroad:

[0051] The deep algorithms in traditional test text classification methods require a large amount of training data, which makes them difficult to apply to small and medium-sized test data. In addition, they cannot guarantee good prediction performance using limited training data in multi-label classification tasks. The present invention constructs a deep neural network consisting of multiple hidden layers and filters out labels that appear too infrequently, thereby ensuring classification accuracy under the conditions of small and medium-sized training data.

[0052] (3) Whether the technical solution of the present invention solves the technical problems that people have been eager to solve but have not been able to solve successfully:

[0053] Traditional test text classification methods struggle to handle a large number of categories in multi-label tasks, resulting in low model accuracy. Based on the LP algorithm, this paper transforms multi-label tasks into multi-classification problems, simplifying the problem complexity and pre-screening rare categories to improve model prediction accuracy.

[0054] (4) Whether the technical solution of the present invention overcomes technical prejudice:

[0055] Traditional test question text classification methods only consider using one of the one-hot, TF-IDF or word vector methods when embedding test questions, and do not fully utilize the text information of the test questions to model the test questions. The present invention fuses vectors on the premise of sentence embedding representation of test questions based on the word frequency method and the word vector method, so as to fully utilize more effective text features and improve prediction performance.

[0056] Fourth, the present invention provides a multi-label-based difficulty-knowledge-point-solution question annotation system. The following are significant technological advances brought about by this system:

[0057] 1) Multi-layered test text embedding features: The system uses a dual approach based on TF-IDF and Word2vec to embed test text and further fuse these representations. This approach captures not only the statistical information of the text (via TF-IDF) but also its semantic information (via Word2vec). This ensures rich and diverse feature representations of the test text, thereby improving the accuracy of difficulty classification, knowledge point prediction, and problem-solving strategy prediction.

[0058] 2) Multi-label classification strategy: The system uses multi-label classification to classify the knowledge points and solution ideas of test questions. This means that a test question can be labeled with multiple knowledge points and / or multiple solution ideas, reflecting the complexity of real-life test questions and providing teachers and scholars with more detailed and accurate information.

[0059] 3) Multi-level prediction of knowledge points: The system first predicts the third-level knowledge points of the test question, and then backtracks to the first- and second-level knowledge points. This hierarchical prediction method ensures accurate labeling of knowledge points and provides a complete hierarchical structure of the knowledge involved in the test question, which is very valuable for course design and learning path planning.

[0060] 4) Application of Deep Neural Networks: The system uses deep neural networks to predict difficulty, knowledge points, and problem-solving strategies. Deep neural networks are excellent at capturing complex patterns in text, enabling efficient and accurate predictions in the task of question annotation.

[0061] 5) Preprocessing of test text containing formulas: Many test questions contain mathematical or scientific formulas, often represented by specific LaTeX or HTML characters. The system can convert these characters into Chinese interpretations. This preprocessing ensures the integrity and coherence of the text information and improves the quality of the embedded features.

[0062] 6) Adaptive annotation system: The system has built a flexible difficulty-knowledge point-problem-solving idea annotation system that can easily adapt to the needs of different subjects or courses, providing educators in different fields with a universal test question annotation tool.

[0063] Overall, this system brings multiple technological advances to test annotation, which not only improve the accuracy and efficiency of annotation, but also provide valuable tools for educational research and applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 This is a flow chart of a method for marking test questions based on difficulty, knowledge points, and problem-solving ideas based on multiple labels provided by an embodiment of the present invention;

[0065] Figure 2This is a principle diagram of a method for marking test questions based on difficulty, knowledge points, and problem-solving ideas based on multiple labels provided by an embodiment of the present invention;

[0066] Figure 3 This is a schematic diagram of the structure of a multi-label-based difficulty-knowledge point-solution question annotation system provided by an embodiment of the present invention:

[0067] In the figure, 1. Test question text embedding fusion representation module, 2. Test question difficulty multi-classification prediction module, 3. Test question multi-level knowledge point prediction module, 4. Test question solution idea prediction module; DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0069] like Figure 1 As shown, the method for marking test questions based on difficulty, knowledge points, and problem-solving ideas based on multiple labels provided by an embodiment of the present invention includes the following steps:

[0070] S101: Collect test data and text information including question stems, standard answers, and solutions. Label the test questions based on their difficulty, knowledge points, and problem-solving strategies. A dataset is constructed using the test text as features and difficulty, knowledge points, and problem-solving strategies as labels.

[0071] S102: Preprocess the test text including the formula, use TF-IDF and Word2vec methods to embed the sentence level, and fuse them to obtain multi-level test text embedding features;

[0072] S103: Build a test question annotation model based on a deep neural network to perform multi-classification prediction on the difficulty of the test questions and implement multi-label classification of the test question knowledge points and problem-solving ideas.

[0073] The working principle of the multi-label-based difficulty-knowledge point-solution method is as follows:

[0074] S101: Data Collection and Annotation

[0075] Collecting test data: First, you need to collect a large number of test questions as raw data. These test questions need to include the question stem, standard answers, and explanations. This data can be obtained from online education platforms, books, or question banks provided by teachers.

[0076] Question Annotation: Each question is annotated based on its difficulty, knowledge points, and solution strategies. For example, difficulty can be marked as "easy," "medium," or "hard"; knowledge points include "quadratic equations" and "calculus"; and solutions include "algebraic methods" and "geometric methods." This annotation process can be done manually by experts or by referencing existing materials.

[0077] Constructing a dataset: Using the above labeled information, we can construct a dataset where each sample consists of the question text, and the labels include difficulty, knowledge points, and problem-solving ideas.

[0078] S102: Text Preprocessing and Embedding

[0079] Preprocessing: Test text contains formulas, symbols, and special characters, which will affect the quality of text representation. Therefore, these characters need to be properly preprocessed, such as converting them into specific tags or deleting them.

[0080] Text Embedding: Text is embedded using TF-IDF and Word2vec. TF-IDF is a statistical method that reflects the importance of a word in a document, while Word2vec is a neural network method that captures the semantic information of words. In this method, we first use these two methods to obtain sentence-level embeddings for text, and then fuse them to obtain richer, multi-layered text embedding features.

[0081] S103: Deep Neural Network Model Construction and Prediction

[0082] Model Construction: Build a deep neural network model, including multiple hidden layers, activation functions, and a final output layer. The purpose of this model is to predict the difficulty, knowledge points, and solution strategies of the test questions based on the test text features.

[0083] Difficulty prediction: Since difficulty is an ordered category (e.g., easy, medium, hard), it can be viewed as a multi-classification task.

[0084] Knowledge point and problem-solving ideas prediction: These two attributes contain multiple labels (for example, a question involves multiple knowledge points or multiple problem-solving ideas), so they are multi-label classification tasks.

[0085] Through the above steps, this method can automatically mark the difficulty, knowledge points and problem-solving ideas of test questions, thereby providing educators and learners with more accurate test question analysis and recommendations.

[0086] The principle diagram of the method for marking test questions based on difficulty, knowledge points and problem-solving ideas based on multiple labels provided by the embodiment of the present invention is as follows: Figure 2 shown.

[0087] like Figure 3 As shown, the multi-label-based difficulty-knowledge point-solution question annotation system provided by the embodiment of the present invention includes:

[0088] The test question text embedding fusion representation module 1 is used to represent the preprocessed test question text based on word frequency and word vector methods, obtain sentence-level test question embedding representation, and perform feature fusion to obtain multi-level test question text embedding features;

[0089] The test question difficulty multi-classification prediction module 2 is used to construct a deep neural network by embedding features of multi-level test question texts to predict the difficulty level parameters of the test questions;

[0090] The test question multi-level knowledge point prediction module 3 is used to construct a deep neural network by embedding features of multi-level test question texts, predict the third-level knowledge points tested by the test questions, and thus trace back to the first-level and second-level knowledge points;

[0091] The test question solution prediction module 4 is used to construct a deep neural network through multi-level test question text embedding features to predict the solution label of the test question.

[0092] The detailed working principle of the multi-label-based difficulty-knowledge point-solution question annotation system is as follows:

[0093] ###1. Test Question Text Embedding Fusion Representation Module 1:

[0094] Preprocessing: This module first preprocesses the test text to remove noise, such as special characters, symbols, or irrelevant information, to ensure the text is clean.

[0095] Text representation: Next, the module uses word frequency (such as TF-IDF) and word vector (such as Word2vec) methods to represent text. The word frequency method mainly represents the frequency of words in the text, while the word vector method attempts to capture the semantic relationship between words.

[0096] Sentence-level question embeddings: Both methods can generate embedding representations of sentences or paragraphs, usually by aggregating the word embeddings therein.

[0097] Feature fusion: In order to extract richer and more comprehensive features, this module will fuse features based on word frequency and word vectors to generate multi-level test text embedding features.

[0098] ###2. Test Question Difficulty Multi-Classification Prediction Module 2:

[0099] This module uses multi-level test text embedding features to build a deep neural network to predict the difficulty of the test questions. This is an ordered multi-classification task, such as "easy", "medium" and "difficult".

[0100] ###3. Test question multi-level knowledge point prediction module 3:

[0101] Predicting third-level knowledge points: First, this module uses multi-level test text embedding features to build a deep neural network to predict the third-level knowledge points tested in the test questions.

[0102] Backtracking to the first and second level knowledge points: Once we have determined the third level knowledge points, we can backtrack to their corresponding first and second level knowledge points according to the hierarchical structure of the knowledge system.

[0103] ###4. Test question solution prediction module 4:

[0104] Leveraging multi-level test text embedding features, this module builds a deep neural network to predict the solution to the test questions. This is a multi-label classification task because a single question can have multiple solutions.

[0105] In general, this system uses deep learning and natural language processing technologies to automatically extract features from test text and predict the difficulty, knowledge points, and solution strategies of the test questions. This not only helps educators better understand the attributes of test questions, but also provides more targeted learning suggestions for students.

[0106] Table 2 shows the symbols involved in the embodiments of the present invention.

[0107] Table 2 Symbols involved in the embodiments of the present invention

[0108]

[0109]

[0110] Example 1

[0111] The method for marking test questions based on difficulty, knowledge points, and problem-solving strategies based on multiple labels provided by the embodiment of the present invention specifically includes:

[0112] (1) Collect test data and text information including question stems, standard answers, and solutions. Label the test questions based on their difficulty, knowledge points at each level, and problem-solving strategies. Use the test text as a feature and difficulty, knowledge points, and problem-solving strategies as labels to construct a dataset.

[0113] (2) Preprocess the test text including formulas, use TF-IDF and Word2vec methods to embed sentence-level representation, and fuse them to obtain multi-level test text embedding features;

[0114] (3) Construct a test question annotation model based on deep neural networks to perform multi-classification predictions on the difficulty of test questions and implement multi-label classification of the three-level knowledge points and problem-solving ideas of test questions.

[0115] The first step provided in the embodiment of the present invention specifically includes:

[0116] (1.1) Collect test data and text information including the question stem, standard answers, and analysis:

[0117] The present invention collects subjective and objective junior high school mathematics test questions from the People's Education Press textbooks and the original stone edition textbooks, and collects the test questions' stems, standard answers, and analyses. Since some questions in the People's Education Press textbooks contain several sub-questions under the main stem, the test questions with sub-questions are split, resulting in a total of 34,510 test questions. After splitting the sub-questions, the new stems are obtained by splicing the main stem and the atomic stem.

[0118] (1.2) Test questions are annotated based on their difficulty, knowledge points at each level, and problem-solving strategies. The test text is used as a feature, and the difficulty, knowledge points, and problem-solving strategies are used as labels to construct a dataset:

[0119] First, the difficulty labels, knowledge point labels, and problem-solving ideas labels of the test questions are manually labeled. The difficulty of the test questions is labeled into three levels: easy, medium, and difficult, represented by discrete values ​​1, 5, and 9. The knowledge points of the test questions correspond to the pre-summarized junior high school mathematics knowledge concept system and are divided into four levels. The knowledge points between the levels have a one-to-many parent-child dependency relationship. The problem-solving ideas of the test questions are standard method prompts for solving problems using the underlying knowledge points;

[0120] Since some of the test questions collected in the previous step were missing the solution ideas or knowledge point annotations during manual annotation, only 19,132 test questions with solution idea annotations were used to construct the data set, and the ratio of the training set to the test set was 4:1. The order of the test questions in the training set was randomly shuffled.

[0121] The second step provided in the embodiment of the present invention specifically includes:

[0122] (2.1) Preprocess the test text containing formulas:

[0123] First, the text is preprocessed. This involves using regular expressions to remove hyperlinks from the test text and replacing three or more consecutive spaces with the character \t. Then, a search-and-replace algorithm is used to modify the latex and html characters in the test text to their specific Chinese meanings.

[0124] Secondly, the test text is segmented and stop words are removed;

[0125] (2.2) Use TF-IDF and Word2vec methods to perform sentence-level embedding representation:

[0126] First, based on the TF-IDF algorithm, we train on the text of the training set to obtain the sentence embedding representation x based on word frequency for the training set and the test set respectively. tf-idf :

[0127]

[0128] Where TF represents word frequency, N d，t is the total number of times word t appears in document d, N d is the total number of times all words appear in document d, IDF stands for inverse document frequency, where M is the total number of all documents, M t Indicates the number of documents in which word t appears;

[0129] Secondly, the CBOW algorithm in the word vector model Word2vec is used to process the test questions, predict the target word according to the context of the word, and obtain the word-level embedding representation of the training set and test set test questions. Then, the average pooling operation is performed and normalized to obtain the sentence embedding representation x based on the word vector for the training set and test set respectively. w2v ;

[0130] (2.3) Fusion obtains multi-level test text embedding features:

[0131] Use vector concatenation to tf-idf and x w2v Perform feature fusion to obtain multi-level test text embedding features:

[0132]

[0133] Step three provided in this embodiment of the present invention specifically includes:

[0134] (3.1) Construct a test question annotation model based on a deep neural network to perform multi-classification prediction of the difficulty of the test questions:

[0135] Design a deep neural network consisting of two hidden layers, where the number of hidden units is 100 and the output of each hidden layer is transformed by the activation function φ:

[0136] H diff =φ(W2·φ(W1·x text +b1))+b2

[0137]

[0138] Where x textis the multi-level test text embedding feature as the input of the deep neural network, W1, W2 are the weight matrices of the hidden layer, b1, b2 are the corresponding bias parameters, H diff is the intermediate representation after passing through two hidden layers, W d is the weight matrix of the output layer, b d is the corresponding bias parameter, the activation function φ is set to ReLU, and This is the predicted difficulty value. During the training process, the Adam algorithm is used to select the optimal solution, and the learning rate is set to 0.001;

[0139] (3.2) Implement multi-label classification of the three-level knowledge points and problem-solving ideas of the test questions:

[0140] The present invention takes the prediction of the first three levels of knowledge points as the target task. In the classification of test knowledge points, since knowledge points have their own levels, each child knowledge point only belongs to one parent knowledge point of the previous level. It only needs to predict the third level knowledge point to trace back to its corresponding first and second level knowledge point labels.

[0141] Specifically, as a multi-label classification task, we need to find spaces to separate the multiple knowledge points and multiple question types in the test questions, forming a list of all knowledge points and all solution ideas for all test questions. We also filter out knowledge points and solution ideas that appear less than or equal to 1 time, as well as their unique associations. There are a total of 70 third-level knowledge points and 276 solution ideas. First, we use a one-hot method to further annotate the knowledge points and solution ideas corresponding to each test question. Then, we use the Label Powerset method to convert each label combination in the training data into a class for prediction.

[0142] In the implementation process, we also designed a deep neural network consisting of two hidden layers with 100 hidden units, and embedded the feature x with multi-level test text. text For input, the knowledge points and problem-solving ideas of the pre-test questions are:

[0143]

[0144]

[0145] Where W1, W2, W3, W4 are the weight matrices of the hidden layers, b1, b2, b3, b4 are the corresponding bias parameters, and W k ,W t is the weight matrix of the output layer, b k ,b t is the corresponding bias parameter, the activation function φ is ReLU, The predicted values ​​of the third-level knowledge points and problem-solving ideas of the test questions;

[0146] After obtaining the predicted value of the third-level knowledge point, a reverse search is performed according to the knowledge point directory tree to determine its corresponding first-level and second-level knowledge points to complete the prediction task.

[0147] In this embodiment of the present invention, the main development environment includes: Windows 10, GTX 3060, Pytorch2.0.0, Python3.9, and the specific hyperparameter settings of the model are shown in Table 3 below.

[0148] Table 3 Experimental model hyperparameter settings

[0149] Hyperparameters Numerical batch_size 200 epoch 200 learning_rate 0.001 hidden_layer_size 100 activation relu solver adam

[0150] This paper compares a multi-label approach to question annotation based on difficulty, knowledge points, and problem-solving strategies with a traditional text classification method, comparing accuracy and recall scores. The accuracy score is the frequency of correct predictions out of the total number of predictions, and refers to the ratio of the sum of true positives and true negatives in all predictions to the total number of samples. A higher accuracy indicates better model classification performance. The recall score measures the ratio of the sum of true positives and true negatives in all predictions to the total number of samples. It is an effective indicator of prediction performance when the classes are unbalanced. A higher recall score indicates a better model's ability to distinguish between positive and negative samples.

[0151] The present invention compares the difficulty-knowledge point-solution idea test question annotation method based on multi-labels with the traditional test question text classification method. In order to achieve fair comparison, the corresponding hyperparameters of the same modules in the two models are set to the same. The accuracy score and recall rate score comparison results of the multi-label difficulty-knowledge point-solution idea test question annotation method and the traditional test question text classification method on this dataset are shown in Table 1.

[0152] Table 1 Comparison of experimental results of different methods

[0153]

[0154] The experimental results show that the multi-label-based difficulty-knowledge point-solution method proposed in the present invention has improved the accuracy corresponding to the optimal epoch in the prediction of difficulty, knowledge points and solution ideas on this data, and the recall score has been improved. This shows that the present invention fully collects valid text information including the question stem, standard answers, and analysis, and after pre-processing the test text including the formula, uses TF-IDF and Word2vec methods for embedding representation, and fuses them to obtain multi-level test text embedding features; then constructs a test annotation model based on a deep neural network, performs multi-classification prediction on the difficulty of the test question, and realizes multi-label classification of the test question knowledge points and solution ideas, improving the accuracy of the model in the three tasks of predicting the difficulty of the test question, knowledge points and solution ideas, and the results are better than the traditional test text classification method. The experiment shows that in terms of accuracy and recall, the multi-label-based difficulty-knowledge point-solution method proposed in the present invention is more effective than the traditional test text classification method. In short, the present invention has the best experimental effect.

[0155] Example 2: Personalized test question recommendation system based on difficulty

[0156] 1) Data collection: Collect a large number of test questions from educational institutions and online platforms, including question stems, options, answers, explanations, etc.

[0157] 2) Question difficulty rating: Use the above-mentioned multi-label-based difficulty-knowledge point-solution question annotation system to assign a difficulty rating to each question.

[0158] 3) User capability assessment: Have users complete a preliminary capability assessment test, and then determine the user's current knowledge level based on the score.

[0159] 4) Recommendation Engine: Based on the user's ability assessment results, it recommends test questions that match the user's ability, ensuring that the test questions are neither too easy nor too difficult.

[0160] 5) Feedback mechanism: After users complete the recommended test questions, they can provide feedback for each question to further optimize the recommendation engine.

[0161] Example 3: Adaptive Learning Path Planning System

[0162] 1) Learning goal setting: Allow users to set their learning goals, such as mastering a subject or preparing for a test.

[0163] 2) Knowledge point map construction: Based on the textbooks and examination syllabus, a relationship map between knowledge points is constructed.

[0164] 3) Current knowledge level assessment: Use an assessment test to determine the user's current knowledge level.

[0165] 4) Learning path planning: Based on the user's learning goals and current knowledge level, a learning path is planned on the knowledge point map.

[0166] 5) Test question recommendation: When users learn each knowledge point, a multi-label-based difficulty-knowledge point-solution question annotation system is used to recommend relevant practice questions to users.

[0167] 6) Progress tracking and feedback: Track users’ learning progress and dynamically adjust the learning path based on their performance.

[0168] The above two embodiments both combine a multi-label-based difficulty-knowledge point-solution question annotation system to provide users with a more personalized and effective learning experience.

[0169] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with this technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered by the scope of protection of the present invention.

Claims

1. A multi-label-based difficulty-knowledge-solution method for test questions, characterized by: include: Construct a test question annotation system based on difficulty, knowledge points and problem-solving ideas; Test question data is collected, and text information including question stems, standard answers, and solutions is collected. Test questions are annotated based on their difficulty, knowledge points, and problem-solving strategies. A dataset is constructed using the test question text as features and difficulty, knowledge points, and problem-solving strategies as labels. Test question texts containing formulas are preprocessed and embedded using TF-IDF and Word2vec methods, respectively. These are then fused to obtain multi-level test question text embedding features. Build a test question annotation model based on deep neural networks, perform multi-classification predictions on the difficulty of test questions, and implement multi-label classification of test question knowledge points and problem-solving ideas.

2. The multi-label-based difficulty-knowledge-solution method for test questions according to claim 1 is characterized in that: The difficulty labels of the test questions include three levels: easy, medium, and hard, represented by discrete values ​​1, 5, and 9. The knowledge points of the test questions correspond to a pre-organized knowledge concept system and are divided into four levels. The knowledge points between the levels have a one-to-many parent-child dependency relationship. The solution ideas of the test questions are standard method prompts for solving problems using the underlying knowledge points. Using the test text as a feature and difficulty, knowledge points, and problem-solving ideas as labels, we filter out some test questions that lack problem-solving ideas or knowledge point annotations, and only use the test questions with problem-solving idea annotations to construct the dataset, and divide it into training and test sets.

3. The multi-label-based difficulty-knowledge-solution method for question annotation according to claim 1 is characterized in that: The process of preprocessing the test text containing the formula includes: Initially preprocess the text, using regular expressions to remove hyperlinks from the test text and replace three or more consecutive spaces with the character \t. Then, use a search-and-replace algorithm to modify the latex and html characters in the test text to their specific Chinese meanings. Perform word segmentation on the test text and remove stop words.

4. The multi-label-based difficulty-knowledge-solution method for test questions according to claim 1 is characterized in that: The process of using TF-IDF and Word2vec methods to perform sentence embedding representation and fusing them to obtain multi-level test text embedding features includes: Based on the TF-IDF algorithm, training is performed on the text of the training set to obtain the sentence embedding representation x based on word frequency for the training set and the test set respectively. tf-idf : Where N d，t is the total number of times word t appears in document d, N d is the total number of times all words appear in document d, M is the total number of all documents, M t Indicates the number of documents in which word t appears; The CBOW algorithm in the word vector model Word2vec is used to process the test questions. The target word is predicted according to the context of the word to obtain the word-level embedding representation of the training set and the test set test questions. Then, the average pooling operation is performed and normalized to obtain the sentence embedding representation x based on the word vector for the training set and the test set respectively. w2v ; Use vector concatenation to tf-idf and x w2v Perform feature fusion to obtain multi-level test text embedding features:

5. The multi-label-based difficulty-knowledge point-solution method for test questions according to claim 1 is characterized in that: The process of constructing a deep neural network-based test question annotation model and performing multi-classification prediction on the difficulty of test questions includes: Design a deep neural network consisting of two hidden layers, where the number of hidden units is 100 and the output of each hidden layer is transformed by the activation function φ: H diff =φ(W2·φ(W1·x text +b1))+b2 Where x text is the multi-level test text embedding feature as the input of the deep neural network, W1, W2 are the weight matrices of the hidden layer, b1, b2 are the corresponding bias parameters, H diff is the intermediate representation after passing through two hidden layers, W d is the weight matrix of the output layer, b d is the corresponding bias parameter, the activation function φ is set to ReLU, and This is the predicted difficulty value.

6. The multi-label-based difficulty-knowledge point-solution method for test questions according to claim 1 is characterized in that: The process of implementing multi-label classification of test knowledge points and problem-solving ideas includes: Find spaces to separate multiple knowledge points and multiple question types in the test questions, forming a list of all knowledge points and all solution ideas for all test questions. Then, filter out knowledge points and solution ideas that appear less than or equal to 1 time, as well as their unique associations. Use a one-hot method to further annotate the knowledge points and solution ideas corresponding to each test question. Then, use the Label Powerset method to convert each label combination in the training data into a class for prediction. The design consists of two hidden layers with 100 hidden units, and uses multi-level test text embedding features x text For input, the knowledge points and problem-solving ideas of the pre-test questions are: Where W3, W4, W5, W6 are the weight matrices of the hidden layers, b3, b4, b5, b6 are the corresponding bias parameters, and W k ,W t is the weight matrix of the output layer, b k ,b t is the corresponding bias parameter, the activation function φ is ReLU, The predicted values ​​of the third-level knowledge points and problem-solving ideas of the test questions; After obtaining the predicted value of the third-level knowledge point, reverse search is performed according to the knowledge point directory tree to determine its corresponding first-level and second-level knowledge points.

7. A multi-label difficulty-knowledge point-solution question annotation system implementing the annotation method according to any one of claims 1 to 6, characterized in that: include: A multi-label-based difficulty-knowledge-point-solution question annotation system, characterized by including: The test question annotation system construction module is used to build an annotation system for difficulty, knowledge points, and problem-solving ideas; The data collection module is used to collect test text information including the question stem, standard answers, and analysis, and to add labels for difficulty, knowledge points, and problem-solving ideas; The text preprocessing module is used to preprocess the test text containing formulas and use TF-IDF and Word2vec methods to embed them and further obtain multi-level test text embedding features; The deep neural network model module is used for multi-classification prediction of test question difficulty and multi-label classification of knowledge points and problem-solving ideas.

8. The marking system according to claim 7, characterized in that: The system further comprises: The label filtering module is used to filter out test questions that lack solution ideas or knowledge point annotations, and construct the test questions with solution idea annotations into a data set; The dataset splitting module is used to divide the above dataset into training set and test set.

9. The marking system according to claim 7, characterized in that: The text preprocessing module further includes: Regular expression processing unit, used to remove hyperlinks in the test text and replace consecutive spaces; The word segmentation processing unit is used to segment the test text and remove stop words.

10. The marking system according to claim 7, characterized in that: The deep neural network model module further includes: The difficulty prediction unit uses a neural network consisting of two hidden layers to predict the difficulty of the test questions; The multi-label classification unit is used to perform multi-label classification on the knowledge points and problem-solving ideas of the test questions. The LabelPowerset method is used to convert each label combination in the training data into a class for prediction.

Images