An Automatic Evaluation Method for Course Teaching Cases Based on Bloom's Taxonomy

Abstract

The present invention discloses an automatic evaluation method for course teaching cases based on the Bloom classification method. First, teaching cases from the computer science courses to be evaluated are collected and organized into a document containing several teaching cases. Then, the organized case data is annotated and standardized into the format of a standard data set. Next, a classification model based on a pre-trained language model and a convolutional neural network is built, and the pre-processed case data set is used to train the model. Finally, the data set of the course teaching cases to be evaluated is fed into the trained classification model based on the pre-trained language model and the convolutional neural network for text classification, and an accurate classification result of the data set of the test cases to be classified is obtained. The method of the present invention constructs a small-sample data set of course teaching cases. The built evaluation model enhances the ability to recognize and segment Chinese words, has better semantic representation ability and the ability to capture context semantics, and improves the performance of automatic classification of course teaching cases.

Description

Technical Field

[0001] The present invention relates to the technical field of Chinese text classification, and particularly to an automatic evaluation method for course teaching cases based on Bloom's Taxonomy. Background Art

[0002] There are mainly two means, manual evaluation and machine evaluation, for applying Bloom's Taxonomy to the evaluation and classification stage of educational achievements in specific fields. Early manual evaluation was not only time-consuming and laborious, but also the evaluation results largely depended on the subjective tendencies of educators. The evaluation means based on machine learning standardized the evaluation criteria, making it possible for evaluators to have evidence to follow while reducing the evaluation workload. It mainly includes methods based on supervised learning and methods based on deep learning.

[0003] The method based on supervised learning adjusts the parameters of the classifier by using samples of known categories, so that the classifier classifies the samples according to the learned rules. Yusof et al. tried different combinations of feature engineering and supervised learning models. The support vector machine (SVM) classifier they used could judge the data according to the confidence level, but this method had poor results for high-dimensional classification of cognitive processes. Zhang et al. proposed a classification method of category frequency-inverse document frequency (CF-IDF). This method classifies problems by using the frequency of each category label. This method can better judge the sample labels with fewer occurrences, but has poor judgment effects for labels with a large number of samples. Omar et al. applied techniques based on natural language processing for identification, used important classification keywords in the judgment process, and used a rule-based method to identify the expected cognitive process dimension level. Due to the insufficient scale of the training set, this method could not fully learn the rules, and the text sentences it used were too simple and direct, resulting in accurate evaluation effects only in the memory and understanding dimensions. The method based on supervised learning is suitable for data samples with a small scale and short text length. However, supervised learning requires reviewing the data samples used for training and designing relevant judgment rules, and the training requires a large amount of computing time.

[0004] The deep learning-based method can automatically learn the basic features of the text through the model and combine them into advanced features to achieve feature extraction of the text, and select the appropriate classifier according to the specific task to achieve sample evaluation. With the continuous improvement of computer computing power and the explosive growth of unsupervised text data, deep learning models have been widely used in the field of text classification. Manjushree et al. applied the deep learning-based models convolutional neural network (CNN) and long short-term memory network (LSTM) to classify the evaluation samples into the cognitive process dimension of Bloom taxonomy. They collected and manually annotated 844 instance samples from the software engineering course. The team experimented with the data samples in a ratio of 7:3 between the training set and the test set. On the test set using the CNN model, the author team achieved good results in the low dimension of the cognitive process, but the evaluation effect was not good for the evaluation and above dimensions. In addition, most of the existing data sets are English data, and there is a lack of Chinese-related field data sets. The deep learning model for automatic classification of Chinese case data sets is not effective. Therefore, it is urgent to develop an automatic evaluation method for Chinese case data sets suitable for computer science course teaching. Summary of the invention

[0005] The present invention aims to solve the problems that the existing small sample case data sets are limited in size, the training samples contain insufficient information, and the automatic classification effect of the teaching case data sets is poor. A method for automatic evaluation of course teaching cases based on Bloom taxonomy is provided.

[0006] To solve the above problems, the present invention is achieved through the following technical solutions:

[0007] An automatic evaluation method for course teaching cases based on Bloom's taxonomy includes the following steps:

[0008] Step 1: Collect teaching cases from the computer science courses to be evaluated, including textbooks and lesson plans, and organize them into a document containing several teaching cases;

[0009] When sorting, extract the content representing the teaching objectives of this section from the case into sentences. Because the teaching objectives contain the keywords in the case content, the sentences representing the teaching objectives are used as data text for automatic classification of the case. The original textbook and lesson plan content are screened, and non-text content and text containing many formula symbols in the case are deleted.

[0010] Step 2: Label the organized teaching cases according to the Bloom's Taxonomy Verb Dictionary. The verb dictionary contains correlative words in the cognitive process dimension and the knowledge dimension, and the correlative words are verbs. According to the words and content in the teaching cases that are most relevant to the correlative words, classify the cases into the cognitive process dimension and the knowledge dimension, and then organize the labeled teaching cases into the format of a standard dataset, the content of which includes Bloom's Taxonomy labels and teaching objectives extracted from the cases.

[0011] Step 3: Build a deep learning course case evaluation model based on Bloom's Taxonomy.

[0012] Step 4: Use the case dataset organized in Step 2 to train the evaluation model built in Step 3. Input the course case dataset, calculate the loss for each batch of data through the loss function, and use the Adam gradient descent algorithm for update. After iteratively training for epoch times, obtain the trained evaluation model.

[0013] Step 5: Input the dataset of the course teaching cases to be evaluated into the evaluation model trained in Step 4 for evaluation.

[0014] Step 6: Output the evaluation results to obtain the accurate classification results of the dataset of the course teaching cases to be evaluated. The results include the macro accuracy rate and the macro F1 value. The formula for the macro accuracy rate is:

[0015]

[0016] The formula for the macro F1 value is:

[0017] Among them, K is the number of categories, P i represents the accuracy rate, P macro represents the macro accuracy value, and R macro represents the macro recall value.

[0018] The evaluation model includes a pre-trained language model and a classification model of a convolutional neural network, and is composed of a backbone structure, a text convolution structure, and a downsampling structure connected in sequence. Specifically, it is composed of an input layer, a pre-training layer, a word embedding layer, a text convolution layer, and an output layer. The output end of the input layer Input is connected to the input end of the encoder module, the output end of the encoder module is connected to the input end of the word embedding layer, the output end of the word embedding layer is connected to the input end of the text convolution layer, and the output end of the text convolution layer is connected to the input end of the output layer.

[0019] The encoder module consists of a multi-head self-attention module, a first-layer and a second-layer residual module, and two feed-forward network modules; the input end of the multi-head self-attention module forms the input end of the pre-training layer, the output end of the multi-head self-attention module is connected to the input end of the first-layer residual module, the output end of the first-layer residual module is connected to the input ends of the two feed-forward network modules, the output ends of the two feed-forward networks are connected to the input end of the second-layer residual module, and the output end of the second-layer residual module forms the output end of the pre-training layer.

[0020] The text convolution layer consists of a convolution kernel, a feature matrix, and a fully connected layer; the input end of the convolution kernel forms the input end of the text convolution layer, the convolution kernel performs a convolution operation with the input data to obtain a feature matrix, the calculated feature matrix is sent to the input end of the fully connected layer, after the fully connected layer performs a pooling operation on the feature matrix, the result is sent to the output end of the fully connected layer, and the output end of the fully connected layer uses the SoftMax activation function for calculation, and the output of the SoftMax activation function forms the output end of the text convolution layer.

[0021] The pre-trained language model is a pre-trained model using Chinese word segmentation, with a pre-segmentation operation added to the model to identify and segment Chinese words; the convolutional neural network is a convolutional neural network using n-gram feature representation, and the network uses word vectors containing semantic information for feature extraction and classification.

[0022] Compared with the prior art, the course case dataset constructed by the method of the present invention comprehensively solves the problems of the small scale of the existing Chinese course case dataset in computer science, non-standard annotation, lack of evaluation links for two dimensions of the Bloom taxonomy, inaccurate Chinese word segmentation for case data in traditional machine learning text classification technology, and insufficient semantic learning of word vectors. It constructs a small-sample course teaching case dataset, standardizes the annotation criteria, proposes an automatic evaluation model based on a pre-trained language model and a text convolutional neural network, enhances the ability to identify and segment Chinese words, has better semantic representation ability and the ability to capture context semantics, improves the performance of automatic classification of the case dataset, and further improves the automatic evaluation performance of applying the Bloom taxonomy to the Chinese course case dataset in computer science. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a flowchart of the evaluation method of the present invention;

[0024] Figure 2 is a schematic diagram of the evaluation model structure in the evaluation method of the present invention;

[0025] Figure 3 is a schematic diagram of the structure of the encoder module in the evaluation model of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0026] The present invention will be further described in detail below in conjunction with the embodiments and drawings, but the present invention is not limited thereto.

[0027] Example

[0028] An automatic evaluation method for course teaching cases based on Bloom's taxonomy, such as Figure 1 As shown, the specific steps include:

[0029] Step 1: Collect teaching cases from the computer science courses to be evaluated, including textbooks and lesson plans, and organize them into a document containing several teaching cases;

[0030] The collected case collection mainly comes from computer science course textbooks and textbook-related teaching plans. When sorting, the content representing the teaching objectives of this section in the case is extracted into sentences. Because the teaching objectives contain keywords in the case content, the sentences representing the teaching objectives are used as data texts for automatic case classification. The original textbook and teaching plan content is screened to delete non-text content and text containing many formula symbols in the case.

[0031] Step 2: Label the organized teaching cases according to the Bloom taxonomy verb dictionary. The verb dictionary contains the conjunctions in the cognitive process dimension and the knowledge dimension. The conjunctions are verbs. Based on the words and content most relevant to the conjunctions in the teaching cases, the cases are divided into the cognitive process dimension and the knowledge dimension. Then, the labeled teaching cases are organized into a standard data set format, which includes Bloom taxonomy labels and the teaching objectives extracted from the cases.

[0032] Step 3: Build a classification model based on the pre-trained language model and convolutional neural network, set network parameters, calculate the loss function of batch data to get the loss, and use the Adam gradient descent algorithm to update it, iterating the training epoch times;

[0033] Step 4: Use the case data set organized in step 2 to train the evaluation model built in step 3. Input the course case data set, calculate the batch data through the loss function to get the loss, and use the Adam gradient descent algorithm to update. After iterative training epochs, the trained evaluation model is obtained.

[0034] Step 5: Input the teaching case data set of the course to be evaluated into the evaluation model trained in step 4 for evaluation;

[0035] Step 6: Output the evaluation results to obtain the accurate classification results of the teaching case data set of the course to be evaluated. The results include macro accuracy and macro F1 value. The macro accuracy calculation formula is:

[0036]

[0037] The calculation formula for the macro F1 value is as follows:

[0038] Among them, K is the number of categories, P i represents the accuracy rate, and P macro represents the macro-accuracy value, and R macro represents the macro-recall value.

[0039] The evaluation model, as Figure 2 shown, includes a pre-trained language model and a classification model of a convolutional neural network, and is composed of a backbone structure, a text convolution structure, and a downsampling structure connected in sequence, and is specifically composed of an input layer, a pre-training layer, a word embedding layer, a text convolution layer, and an output layer;

[0040] 1) Backbone structure

[0041] In the backbone structure, the output end of the input layer Input is connected to the input end of the encoder module. See Figure 3 , the encoder module is composed of a multi-head self-attention module, a first-layer and a second-layer residual module, and two feed-forward network modules; the input end of the multi-head self-attention module forms the input end of the encoder module, the output end of the multi-head self-attention module is connected to the input end of the first-layer residual module, the output end of the first-layer residual module is connected to the input ends of the two feed-forward network modules, the output ends of the two feed-forward networks are connected to the input end of the second-layer residual module, and the output end of the second-layer residual module forms the output end of the encoder module. The output end of the encoder module is connected to the input end of the word embedding layer;

[0042] The encoder module sends the input information into the multi-head self-attention module. When encoding a specific word, this module views other words in the sentence to determine the weight of the specific word. The calculation formula of the multi-head self-attention mechanism is The output end of the multi-head self-attention module is connected to the first-layer residual module. The residual module is used to pass the output information deeper. The output of the first-layer residual module is connected to two parallel feed-forward neural networks. The output ends of the two feed-forward neural networks are connected to the second-layer residual module. The feed-forward neural network uses the ReLU activation function, and the calculation formula is FFN(x) = max(0, xW1 + b1)W2 + b2;

[0043] 2) Text convolution structure

[0044] In the text convolution structure, the text convolution layer consists of a convolution kernel, a feature matrix, and a fully connected layer; the input end of the convolution kernel forms the input end of the text convolution layer. The convolution kernel performs a convolution operation with the input data to obtain a feature matrix. The calculated feature matrix is sent to the input end of the fully connected layer. After the fully connected layer performs a pooling operation on the feature matrix, the result is sent to the output end of the fully connected layer. The output end of the fully connected layer uses the SoftMax activation function for calculation. The output of the SoftMax activation function forms the output end of the text convolution layer. The calculation formula of the SoftMax activation function is where C is the number of output nodes;

[0045] 3) Downsampling structure

[0046] In the downsampling structure, the input end of the pooling layer forms the input end of the downsampling structure. The max-pooling function in the pooling layer extracts the maximum value in the feature vector to represent the feature and cascades the extracted features and inputs them to the output end of the pooling layer. The calculation formula of the max-pooling function is c is the feature vector in the current sliding window, represents the maximum value in the feature vector. The output end of the pooling layer is connected to the input end of the fully connected layer. The fully connected layer uses the SoftMax activation function for normalization to obtain a probability distribution and inputs it to the input end of the fully connected layer. The output end of the fully connected layer forms the output end of the downsampling structure.

[0047] It should be noted that although the embodiments described above of the present invention are illustrative, this is not a limitation of the present invention. Therefore, the present invention is not limited to the above specific embodiments. Without departing from the principle of the present invention, any other embodiments obtained by those skilled in the art under the inspiration of the present invention are deemed to be within the protection scope of the present invention.

Claims

1. An automatic evaluation method for course teaching cases based on Bloom's taxonomy, characterized in that The steps include: Step 1: Collect teaching cases from the computer science courses to be evaluated, including textbooks and lesson plans, and organize them into a document containing several teaching cases; When collating, extract the content representing the teaching objectives of this section from the case into sentences, and use the sentences representing the teaching objectives as data text for automatic case classification. Screen the original textbook and lesson plan content, and delete non-text content and text containing many formula symbols in the case. Step 2: Label the organized teaching cases according to the Bloom taxonomy verb dictionary. The verb dictionary contains the conjunctions in the cognitive process dimension and the knowledge dimension. The conjunctions are verbs. Based on the words and content most relevant to the conjunctions in the teaching cases, the cases are divided into the cognitive process dimension and the knowledge dimension. Then, the labeled teaching cases are organized into a standard data set format, which includes Bloom taxonomy labels and the teaching objectives extracted from the cases. Step 3: Build a deep learning course case evaluation model based on Bloom’s taxonomy; Step 4: Use the case data set organized in step 2 to train the evaluation model built in step 3. Input the course case data set, calculate the batch data through the loss function to get the loss, and use the Adam gradient descent algorithm to update. After iterative training epochs, the trained evaluation model is obtained. Step 5: Input the teaching case data set of the course to be evaluated into the evaluation model trained in step 4 for evaluation; Step 6: Output the evaluation results to obtain the accurate classification results of the teaching case data set of the course to be evaluated. The results include macro accuracy and macro F1 value. The macro accuracy calculation formula is: The calculation formula for the macro F1 value is as follows: where K is the number of categories, P i represents the accuracy rate, and P macro represents the macro accuracy value, and R macro represents the macro recall value.

2. The automatic evaluation method for course teaching cases according to claim 1, characterized in that: The evaluation model includes a pre-trained language model and a classification model of a convolutional neural network, which is composed of a backbone structure, a text convolution structure and a downsampling structure connected in sequence, and specifically consists of an input layer, a pre-trained layer, a word embedding layer, a text convolution layer and an output layer; the output end of the input layer Input is connected to the input end of the encoder module, the output end of the encoder module is connected to the input end of the word embedding layer, the output end of the word embedding layer is connected to the input end of the text convolution layer, and the output end of the text convolution layer is connected to the input end of the output layer.

3. The automatic evaluation method for course teaching cases according to claim 2, characterized in that: The pre-trained language model is a pre-trained model using Chinese word segmentation, and a pre-segmentation operation is added to the model to recognize and segment Chinese words; the convolutional neural network is a convolutional neural network using n-gram feature representation, and the network uses word vectors containing semantic information for feature extraction and classification.

4. The automatic evaluation method for course teaching cases according to claim 2, characterized in that: The encoder module consists of a multi-head self-attention module, a first-layer and a second-layer residual module, and two feedforward network modules; the input end of the multi-head self-attention module forms the input end of the pre-training layer, the output end of the multi-head self-attention module is connected to the input end of the first-layer residual module, the output end of the first-layer residual module is connected to the input end of the two feedforward network modules, the output ends of the two feedforward networks are connected to the input end of the second-layer residual module, and the output end of the second-layer residual module forms the output end of the pre-training layer.

5. The automatic evaluation method for course teaching cases according to claim 2, wherein: The text convolution layer consists of a convolution kernel, a feature matrix, and a fully connected layer; the input end of the convolution kernel forms the input end of the text convolution layer. The convolution kernel performs a convolution operation with the input data to obtain a feature matrix. The calculated feature matrix is sent to the input end of the fully connected layer. After the fully connected layer performs a pooling operation on the feature matrix, the result is sent to the output end of the fully connected layer. The output end of the fully connected layer uses the SoftMax activation function for calculation. The output of the SoftMax activation function forms the output end of the text convolution layer. The calculation formula of the SoftMax activation function is where C is the number of output nodes.

6. The automatic evaluation method for course teaching cases according to claim 5, characterized in that: The input end of the pooling layer forms the input end of the downsampling structure. The max-pooling function in the pooling layer extracts the maximum value in the feature vector to represent the feature, and cascades the extracted features and inputs them to the output end of the pooling layer. The calculation formula of the max-pooling function is c is the feature vector in the current sliding window, represents the maximum value in the feature vector. The output end of the pooling layer is connected to the input end of the fully connected layer. The fully connected layer uses the SoftMax activation function for normalization processing to obtain the probability distribution and inputs it to the input end of the fully connected layer. The output end of the fully connected layer forms the output end of the downsampling structure.

Images