Automatic annotation method and device for exercise resources

Abstract

The present invention provides a method and device for automatically annotating exercise resources, the method comprising: inputting the exercise to be classified into a pre-trained label classification model to obtain a label probability distribution output by the label classification model; inputting the exercise to be classified into a pre-trained meta-label classification model to obtain a meta-tag probability distribution output by the meta-label classification model, wherein the meta-tag is an independent phrase characterizing the uniqueness of the tag determined based on the tag; reorganizing the meta-tag probability distribution based on the mapping relationship between the tag and the meta-tag, using the reorganized meta-tag probability distribution to weight the label probability distribution, and determining the annotation result of the exercise to be classified based on the weighted result. The present invention effectively solves the problem that existing classification algorithms have difficulty distinguishing highly similar tags by splitting semantically rich tags into independent meta-tags, thereby improving the accuracy of automatic annotation and the ability to recognize long-tail tags.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing technology, and in particular to a method and device for automatically annotating exercise resources. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, the online education sector is experiencing a booming growth opportunity, encompassing areas such as automated problem solving, exercise recommendations, difficulty prediction, and automated test generation. In real-world industrial production environments, exercise labeling is essential to accurately deliver exercise resources and ensure that the content matches students' knowledge levels.

[0003] Relying on manual annotation by educational experts is time-consuming and biased, so there is an urgent need for an automatic annotation method that can not only automatically annotate exercises but also provide high-precision prediction results. Summary of the Invention

[0004] The present invention provides a method and device for automatically annotating exercise resources, which are used to solve the defect of the prior art that manual annotation of exercise resources is relatively troublesome, and realize a high-precision automatic annotation method for exercise resources.

[0005] The present invention provides an automatic annotation method for exercise resources, comprising:

[0006] Inputting the exercises to be classified into a pre-trained label classification model to obtain the label probability distribution output by the label classification model;

[0007] Inputting the to-be-classified exercise into a pre-trained meta-tag classification model to obtain a meta-tag probability distribution output by the meta-tag classification model, wherein the meta-tag is an independent phrase determined based on the tag that characterizes the uniqueness of the tag;

[0008] The meta-tag probability distribution is reorganized based on the mapping relationship between the tags and the meta-tags, the tag probability distribution is weighted using the reorganized meta-tag probability distribution, and the labeling result of the to-be-classified exercise is determined according to the weighted result.

[0009] According to an automatic labeling method for exercise resources provided by the present invention, before the step of inputting the exercises to be classified into a pre-trained label classification model to obtain the label probability distribution output by the label classification model, the method further includes:

[0010] Using the mathematics texts of the exercise resources as samples and the knowledge points of the mathematics texts as labels to construct a label dataset;

[0011] Pre-training a masked language task on the labeled dataset to obtain a preliminary model;

[0012] The prepared model is fine-tuned on the labeled dataset based on multi-task learning to obtain a label classification model for outputting a label probability distribution.

[0013] According to a method for automatically annotating exercise resources provided by the present invention, the main task of the multi-task learning is defined as a label classification task, and the auxiliary task is defined as a class center learning task, wherein the class center learning task is a distance-based optimization task for clustering key features of the input text.

[0014] According to the method for automatically labeling exercise resources provided by the present invention, before the step of inputting the exercise to be classified into a pre-trained meta-tag classification model to obtain the meta-tag probability distribution output by the meta-tag classification model, the method further includes:

[0015] Using the mathematics texts of the exercise resources as samples and the knowledge points of the mathematics texts as labels to construct a label dataset;

[0016] Determine a meta-tag corresponding to each tag based on a predefined division rule, and construct a meta-tag dataset based on the tag dataset;

[0017] Pre-training a masked language task on the labeled dataset to obtain a preliminary model;

[0018] The prepared model is fine-tuned on the meta-tag dataset to obtain the meta-tag classification model for outputting the meta-tag probability distribution.

[0019] According to the automatic annotation method for exercise resources provided by the present invention, the predefined division rules include the following rules 1 to 4:

[0020] Rule 1 is to extract independent phrases of the tag text and decompose them into at least two alternative meta tags;

[0021] Rule 2 is to discard the alternative meta tags that appear alone in any tag;

[0022] Rule 3 is to discard candidate meta tags whose occurrence frequency is greater than a preset threshold, where the occurrence frequency is the ratio of the total number of occurrences of candidate meta tags to the total number of tags;

[0023] Rule 4 is that when the tag text cannot be decomposed into at least two candidate meta tags, the candidate meta tags are determined according to the knowledge points corresponding to the tags.

[0024] According to the automatic annotation method for exercise resources provided by the present invention, the step of determining the annotation results of the exercises to be classified based on the weighted results specifically includes:

[0025] Determining a confidence score for each tag in the tag probability distribution according to the weighted result;

[0026] Sorting the labels in the label probability distribution in descending order of the confidence scores to obtain a reordered label probability distribution;

[0027] The labels with confidence scores greater than a preset score threshold in the reordered label probability distribution are used as the labeling results of the exercises to be classified.

[0028] The present invention also provides an automatic annotation device for exercise resources, comprising:

[0029] A label probability distribution module is used to input the exercises to be classified into a pre-trained label classification model to obtain the label probability distribution output by the label classification model;

[0030] a meta-tag probability distribution module, configured to input the to-be-classified exercise into a pre-trained meta-tag classification model to obtain a meta-tag probability distribution output by the meta-tag classification model, wherein a meta-tag is an independent phrase determined based on a tag and representing the uniqueness of the tag;

[0031] The labeling module is used to reorganize the meta-tag probability distribution based on the mapping relationship between the tag and the meta-tag, use the reorganized meta-tag probability distribution to weight the tag probability distribution, and determine the labeling result of the to-be-classified exercise according to the weighted result.

[0032] The present invention also provides an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for automatically marking exercise resources as described above is implemented.

[0033] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described methods for automatically marking exercise resources.

[0034] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned methods for automatically marking exercise resources.

[0035] The automatic annotation method and device for exercise resources provided by the present invention split semantically rich tags into independent meta-tags, and pre-train to obtain tag classification models and meta-tag classification models. Based on the meta-tag probability distribution, the results of the tag probability distribution are weighted and guided, effectively solving the problem that the existing classification algorithm has difficulty in distinguishing highly similar tags, and improving the accuracy of automatic annotation and the ability to recognize long-tail tags. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 This is one of the flow charts of the automatic annotation method for exercise resources provided by the present invention;

[0038] Figure 2 This is the second flow chart of the automatic annotation method for exercise resources provided by the present invention;

[0039] Figure 3 This is the third flow chart of the automatic annotation method for exercise resources provided by the present invention;

[0040] Figure 4 It is a structural diagram of the automatic annotation device for exercise resources provided by the present invention;

[0041] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0042] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0043] Exercise resources consist of mathematical text and geometric images. Mathematical text includes descriptive text and mathematical formulas. Labeling knowledge points in exercise resources is a multi-label text classification task.

[0044] Given the rich formulas in mathematical text, traditional machine learning methods extract features from mathematical text using techniques such as the n-gram distribution (a statistical language model), frequency of mathematical symbols, structural kernels, and TF-IDF (term frequency-inverse document frequency). Classification is then performed using Naive Bayesian or Support Vector Machines (SVMs). With the advancement of deep learning, encoders that automatically capture features, such as CNNs (Convolutional Neural Networks) and LSTMs (Long Short-Term Memory), have demonstrated superior performance when processing complex text data. Pre-trained language models, such as BERT (Bidirectional Encoder Representations from Transformers), are frequently used in text classification tasks.

[0045] However, due to the uneven distribution of knowledge points and exam focus, there are a large number of long-tail tags with low sample frequency in the exercise resources in the online education field, and the exercises are difficult to distinguish because some tags have high semantic overlap. As a result, automatic exercise labeling has been difficult to effectively improve.

[0046] To this end, this embodiment proposes a method for annotating exercise resources by combining tags and meta-tags, so as to overcome the underfitting problem of long-tail tags to a certain extent, and at the same time improve the accuracy of semantic distinction of tags with high semantic overlap.

[0047] The following combination Figures 1 to 3 The automatic annotation method of the exercise resource of the present invention is introduced, such as Figure 1 As shown, the method includes:

[0048] Step 101: Input the exercises to be classified into a pre-trained label classification model to obtain the label probability distribution output by the label classification model;

[0049] Exercises to be classified are exercise resources that need to have their label categories determined. The label of an exercise to be classified is the knowledge point to which the exercise belongs, which is represented by a short sentence containing multiple independent phrases.

[0050] Using the BERT model as the initial model, a label classification model is obtained through multiple rounds of training on a dataset in the form of "mathematical text-label". The label classification model outputs each label and its corresponding probability as a label probability distribution.

[0051] It is understandable that the label probability distribution output by the pre-trained label classification model is obtained by training the model based on a dataset in the form of "mathematical text-label". Therefore, it is less sensitive to some labels with high semantic overlap. For example, "Practical Application of Addition and Subtraction of Integers" and "Column Algebraic Expressions and Practical Problems" have similar label semantic overlap and mathematical text performance, which makes the single label probability model less discriminative for such labels.

[0052] Step 102: Input the to-be-classified exercise into a pre-trained meta-tag classification model to obtain a meta-tag probability distribution output by the meta-tag classification model, wherein a meta-tag is an independent phrase determined based on the tag that characterizes the uniqueness of the tag;

[0053] Based on the above reasons, the BERT model is used as the initial model. A meta-tag classification model is obtained through multiple rounds of training on a dataset in the form of "mathematical text-meta-tag". The meta-tag classification model outputs each meta-tag and its corresponding probability as the meta-tag probability distribution.

[0054] The meta-tags are predetermined based on the tags of each exercise resource and are used to represent the part of each tag that is more unique than other tags.

[0055] Taking the tag "Application of inequality properties" as an example, it includes three independent phrases: "inequality", "property" and "application". Among them, the phrase "application" refers to the practical application of the knowledge point of inequality properties. Most knowledge points involve practical application problems, which means that a large number of tags contain "application". Therefore, "application" is discarded, and "inequality" and "property" are determined as the meta-tags of "Application of inequality properties".

[0056] On this basis, the meta-tag probability distribution obtained by using the meta-tag classification model has a better understanding of tags with high semantic overlap. On the other hand, for some long-tail tags, there are meta-tags among the multiple meta-tags determined that represent more exercise resources than the long-tail tags themselves, thereby improving the fitting effect of long-tail tags to a certain extent.

[0057] Step 103 : reorganize the meta-tag probability distribution based on the mapping relationship between the tags and the meta-tags, use the reorganized meta-tag probability distribution to weight the tag probability distribution, and determine the labeling result of the to-be-classified exercise according to the weighted result.

[0058] Since one tag corresponds to multiple meta-tags, before using the meta-tag probability distribution to weight the tag probability distribution, the meta-tag probability distribution needs to be reorganized according to the mapping relationship between the tag and the meta-tag.

[0059] It can be understood that the mapping relationship dict0 between tags and meta tags is constructed when the meta tag corresponding to each tag is determined.

[0060] In a specific embodiment, Figure 2 As shown, after obtaining the tag probability distribution dict1 and the meta tag probability distribution dict2, the meta tag probability distribution dict2 is reorganized according to the predetermined dict0 to obtain the reorganized meta tag probability distribution dict3.

[0061] Specifically, it can be expressed as:

[0062] ;

[0063] ;

[0064] Where, is the jth label y j Corresponding meta tag list The number of meta tags, represents the kth meta tag, Represents the probability of the kth meta-tag in the meta-tag probability distribution.

[0065] That is, for each tag, the sum of the probabilities of all the corresponding meta-tags is calculated, and then the ratio of the obtained probability sum to the number of all meta-tags is calculated. All the ratios are integrated to obtain the reorganized meta-tag probability distribution dict3.

[0066] It can be understood that the meta-tag probability distribution is a mapping relationship between meta-tags and probabilities, and the reorganized meta-tag probability distribution is a mapping relationship between tags and probabilities. On this basis, combining dict3 and dict1 can realize the weighting of the reorganized meta-tag probability distribution on the tag probability distribution, and obtain the reordered tag probability distribution dict4:

[0067] ;

[0068] ;

[0069] Where, is the probability of the jth label in the label probability distribution dict1, Represents the probability of the jth tag in the reorganized meta-tag probability distribution dict3, Represents the probability of the jth label in the reordered label probability distribution dict4.

[0070] On this basis, the final labeling results of the exercises to be classified can be determined according to the reordered label probability distribution dict4.

[0071] Optionally, the label with the highest probability in the reordered label probability distribution dict4 is determined as the labeling result for the exercise to be classified.

[0072] Optionally, a probability threshold is set, and labels with probabilities greater than the probability threshold in the reordered label probability distribution dict4 are determined as labeling results for the exercises to be classified. The probability threshold may be an empirical value.

[0073] Optionally, after arranging the probabilities in the reordered label probability distribution dict4 in descending order, the position where the probability gap occurs is determined, and the label corresponding to the probability before the gap is determined as the labeling result for the exercise to be classified.

[0074] The present invention splits semantically rich tags into independent meta-tags, pre-trains tag classification models and meta-tag classification models, and weights and guides the results of tag probability distribution based on meta-tag probability distribution, effectively solving the problem that existing classification algorithms have difficulty in distinguishing highly similar tags, and improving the accuracy of automatic labeling and the ability to recognize long-tail tags.

[0075] In the automatic annotation method for exercise resources of the present invention, before the step of inputting the exercises to be classified into a pre-trained label classification model to obtain the label probability distribution output by the label classification model, the method further includes:

[0076] Using the mathematics texts of the exercise resources as samples and the knowledge points of the mathematics texts as labels to construct a label dataset;

[0077] In order to pre-train the label classification model, we first collect a large number of exercise resources, extract the mathematical text of the exercise resources as samples, and extract the knowledge points of each exercise resource as the label corresponding to the sample to form a "sample-label" pair.

[0078] Optionally, the exercise resource may be an online exercise resource, or may be obtained by scanning a paper resource using an OCR (Optical Character Recognition) technology.

[0079] Perform data cleaning on the collected exercise resources.

[0080] Alternatively, when extracting and cleaning exercise resources written in HTML markup, first use the Re library supported by Python to write regular rules to extract the "sample-label" pairs in the exercises stored in JSON format; replace special characters in the samples, and remove stop words, hyperlinks, and markup language blocks in the samples. Take an exercise as an example:

[0081] {question:“For example: It is known that f(x) is defined in <strong>R< / strong>For odd functions on x≤0, f(x)=frac 2x+a x-1. If the inequality f(x 2 +tx)+ f(\$\\frac9x\$-x)<0 always holds. Find the range of values ​​for t. ", knowledge point: "Problems with the constant validity of functions"}

[0082] After cleaning, it is arranged into the following form:

[0083] sample: “It is known that f(x) is an odd function defined on R. When x≤0, If the inequality Always holds true. Find the range of values ​​of t. ", label: "The problem of function always holding true".

[0084] On this basis, all cleaned samples are segmented by words. The specific method is as follows: the samples are separated word by word according to spaces. The mathematical formulas presented in LaTeX format are segmented using the im2markup tool. The above word segmentation rules are applied to all samples, the word frequency is counted, and words with a word frequency of 1 are filtered out. The words are combined with the Chinese dictionary v0 of BERT-Base-Chinses to form the expanded dictionary v1.

[0085] Pre-training a masked language task on the labeled dataset to obtain a preliminary model;

[0086] After the label dataset is constructed, in order to obtain a label classification model with better classification effect, BERT is first pre-trained using the masked language modeling task on the label dataset. The converged model weights are saved locally and recorded as pre-trained weights weight1, which means a preliminary model is obtained.

[0087] Optionally, the BERT model used as the training basis is the Pytorch-BERT model provided by Transformer.

[0088] The prepared model is fine-tuned on the labeled dataset based on multi-task learning to obtain a label classification model for outputting a label probability distribution.

[0089] Multi-task learning is a machine learning paradigm that allows a model to learn multiple related tasks simultaneously, thereby improving the model's generalization ability.

[0090] Fine-tuning the preliminary model based on multi-task learning requires defining the main task and auxiliary tasks, where the main task is defined as the label classification task, and the auxiliary tasks are other tasks related to label classification. The label probability distribution is obtained through the output of the main task, and by sharing some features of the auxiliary tasks, more accurate label prediction results are obtained.

[0091] Optionally, during fine-tuning of the preparation model based on multi-task learning technology, one or more auxiliary tasks may be selected.

[0092] Optionally, an auxiliary task can be defined as a meta-label prediction task.

[0093] Optionally, the auxiliary task can be defined as an exercise difficulty prediction task, which inputs an exercise and outputs the difficulty level corresponding to the exercise, aiming to learn the correlation between the difficulty of the exercise resource and the knowledge points.

[0094] Optionally, the auxiliary task can be defined as a meta-tag relevance prediction task, which inputs an exercise and outputs possible meta-tag groups. Each meta-tag group includes two meta-tags and the relevance between the two meta-tags, aiming to learn the correlation between the knowledge point of the current tag and other knowledge points.

[0095] After defining appropriate auxiliary tasks, the preliminary model is trained on the labeled dataset based on multi-task learning, and the trained main task model is finally obtained as the label classification model for outputting the label probability distribution.

[0096] Among them, the label data set is pre-divided into training set train1 and test set test1 in a ratio of 8:2. After multiple rounds of training for the prepared model using the training set, test1 is used for testing. The main task model of the training round with the best classification accuracy is used as the trained label classification model, and its fine-tuning weight weight2 is recorded at the same time.

[0097] It should be noted that the labeled dataset is used for training the main task, and the training of the auxiliary task can construct the corresponding auxiliary task dataset according to the defined task type.

[0098] In the automatic annotation method of exercise resources of the present invention, the main task of the multi-task learning is defined as a label classification task, and the auxiliary task is defined as a class center learning task, wherein the class center learning task is a distance-based optimization task used to cluster key features of the input text.

[0099] In this embodiment, the label classification task is selected as the main task, and the class center learning task is selected as the auxiliary task.

[0100] Among them, combined with reference Figure 2 and Figure 3The class center learning task is a distance-based optimization task for clustering key features of text. The specific implementation steps are as follows:

[0101] First, all labels in the label dataset are extracted, and after deduplication, a virtual dataset D2 in the form of "label-label" pairs is constructed, where each data piece is a "label-label" pair.

[0102] Create a learnable initialization class center matrix C0 of shape [n, d] in the classification layer of BERT. Assume that the number of samples in each batch is a multiple of 8, where The samples come from the training set train1, The samples come from the virtual dataset D2.

[0103] BERT loads the pre-trained weights weight1 and performs fine-tuning training based on the label classification task. When its classification accuracy on the virtual dataset D2 exceeds 99%, the feature vector formed by BERT for each virtual sample is saved locally and stored as a pre-trained class center matrix C1 with a shape of [n, d].

[0104] BERT loads the pre-trained weights weight1, copies the locally stored pre-trained class center matrix C1 to the initialized class center matrix C0, and performs fine-tuning training on the training set train1 by combining the label classification task and the class center learning task.

[0105] Among them, the loss function of the class center learning task is defined as:

[0106] ;

[0107] Where M is the number of samples in each batch, e is the feature vector, c is the class center vector, is the Euclidean distance.

[0108] On this basis, the label probability distribution dict1 output by the trained label classification model can be expressed as: , where y j is the jth label, S 1 It is the label confidence score in the interval [0,1], which represents the probability of the label, obtained by passing the feature vector of the exercise through the classifier and activation function.

[0109] The present invention guides the sample feature vector to be close to the class center vector in a complex high-dimensional vector space through the class center learning task, strengthens the closeness between classes, and effectively reduces the number of fuzzy samples on the classification boundary.

[0110] In the automatic tagging method for exercise resources of the present invention, before the step of inputting the exercise to be classified into a pre-trained meta-tag classification model to obtain the meta-tag probability distribution output by the meta-tag classification model, the method further includes:

[0111] Using the mathematics texts of the exercise resources as samples and the knowledge points of the mathematics texts as labels to construct a label dataset;

[0112] Pre-training a masked language task on the labeled dataset to obtain a preliminary model;

[0113] In this embodiment, before training the meta-tag model, a preparatory model is also pre-trained to facilitate BERT to better understand the meta-tag classification task of the exercise resource. Its training method is the same as that used when training the tag classification model, so it will not be repeated here.

[0114] During the actual training process, the model parameters can be recorded after the preliminary model training is completed, and the same preliminary model can be used as the training basis for the label classification model and the meta-label classification model.

[0115] Determine a meta-tag corresponding to each tag based on a predefined division rule, and construct a meta-tag dataset based on the tag dataset;

[0116] Before training the meta-tag classification model, it is necessary to determine the meta-tag corresponding to each tag based on the deduplicated tags extracted from the tag dataset and the pre-defined partitioning rules, and obtain the mapping relationship dict0 between the tag and the meta-tag.

[0117] The labels in the label dataset are mapped through dict0 to obtain the meta-label dataset D3 in the form of "sample-meta-label" pairs.

[0118] The prepared model is fine-tuned on the meta-tag dataset to obtain the meta-tag classification model for outputting the meta-tag probability distribution.

[0119] The meta-label data set D3 is divided into a training set train2 and a test set test2.

[0120] Fine-tune the pre-acquired preliminary model on the meta-tag dataset. The final fine-tuning weight weight3 is based on the best classification accuracy of the test set test2, and also contains the meta-tag probability distribution dict2 of each exercise in the test set test2.

[0121] The meta tag probability distribution dict2 can be expressed as: .

[0122] Where, is the kth meta-tag, m is the total number of meta-tags, S metais the meta tag confidence score, representing the probability of the meta tag, and the calculation method is the same as S 1 .

[0123] The fine-tuning weight weight3 is the parameter corresponding to the trained meta-label classification model.

[0124] In the automatic annotation method for exercise resources of the present invention, the predefined division rules include the following rules 1 to 4:

[0125] Rule 1 is to extract independent phrases of the tag text and decompose them into at least two alternative meta tags;

[0126] Rule 2 is to discard the alternative meta tags that appear alone in any tag;

[0127] Rule 3 is to discard candidate meta tags whose occurrence frequency is greater than a preset threshold, where the occurrence frequency is the ratio of the total number of occurrences of candidate meta tags to the total number of tags;

[0128] Rule 4 is that when the tag text cannot be decomposed into at least two candidate meta tags, the candidate meta tags are determined according to the knowledge points corresponding to the tags.

[0129] In order to accurately obtain meta-tags based on tag extraction, the above four rules are defined in this embodiment.

[0130] Among them, rule 1 stipulates that a tag corresponds to at least two meta tags. To this end, the tag is segmented and multiple independent phrases obtained by segmentation are extracted as candidate meta tags for the tag.

[0131] Rule 2 limits the frequency of meta tags. Meta tags should appear in other tags as much as possible and avoid appearing alone in the current tag. Therefore, alternative meta tags that appear alone in the current tag are discarded, so that all retained alternative meta tags appear in multiple tags.

[0132] Rule three also limits the frequency of meta-tags. The preset threshold is determined based on experience and is set to 10% in this embodiment. If an alternative meta-tag appears in more than 10% of the tags, it is judged as a common and difficult to distinguish between different tags and should be discarded.

[0133] Rule 4 is a supplement to meta tags. When the tag text cannot be decomposed into at least two alternative meta tags, that is, when it is difficult to decompose the meta tags it contains from the tag name alone, the alternative meta tags should be determined based on the knowledge points to which the tags belong. Specifically, the relevant definition theorems in the textbooks should be consulted to include the most commonly used concepts.

[0134] Based on the above method, the candidate meta tags obtained by tag decomposition are screened and supplemented, and the final candidate meta tags are used as meta tags. Based on this, the mapping relationship dict0 between tags and meta tags is constructed:

[0135] ;

[0136] In the formula, y represents the label, n is the label always, y meta is a meta tag, is the list of meta tags corresponding to the jth tag, and m is the total number of all meta tags.

[0137] The constructed part of dict0 is shown in Table 1 below:

[0138] Table 1

[0139]

[0140] In the automatic annotation method for exercise resources of the present invention, the step of determining the annotation results of the exercises to be classified based on the weighted results specifically includes:

[0141] Determining a confidence score for each tag in the tag probability distribution according to the weighted result;

[0142] In this embodiment, the reorganized meta-tag probability distribution is weighted to the tag probability distribution to obtain a reordered tag probability distribution dict4. Optionally, the probability corresponding to each tag in dict4 is directly used as the confidence score of each tag.

[0143] Sorting the labels in the label probability distribution in descending order of the confidence scores to obtain a reordered label probability distribution;

[0144] The labels with confidence scores greater than a preset score threshold in the reordered label probability distribution are used as the labeling results of the exercises to be classified.

[0145] Sort the labels in dict4 in descending order of confidence scores, and put the confidence scores s r The labels corresponding to scores greater than the preset score threshold are used as the annotation results for the classification exercises.

[0146] Optionally, the preset score threshold is an empirical value, which is set to 0.5 in this embodiment.

[0147] It should be noted that, in the verification using the test set, it was found that the labels finally obtained through this embodiment have a labeling accuracy greater than 90% when the label with the highest probability and the meta-label point to the same label in the label probability distribution and meta-label probability distribution output by the label classification model and the meta-label classification model.

[0148] The automatic annotation device for exercise resources provided by the present invention is described below. The automatic annotation device for exercise resources described below and the automatic annotation method for exercise resources described above can be referenced to each other.

[0149] like Figure 4 As shown, the automatic annotation device for exercise resources includes a label probability distribution module 401, a meta-label probability distribution module 402 and an annotation module 403;

[0150] The label probability distribution module 401 is used to input the exercises to be classified into the pre-trained label classification model to obtain the label probability distribution output by the label classification model;

[0151] Exercises to be classified are exercise resources that need to have their label categories determined. The label of an exercise to be classified is the knowledge point to which the exercise belongs, which is represented by a short sentence containing multiple independent phrases.

[0152] Using the BERT model as the initial model, a label classification model is obtained through multiple rounds of training on a dataset in the form of "mathematical text-label". The label classification model outputs each label and its corresponding probability as a label probability distribution.

[0153] It is understandable that the label probability distribution output by the pre-trained label classification model is obtained by training the model based on a dataset in the form of "mathematical text-label". Therefore, it is less sensitive to some labels with high semantic overlap. For example, "Practical Application of Addition and Subtraction of Integers" and "Column Algebraic Expressions and Practical Problems" have similar label semantic overlap and mathematical text performance, which makes the single label probability model less discriminative for such labels.

[0154] Step 102: Input the to-be-classified exercise into a pre-trained meta-tag classification model to obtain a meta-tag probability distribution output by the meta-tag classification model, wherein a meta-tag is an independent phrase determined based on the tag that characterizes the uniqueness of the tag;

[0155] Based on the above reasons, the BERT model is used as the initial model. A meta-tag classification model is obtained through multiple rounds of training on a dataset in the form of "mathematical text-meta-tag". The meta-tag classification model outputs each meta-tag and its corresponding probability as the meta-tag probability distribution.

[0156] The meta-tags are predetermined based on the tags of each exercise resource and are used to represent the part of each tag that is more unique than other tags.

[0157] Taking the tag "Application of inequality properties" as an example, it includes three independent phrases: "inequality", "property" and "application". Among them, the phrase "application" refers to the practical application of the knowledge point of inequality properties. Most knowledge points involve practical application problems, which means that a large number of tags contain "application". Therefore, "application" is discarded, and "inequality" and "property" are determined as the meta-tags of "Application of inequality properties".

[0158] On this basis, the meta-tag probability distribution obtained by using the meta-tag classification model has a better understanding of tags with high semantic overlap. On the other hand, for some long-tail tags, there are meta-tags among the multiple meta-tags determined that represent more exercise resources than the long-tail tags themselves, thereby improving the fitting effect of long-tail tags to a certain extent.

[0159] A meta-tag probability distribution module 402 is configured to input the to-be-classified exercise into a pre-trained meta-tag classification model to obtain a meta-tag probability distribution output by the meta-tag classification model, wherein a meta-tag is an independent phrase determined based on a tag that characterizes the uniqueness of the tag;

[0160] Based on the above reasons, the BERT model is used as the initial model. A meta-tag classification model is obtained through multiple rounds of training on a dataset in the form of "mathematical text-meta-tag". The meta-tag classification model outputs each meta-tag and its corresponding probability as the meta-tag probability distribution.

[0161] The meta-tags are predetermined based on the tags of each exercise resource and are used to represent the part of each tag that is more unique than other tags.

[0162] Taking the tag "Application of inequality properties" as an example, it includes three independent phrases: "inequality", "property" and "application". Among them, the phrase "application" refers to the practical application of the knowledge point of inequality properties. Most knowledge points involve practical application problems, which means that a large number of tags contain "application". Therefore, "application" is discarded, and "inequality" and "property" are determined as the meta-tags of "Application of inequality properties".

[0163] On this basis, the meta-tag probability distribution obtained by using the meta-tag classification model has a better understanding of tags with high semantic overlap. On the other hand, for some long-tail tags, there are meta-tags among the multiple meta-tags determined that represent more exercise resources than the long-tail tags themselves, thereby improving the fitting effect of long-tail tags to a certain extent.

[0164] The labeling module 403 is configured to reorganize the meta-tag probability distribution based on the mapping relationship between the tags and the meta-tags, weight the tag probability distribution using the reorganized meta-tag probability distribution, and determine the labeling result of the to-be-classified exercise according to the weighted result.

[0165] Since one tag corresponds to multiple meta-tags, before using the meta-tag probability distribution to weight the tag probability distribution, the meta-tag probability distribution needs to be reorganized according to the mapping relationship between the tag and the meta-tag.

[0166] It can be understood that the mapping relationship dict0 between tags and meta tags is constructed when the meta tag corresponding to each tag is determined.

[0167] In a specific embodiment, Figure 2 As shown, after obtaining the tag probability distribution dict1 and the meta tag probability distribution dict2, the meta tag probability distribution dict2 is reorganized according to the predetermined dict0 to obtain the reorganized meta tag probability distribution dict3.

[0168] Specifically, it can be expressed as:

[0169] ;

[0170] ;

[0171] Where, is the jth label y j Corresponding meta tag list The number of meta tags, represents the kth meta tag, Represents the probability of the kth meta-tag in the meta-tag probability distribution.

[0172] That is, for each tag, the sum of the probabilities of all the corresponding meta-tags is calculated, and then the ratio of the obtained probability sum to the number of all meta-tags is calculated. All the ratios are integrated to obtain the reorganized meta-tag probability distribution dict3.

[0173] It can be understood that the meta-tag probability distribution is a mapping relationship between meta-tags and probabilities, and the reorganized meta-tag probability distribution is a mapping relationship between tags and probabilities. On this basis, combining dict3 and dict1 can realize the weighting of the reorganized meta-tag probability distribution on the tag probability distribution, and obtain the reordered tag probability distribution dict4:

[0174] ;

[0175] ;

[0176] Where, is the probability of the jth label in the label probability distribution dict1, Represents the probability of the jth tag in the reorganized meta-tag probability distribution dict3, Represents the probability of the jth label in the reordered label probability distribution dict4.

[0177] On this basis, the final labeling results of the exercises to be classified can be determined according to the reordered label probability distribution dict4.

[0178] Optionally, the label with the highest probability in the reordered label probability distribution dict4 is determined as the labeling result for the exercise to be classified.

[0179] Optionally, a probability threshold is set, and labels with probabilities greater than the probability threshold in the reordered label probability distribution dict4 are determined as labeling results for the exercises to be classified. The probability threshold may be an empirical value.

[0180] Optionally, after arranging the probabilities in the reordered label probability distribution dict4 in descending order, the position where the probability gap occurs is determined, and the label corresponding to the probability before the gap is determined as the labeling result for the exercise to be classified.

[0181] The present invention splits semantically rich tags into independent meta-tags, pre-trains tag classification models and meta-tag classification models, and weights and guides the results of tag probability distribution based on meta-tag probability distribution, effectively solving the problem that existing classification algorithms have difficulty in distinguishing highly similar tags, and improving the accuracy of automatic labeling and the ability to recognize long-tail tags.

[0182] Figure 5 An example of a physical structure diagram of an electronic device is shown below. Figure 5As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 may call logic instructions in the memory 530 to execute an automatic annotation method for exercise resources, the method comprising: inputting the exercise to be classified into a pre-trained label classification model to obtain a label probability distribution output by the label classification model; inputting the exercise to be classified into a pre-trained meta-label classification model to obtain a meta-tag probability distribution output by the meta-label classification model, wherein a meta-tag is an independent phrase that characterizes the uniqueness of the tag determined based on the tag; reorganizing the meta-tag probability distribution based on the mapping relationship between the tag and the meta-tag, weighting the tag probability distribution using the reorganized meta-tag probability distribution, and determining the annotation result of the exercise to be classified based on the weighted result.

[0183] Furthermore, the logic instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0184] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the automatic labeling method of exercise resources provided by the above methods, which method includes: inputting the exercises to be classified into a pre-trained label classification model to obtain the label probability distribution output by the label classification model; inputting the exercises to be classified into a pre-trained meta-label classification model to obtain the meta-label probability distribution output by the meta-label classification model, wherein the meta-label is an independent phrase determined based on the label to characterize the uniqueness of the label; reorganizing the meta-label probability distribution based on the mapping relationship between the label and the meta-label, using the reorganized meta-label probability distribution to weight the label probability distribution, and determining the labeling result of the exercises to be classified based on the weighted result.

[0185] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for automatically labeling exercise resources provided by the above-mentioned methods, the method comprising: inputting the exercises to be classified into a pre-trained label classification model to obtain a label probability distribution output by the label classification model; inputting the exercises to be classified into a pre-trained meta-label classification model to obtain a meta-tag probability distribution output by the meta-label classification model, wherein the meta-tag is an independent phrase characterizing the uniqueness of the tag determined based on the tag; reorganizing the meta-tag probability distribution based on the mapping relationship between the tag and the meta-tag, using the reorganized meta-tag probability distribution to weight the tag probability distribution, and determining the labeling result of the exercises to be classified based on the weighted result.

[0186] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0187] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0188] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for automatically marking exercise resources, characterized in that: include: Inputting the exercises to be classified into a pre-trained label classification model to obtain the label probability distribution output by the label classification model; Inputting the to-be-classified exercise into a pre-trained meta-tag classification model to obtain a meta-tag probability distribution output by the meta-tag classification model, wherein the meta-tag is an independent phrase determined based on the tag that characterizes the uniqueness of the tag; Reorganizing the meta-tag probability distribution based on the mapping relationship between the tag and the meta-tag, weighting the tag probability distribution using the reorganized meta-tag probability distribution, and determining the labeling result of the to-be-classified exercise according to the weighted result; Before the step of inputting the to-be-classified exercises into a pre-trained meta-tag classification model to obtain the meta-tag probability distribution output by the meta-tag classification model, the method further includes: Using the mathematics texts of the exercise resources as samples and the knowledge points of the mathematics texts as labels to construct a label dataset; Determine a meta-tag corresponding to each tag based on a predefined division rule, and construct a meta-tag dataset based on the tag dataset; Pre-training a masked language task on the labeled dataset to obtain a preliminary model; Fine-tuning the prepared model on the meta-tag dataset to obtain the meta-tag classification model for outputting the meta-tag probability distribution; The predefined division rules include the following rules 1 to 4: Rule 1 is to extract independent phrases of the tag text and decompose them into at least two alternative meta tags; Rule 2 is to discard the alternative meta tags that appear alone in any tag; Rule 3 is to discard candidate meta tags whose occurrence frequency is greater than a preset threshold, where the occurrence frequency is the ratio of the total number of occurrences of candidate meta tags to the total number of tags; Rule 4 is that when the tag text cannot be decomposed into at least two candidate meta tags, the candidate meta tags are determined according to the knowledge points corresponding to the tags.

2. The automatic annotation method for exercise resources according to claim 1, characterized in that: Before the step of inputting the exercises to be classified into the pre-trained label classification model to obtain the label probability distribution output by the label classification model, the method further includes: Using the mathematics texts of the exercise resources as samples and the knowledge points of the mathematics texts as labels to construct a label dataset; Pre-training a masked language task on the labeled dataset to obtain a preliminary model; The prepared model is fine-tuned on the labeled dataset based on multi-task learning to obtain a label classification model for outputting a label probability distribution.

3. The automatic annotation method for exercise resources according to claim 2, characterized in that: The main task of the multi-task learning is defined as a label classification task, and the auxiliary task is defined as a class center learning task, wherein the class center learning task is a distance-based optimization task for clustering key features of the input text.

4. The automatic annotation method for exercise resources according to claim 1, characterized in that: The step of determining the marking results of the exercises to be classified according to the weighted results specifically includes: Determining a confidence score for each tag in the tag probability distribution according to the weighted result; Sorting the labels in the label probability distribution in descending order of the confidence scores to obtain a reordered label probability distribution; The labels with confidence scores greater than a preset score threshold in the reordered label probability distribution are used as the labeling results of the exercises to be classified.

5. An automatic annotation device for exercise resources, characterized in that: include: A label probability distribution module is used to input the exercises to be classified into a pre-trained label classification model to obtain the label probability distribution output by the label classification model; a meta-tag probability distribution module, configured to input the to-be-classified exercise into a pre-trained meta-tag classification model to obtain a meta-tag probability distribution output by the meta-tag classification model, wherein a meta-tag is an independent phrase determined based on a tag and representing the uniqueness of the tag; a labeling module, configured to reorganize the meta-tag probability distribution based on a mapping relationship between the tags and the meta-tags, weight the tag probability distribution using the reorganized meta-tag probability distribution, and determine a labeling result for the to-be-classified exercise according to the weighted result; It also includes a pre-training module for constructing a labeled data set using the mathematical text of the exercise resource as a sample and the knowledge points of the mathematical text as labels; Determine a meta-tag corresponding to each tag based on a predefined division rule, and construct a meta-tag dataset based on the tag dataset; Pre-training a masked language task on the labeled dataset to obtain a preliminary model; Fine-tuning the prepared model on the meta-tag dataset to obtain the meta-tag classification model for outputting the meta-tag probability distribution; The predefined division rules include the following rules 1 to 4: Rule 1 is to extract independent phrases of the tag text and decompose them into at least two alternative meta tags; Rule 2 is to discard the alternative meta tags that appear alone in any tag; Rule 3 is to discard candidate meta tags whose occurrence frequency is greater than a preset threshold, where the occurrence frequency is the ratio of the total number of occurrences of candidate meta tags to the total number of tags; Rule 4 is that when the tag text cannot be decomposed into at least two candidate meta tags, the candidate meta tags are determined according to the knowledge points corresponding to the tags.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the automatic annotation method for exercise resources according to any one of claims 1 to 4 is implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for automatically marking exercise resources according to any one of claims 1 to 4 is implemented.

8. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for automatically marking exercise resources according to any one of claims 1 to 4 is implemented.

Images