Automatic annotation method and device for exercise resources

Abstract

The invention provides an exercise resource automatic labeling method and device, and the method comprises the steps: inputting to-be-classified exercises into a pre-trained label classification model, and obtaining the label probability distribution outputted by the label classification model; the to-be-classified exercises are input into a pre-trained meta-label classification model, meta-label probability distribution output by the meta-label classification model is obtained, and meta-labels are independent phrases which are determined based on labels and represent the uniqueness of the labels; and recombining the meta-label probability distribution based on the mapping relationship between the labels and the meta-labels, weighting the label probability distribution by using the recombined meta-label probability distribution, and determining a labeling result of the to-be-classified exercises according to a weighting result. According to the method, the labels with rich semantics are split into independent meta-labels, so that the problem that high-similarity labels are difficult to distinguish by an existing classification algorithm is effectively solved, and the accuracy of automatic labeling and the recognition capability of long-tail labels are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to an automatic annotation method and device for exercise resources. Background Art

[0002] With the rapid development of artificial intelligence technology, the field of online education has ushered in a booming development opportunity, including but not limited to automatic problem solving, exercise recommendation, difficulty prediction and automatic test paper generation. In a real industrial production environment, in order to achieve accurate push of exercise resources and ensure that the exercise content matches the students' knowledge level, exercise labels are essential.

[0003] Manual labeling by educational experts is time-consuming and biased, so there is an urgent need for an automatic labeling method that can not only automatically label exercises but also give high-precision prediction results. Summary of the invention

[0004] The present invention provides an automatic marking method and device for exercise resources, which are used to solve the defect of the prior art that manual marking of exercise resources is troublesome, and realize a high-precision automatic marking method for exercise resources.

[0005] The present invention provides an automatic annotation method for exercise resources, comprising: Inputting the to-be-classified exercises into a pre-trained label classification model to obtain a label probability distribution output by the label classification model; Inputting the to-be-classified exercise into a pre-trained meta-tag classification model to obtain a meta-tag probability distribution output by the meta-tag classification model, wherein the meta-tag is an independent phrase that characterizes the uniqueness of the tag and is determined based on the tag; The meta-tag probability distribution is reorganized based on the mapping relationship between the tag and the meta-tag, the tag probability distribution is weighted using the reorganized meta-tag probability distribution, and the labeling result of the to-be-classified exercise is determined according to the weighted result.

[0006] According to an automatic annotation method for exercise resources provided by the present invention, before the step of inputting the exercises to be classified into a pre-trained label classification model to obtain the label probability distribution output by the label classification model, the method further includes: Using the mathematics text of the exercise resource as a sample and the knowledge points of the mathematics text as labels to construct a label data set; Pre-training a masked language task on the labeled dataset to obtain a preliminary model; The prepared model is fine-tuned on the labeled data set based on multi-task learning to obtain a label classification model for outputting label probability distribution.

[0007] According to an automatic annotation method for exercise resources provided by the present invention, the main task of the multi-task learning is defined as a label classification task, and the auxiliary task is defined as a class center learning task, wherein the class center learning task is a distance-based optimization task for clustering key features of the input text.

[0008] According to an automatic annotation method for exercise resources provided by the present invention, before the step of inputting the exercise to be classified into a pre-trained meta-tag classification model to obtain the meta-tag probability distribution output by the meta-tag classification model, the method further includes: Using the mathematics text of the exercise resource as a sample and the knowledge points of the mathematics text as labels to construct a label data set; Determine a meta tag corresponding to each tag based on a predefined division rule, and construct a meta tag dataset based on the tag dataset; Pre-training a masked language task on the labeled dataset to obtain a preliminary model; The prepared model is fine-tuned on the meta-tag dataset to obtain the meta-tag classification model for outputting the meta-tag probability distribution.

[0009] According to an automatic annotation method for exercise resources provided by the present invention, the predefined division rules include the following rules 1 to 4: Rule 1 is to extract independent phrases of the tag text and decompose them into at least two candidate meta tags; Rule 2 is to discard the alternative meta tags that appear alone in any tag; Rule three is to discard candidate meta tags whose occurrence frequency is greater than a preset threshold, where the occurrence frequency is the ratio of the total number of occurrences of the candidate meta tags to the total number of tags; Rule 4 is that when the tag text cannot be decomposed into at least two candidate meta tags, the candidate meta tags are determined according to the knowledge points corresponding to the tags.

[0010] According to an automatic annotation method for exercise resources provided by the present invention, the step of determining the annotation result of the exercise to be classified according to the weighted result specifically includes: Determining a confidence score for each tag in the tag probability distribution according to the weighted result; Sorting the labels in the label probability distribution in descending order of the confidence scores to obtain a reordered label probability distribution; The labels whose confidence scores in the reordered label probability distribution are greater than a preset score threshold are used as the labeling results of the exercises to be classified.

[0011] The present invention also provides an automatic annotation device for exercise resources, comprising: A label probability distribution module is used to input the to-be-classified exercises into a pre-trained label classification model to obtain the label probability distribution output by the label classification model; A meta-tag probability distribution module, used for inputting the to-be-classified exercise into a pre-trained meta-tag classification model to obtain a meta-tag probability distribution output by the meta-tag classification model, wherein the meta-tag is an independent phrase that characterizes the uniqueness of the tag and is determined based on the tag; The labeling module is used to reorganize the meta-tag probability distribution based on the mapping relationship between the tag and the meta-tag, use the reorganized meta-tag probability distribution to weight the tag probability distribution, and determine the labeling result of the to-be-classified exercise according to the weighted result.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the automatic marking method for exercise resources as described above is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for automatically marking exercise resources as described in any one of the above is implemented.

[0014] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned methods for automatically marking exercise resources.

[0015] The automatic annotation method and device for exercise resources provided by the present invention split semantically rich tags into independent meta-tags, and pre-train to obtain tag classification models and meta-tag classification models, and weightedly guide the results of tag probability distribution based on meta-tag probability distribution, thereby effectively solving the problem that existing classification algorithms are difficult to distinguish high-similarity tags, and improving the accuracy of automatic annotation and the ability to recognize long-tail tags. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is one of the flow charts of the automatic annotation method of exercise resources provided by the present invention; Figure 2 This is the second flow chart of the automatic annotation method for exercise resources provided by the present invention; Figure 3 This is the third flow chart of the automatic marking method for exercise resources provided by the present invention; Figure 4 It is a structural schematic diagram of the automatic marking device for exercise resources provided by the present invention; Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] The exercise resources consist of mathematical texts and geometric pictures. The mathematical texts include descriptive texts and mathematical formulas. Labeling the knowledge points of the exercise resources belongs to the multi-label classification text task.

[0020] In view of the characteristics of mathematical texts with multiple formulas, traditional machine methods extract the features of mathematical texts through techniques such as n-gram distribution (a statistical language model) frequency of mathematical symbols, structural kernels, and TF-IDF (term frequency-inverse document frequency), and then use naive Bayes or SVM (Support Vector Machine) for classification. With the development of deep learning, encoders that automatically capture features, such as CNN (Convolutional Neural Network) and LSTM (Long Short-Term Memory), have shown superior performance in processing complex text data. Pre-trained language models represented by BERT (Bidirectional Encoder Representations from Transformers, a pre-trained language model) are often used to design text classification tasks.

[0021] However, due to the uneven distribution of knowledge points and exam focuses, there are a large number of long-tail tags with low sample frequency in the exercise resources in the online education field, and the exercises are difficult to distinguish because some tags have high semantic overlap, making it difficult to effectively improve the automatic annotation of exercises.

[0022] To this end, this embodiment proposes a method for annotating exercise resources by combining tags and meta-tags, so as to overcome the under-fitting problem of long-tail tags to a certain extent, and at the same time improve the accuracy of semantic distinction of tags with high semantic overlap.

[0023] Combine the following Figures 1 to 3 The automatic annotation method of the exercise resource of the present invention is introduced, such as Figure 1 As shown, the method includes: Step 101, inputting the exercises to be classified into a pre-trained label classification model to obtain the label probability distribution output by the label classification model; The exercises to be classified are the exercise resources whose label categories need to be determined. The label of the exercises to be classified is the knowledge point to which the exercises to be classified belong, which is expressed as a short sentence containing multiple independent phrases.

[0024] Taking the BERT model as the initial model, a label classification model is obtained through multiple rounds of training on a dataset in the form of "mathematical text-label". The label classification model outputs each label and its corresponding probability as the label probability distribution.

[0025] It is understandable that the label probability distribution output by the pre-trained label classification model is obtained by training the model based on a data set in the form of "mathematical text-label". Therefore, it is less sensitive to some labels with high semantic overlap, such as "practical application of addition and subtraction of integers" and "column algebraic expressions and practical problems". The label semantic overlap and the performance of mathematical text are relatively similar, which makes the single label probability model have low discrimination for such labels.

[0026] Step 102, inputting the to-be-classified exercise into a pre-trained meta-tag classification model to obtain a meta-tag probability distribution output by the meta-tag classification model, wherein a meta-tag is an independent phrase that characterizes the uniqueness of the tag and is determined based on the tag; Based on the above reasons, the BERT model is used as the initial model, and a meta-tag classification model is obtained through multiple rounds of training on a dataset in the form of "mathematical text-meta-tag". The meta-tag classification model outputs each meta-tag and its corresponding probability as the meta-tag probability distribution.

[0027] The meta-tags are predetermined based on the tags of each exercise resource and are used to characterize the part of each tag that is more unique than other tags.

[0028] Taking the tag "Application of inequality properties" as an example, it includes three independent phrases "inequality", "property" and "application". Among them, the phrase "application" refers to the practical application of the knowledge point of inequality properties. Most knowledge points involve practical application problems, which makes a large number of tags contain "application". Therefore, "application" is discarded and "inequality" and "property" are determined as the meta tags of "application of inequality properties".

[0029] On this basis, the meta-tag probability distribution obtained by using the meta-tag classification model has a better understanding of tags with high semantic overlap on the one hand, and on the other hand, for some long-tail tags, there are meta-tags among the multiple meta-tags determined that represent more exercise resources than the long-tail tags themselves, thereby improving the fitting effect of long-tail tags to a certain extent.

[0030] Step 103, reorganizing the meta-tag probability distribution based on the mapping relationship between the tag and the meta-tag, using the reorganized meta-tag probability distribution to weight the tag probability distribution, and determining the labeling result of the to-be-classified exercise according to the weighted result.

[0031] Since one tag corresponds to multiple meta-tags, before using the meta-tag probability distribution to weight the tag probability distribution, the meta-tag probability distribution needs to be reorganized according to the mapping relationship between the tag and the meta-tag.

[0032] It can be understood that the mapping relationship dict0 between tags and meta tags is constructed when the meta tag corresponding to each tag is determined.

[0033] In a specific embodiment, Figure 2 As shown, after obtaining the tag probability distribution dict1 and the meta tag probability distribution dict2, the meta tag probability distribution dict2 is reorganized according to the predetermined dict0 to obtain the reorganized meta tag probability distribution dict3.

[0034] Specifically, it can be expressed as: ; ; In the formula, is the jth label y j Corresponding meta tag list The number of meta tags, represents the kth meta tag, Represents the probability of the kth meta tag in the meta tag probability distribution.

[0035] That is to say, for each tag, the probability sum of all meta tags corresponding to it is calculated, and then the ratio of the obtained probability sum to the number of all meta tags is calculated, and all ratios are integrated to obtain the reorganized meta tag probability distribution dict3.

[0036] It can be understood that the meta-tag probability distribution is the mapping relationship between meta-tags and probabilities, and the reorganized meta-tag probability distribution is the mapping relationship between tags and probabilities. On this basis, by combining dict3 and dict1, the reorganized meta-tag probability distribution can be weighted on the tag probability distribution to obtain the reordered tag probability distribution dict4: ; ; In the formula, is the probability of the jth label in the label probability distribution dict1, represents the probability of the jth tag in the reorganized meta-tag probability distribution dict3, Represents the probability of the jth label in the reordered label probability distribution dict4.

[0037] On this basis, the final labeling results of the exercises to be classified can be determined according to the reordered label probability distribution dict4.

[0038] Optionally, the label with the highest probability in the reordered label probability distribution dict4 is determined as the labeling result for the exercise to be classified.

[0039] Optionally, a probability threshold is set, and labels with probabilities greater than the probability threshold in the reordered label probability distribution dict4 are determined as the labeling results for the exercises to be classified. The probability threshold may be an empirical value.

[0040] Optionally, after arranging the probabilities in the reordered label probability distribution dict4 in descending order, the position where the probability gap occurs is determined, and the label corresponding to the probability before the gap is determined as the labeling result for the exercise to be classified.

[0041] The present invention splits semantically rich tags into independent meta-tags, pre-trains tag classification models and meta-tag classification models, and weights and guides the results of tag probability distribution based on meta-tag probability distribution, effectively solving the problem that existing classification algorithms are difficult to distinguish high-similarity tags, and improving the accuracy of automatic labeling and the ability to recognize long-tail tags.

[0042] In the automatic annotation method of exercise resources of the present invention, before the step of inputting the exercises to be classified into the pre-trained label classification model to obtain the label probability distribution output by the label classification model, the method further includes: Using the mathematics text of the exercise resource as a sample and the knowledge points of the mathematics text as labels to construct a label data set; In order to pre-train the label classification model, we first collect a large number of exercise resources, extract the mathematical texts of the exercise resources as samples, and extract the knowledge points of each exercise resource as the labels corresponding to the samples to form a "sample-label" pair.

[0043] Optionally, the exercise resource may be an online exercise resource, or may be obtained by scanning a paper resource using an OCR (Optical Character Recognition) technology.

[0044] Perform data cleaning on the collected exercise resources.

[0045] Optionally, when extracting and cleaning exercise resources written in HTML markup language, first use the Re library supported by Python to write regular rules to extract the "sample-label" pairs in the exercises stored in json format; replace special characters in the samples, and remove stop words, hyperlinks and markup language blocks in the samples. Take a certain exercise as an example: {question:“For example: given that f(x) is defined in <strong>R< / strong> Odd function on, when x≤0, \$f(x)=\\frac 2x+a x-1\$. If the inequality f(x 2 +tx)+ f(\$\\frac9x\$-x)<0 always holds true, find the value range of t. ", knowledge point: "The problem of the function always holding true"}.

[0046] After cleaning, it is arranged into the following form: sample: “It is known that f(x) is an odd function defined on R. When x≤0, If the inequality Always holds true, find the range of t. ", label: "The problem of function always holding true"}.

[0047] On this basis, all cleaned samples are segmented. The specific method is as follows: the samples are separated word by word according to spaces, and the mathematical formula part presented in LaTeX form is divided using the im2markup tool; the above word segmentation rules are implemented on all samples, the word frequency is counted, and the words with a word frequency of 1 are filtered out, and combined with the Chinese dictionary v0 of BERT-Base-Chinses to form the expanded dictionary v1.

[0048] Pre-training a masked language task on the labeled dataset to obtain a preliminary model; After the label data set is constructed, in order to obtain a label classification model with better classification effect, BERT is first pre-trained using the masked language modeling task on the label data set, and the converged model weights are saved locally and recorded as pre-training weights weight1, that is, a preliminary model is obtained.

[0049] Optionally, the BERT model used as the training basis uses the Pytorch-BERT model provided by Transformer.

[0050] The prepared model is fine-tuned on the labeled data set based on multi-task learning to obtain a label classification model for outputting label probability distribution.

[0051] Multi-task learning is a machine learning paradigm that allows a model to learn multiple related tasks simultaneously, thereby improving the model's generalization ability.

[0052] Fine-tuning the preliminary model based on multi-task learning requires defining the main task and auxiliary tasks, where the main task is defined as the label classification task, and the auxiliary tasks are other tasks related to label classification, so as to obtain the label probability distribution through the output of the main task, and obtain more accurate label prediction results by sharing some features of the auxiliary tasks.

[0053] Optionally, during fine-tuning of the preparation model based on multi-task learning technology, one or more auxiliary tasks may be selected.

[0054] Optionally, an auxiliary task can be defined as a meta-label prediction task.

[0055] Optionally, the auxiliary task can be defined as an exercise difficulty prediction task, which inputs an exercise and outputs the difficulty level corresponding to the exercise, aiming to learn the correlation between the difficulty of the exercise resource and the knowledge points.

[0056] Optionally, the auxiliary task can be defined as a meta-tag relevance prediction task, which inputs an exercise and outputs possible meta-tag groups, each of which includes two meta-tags and the relevance between the two meta-tags, aiming to learn the correlation between the knowledge point of the current tag and other knowledge points.

[0057] After defining appropriate auxiliary tasks, the preliminary model is trained on the labeled dataset based on multi-task learning, and the trained main task model is finally obtained as the label classification model for outputting label probability distribution.

[0058] Among them, the label data set is pre-divided into training set train1 and test set test1 in a ratio of 8:2. After multiple rounds of training for the prepared model using the training set, test1 is used for testing, and the main task model of the training round with the best classification accuracy is used as the trained label classification model, and its fine-tuning weight weight2 is recorded at the same time.

[0059] It should be noted that the labeled dataset is used for the training of the main task, and the training of the auxiliary task can construct the corresponding auxiliary task dataset according to the defined task type.

[0060] In the automatic annotation method of exercise resources of the present invention, the main task of the multi-task learning is defined as a label classification task, and the auxiliary task is defined as a class center learning task, wherein the class center learning task is a distance-based optimization task used to gather key features of the input text.

[0061] In this implementation, the label classification task is selected as the main task, and the class center learning task is selected as the auxiliary task.

[0062] Among them, combined with reference Figure 2 and Figure 3 The class center learning task is a distance-based optimization task for clustering key features of text. The specific implementation steps are as follows: First, all labels in the label data set are extracted, and after deduplication, a virtual data set D2 in the form of "label-label" pairs is constructed, where each piece of data is a "label-label" pair.

[0063] Create a learnable initialization class center matrix C0 of shape [n, d] in the classification layer of BERT. Assume that the number of samples in each batch is a multiple of 8, where The samples come from the training set train1, The samples come from the virtual dataset D2.

[0064] BERT loads the pre-trained weight weight1 and performs fine-tuning training according to the label classification task. When its classification accuracy on the virtual dataset D2 exceeds 99%, the feature vector formed by BERT for each virtual sample is saved locally and stored as a pre-trained class center matrix C1 with a shape of [n, d].

[0065] BERT loads the pre-trained weight weight1, copies the locally stored pre-trained class center matrix C1 to the initialized class center matrix C0, and performs fine-tuning training on the training set train1 by combining the label classification task and the class center learning task.

[0066] Among them, the loss function of the class center learning task is defined as: ; In the formula, M is the number of samples in each batch, e is the feature vector, c is the class center vector, is the Euclidean distance.

[0067] On this basis, the label probability distribution dict1 output by the trained label classification model can be expressed as: , where y j is the jth label, S 1 It is a label confidence score in the interval [0,1], which represents the probability of the label, obtained by passing the feature vector of the exercise through the classifier and activation function.

[0068] The present invention guides the sample feature vector to approach the class center vector in a complex high-dimensional vector space through the class center learning task, strengthens the closeness between classes, and effectively reduces the number of fuzzy samples on the classification boundary.

[0069] In the automatic annotation method for exercise resources of the present invention, before the step of inputting the exercise to be classified into a pre-trained meta-tag classification model to obtain the meta-tag probability distribution output by the meta-tag classification model, the method further includes: Using the mathematics text of the exercise resource as a sample and the knowledge points of the mathematics text as labels to construct a label data set; Pre-training a masked language task on the labeled dataset to obtain a preliminary model; In this implementation, before training the meta-tag model, a preparatory model is also pre-trained to facilitate BERT to better understand the meta-tag classification task of the exercise resource. The training method is the same as that used when training the tag classification model, so it will not be repeated here.

[0070] During the actual training process, the model parameters can be recorded after the preliminary model training is completed, and the same preliminary model can be used as the training basis for the label classification model and the meta-label classification model.

[0071] Determine a meta tag corresponding to each tag based on a predefined division rule, and construct a meta tag dataset based on the tag dataset; Before training the meta-tag classification model, it is also necessary to determine the meta-tag corresponding to each tag based on the deduplicated tags extracted from the tag dataset and the pre-defined partitioning rules, and obtain the mapping relationship between the tag and the meta-tag dict0.

[0072] The labels in the label dataset are mapped through dict0 to obtain the meta-label dataset D3 in the form of "sample-meta-label" pairs.

[0073] The prepared model is fine-tuned on the meta-tag dataset to obtain the meta-tag classification model for outputting the meta-tag probability distribution.

[0074] Divide the meta-label data set D3 into training set train2 and test set test2.

[0075] The pre-acquired preliminary model is fine-tuned on the meta-label dataset, and the final fine-tuning weight weight3 is based on the best classification accuracy of the test set test2, and also contains the meta-label probability distribution dict2 of each exercise in the test set test2.

[0076] The meta tag probability distribution dict2 can be expressed as: .

[0077] In the formula, is the kth meta tag, m is the total number of meta tags, S meta is the meta tag confidence score, representing the probability of the meta tag, and the calculation method is the same as S 1 .

[0078] The fine-tuning weight weight3 is the parameter corresponding to the trained meta-label classification model.

[0079] In the automatic annotation method of exercise resources of the present invention, the predefined division rules include the following rules 1 to 4: Rule 1 is to extract independent phrases of the tag text and decompose them into at least two candidate meta tags; Rule 2 is to discard the alternative meta tags that appear alone in any tag; Rule three is to discard candidate meta tags whose occurrence frequency is greater than a preset threshold, where the occurrence frequency is the ratio of the total number of occurrences of the candidate meta tags to the total number of tags; Rule 4 is that when the tag text cannot be decomposed into at least two candidate meta tags, the candidate meta tags are determined according to the knowledge points corresponding to the tags.

[0080] In order to accurately obtain meta tags based on tag extraction, the above four rules are defined in this implementation.

[0081] Among them, rule 1 stipulates that a tag corresponds to at least two meta tags. To this end, the tag is segmented and multiple independent phrases obtained by the segmentation are extracted as candidate meta tags for the tag.

[0082] The second rule is to limit the frequency of meta tags. Meta tags should appear in other tags as much as possible and avoid appearing alone in the current tag. Therefore, the alternative meta tags that appear alone in the current tag are discarded, so that all the retained alternative meta tags appear in multiple tags.

[0083] Rule three also limits the frequency of occurrence of meta tags, where the preset threshold is determined based on experience and is set to 10% in this implementation. If an alternative meta tag appears in more than 10% of the tags, it is judged to be a common tag that is difficult to distinguish between different tags and should be discarded.

[0084] Rule 4 is a supplement to meta tags. When the tag text cannot be decomposed into at least two alternative meta tags, that is, when it is difficult to decompose the meta tags contained in the tag from the name of the tag alone, the alternative meta tags should be determined based on the knowledge points to which the tags belong. Specifically, the relevant definition theorems in the textbooks should be consulted to include the most commonly used concepts.

[0085] Based on the above method, the candidate meta tags obtained by tag decomposition are screened and supplemented, and the last candidate meta tags retained are used as meta tags, and the mapping relationship dict0 between tags and meta tags is constructed based on this: ; In the formula, y represents the label, n is the label always, y meta is a meta tag, is the list of meta tags corresponding to the jth tag, and m is the total number of all meta tags.

[0086] The constructed part of dict0 is shown in Table 1 below: Table 1

[0087] In the automatic annotation method for exercise resources of the present invention, the step of determining the annotation result of the exercise to be classified according to the weighted result specifically includes: Determining a confidence score for each tag in the tag probability distribution according to the weighted result; In this implementation, the reorganized meta-tag probability distribution is weighted to the tag probability distribution to obtain a reordered tag probability distribution dict4. Optionally, the probability corresponding to each tag in dict4 is directly used as the confidence score of each tag.

[0088] Sorting the labels in the label probability distribution in descending order of the confidence scores to obtain a reordered label probability distribution; The labels whose confidence scores in the reordered label probability distribution are greater than a preset score threshold are used as the labeling results of the exercises to be classified.

[0089] Sort the labels in dict4 in descending order of confidence scores, and put the confidence scores s r The labels corresponding to scores greater than the preset score threshold are used as the annotation results for the classification exercises.

[0090] Optionally, the preset score threshold is an empirical value, which is set to 0.5 in this implementation.

[0091] It should be noted that, in the verification using the test set, it was found that the labels finally obtained through this implementation, in the label probability distribution and meta-label probability distribution output by the label classification model and the meta-label classification model, when the label with the highest probability and the meta-label point to the same label, the labeling accuracy is greater than 90%.

[0092] The automatic annotation device for exercise resources provided by the present invention is described below. The automatic annotation device for exercise resources described below and the automatic annotation method for exercise resources described above can be referenced to each other.

[0093] like Figure 4 As shown, the automatic annotation device for exercise resources includes a label probability distribution module 401, a meta-label probability distribution module 402 and an annotation module 403; The label probability distribution module 401 is used to input the to-be-classified exercises into the pre-trained label classification model to obtain the label probability distribution output by the label classification model; The exercises to be classified are the exercise resources whose label categories need to be determined. The label of the exercises to be classified is the knowledge point to which the exercises to be classified belong, which is expressed as a short sentence containing multiple independent phrases.

[0094] Taking the BERT model as the initial model, a label classification model is obtained through multiple rounds of training on a dataset in the form of "mathematical text-label". The label classification model outputs each label and its corresponding probability as the label probability distribution.

[0095] It is understandable that the label probability distribution output by the pre-trained label classification model is obtained by training the model based on a data set in the form of "mathematical text-label". Therefore, it is less sensitive to some labels with high semantic overlap, such as "practical application of addition and subtraction of integers" and "column algebraic expressions and practical problems". The label semantic overlap and the performance of mathematical text are relatively similar, which makes the single label probability model have low discrimination for such labels.

[0096] Step 102, inputting the to-be-classified exercise into a pre-trained meta-tag classification model to obtain a meta-tag probability distribution output by the meta-tag classification model, wherein a meta-tag is an independent phrase that characterizes the uniqueness of the tag and is determined based on the tag; Based on the above reasons, the BERT model is used as the initial model, and a meta-tag classification model is obtained through multiple rounds of training on a dataset in the form of "mathematical text-meta-tag". The meta-tag classification model outputs each meta-tag and its corresponding probability as the meta-tag probability distribution.

[0097] The meta-tags are predetermined based on the tags of each exercise resource and are used to characterize the part of each tag that is more unique than other tags.

[0098] Taking the tag "Application of inequality properties" as an example, it includes three independent phrases "inequality", "property" and "application". Among them, the phrase "application" refers to the practical application of the knowledge point of inequality properties. Most knowledge points involve practical application problems, which makes a large number of tags contain "application". Therefore, "application" is discarded and "inequality" and "property" are determined as the meta tags of "application of inequality properties".

[0099] On this basis, the meta-tag probability distribution obtained by using the meta-tag classification model has a better understanding of tags with high semantic overlap on the one hand, and on the other hand, for some long-tail tags, there are meta-tags among the multiple meta-tags determined that represent more exercise resources than the long-tail tags themselves, thereby improving the fitting effect of long-tail tags to a certain extent.

[0100] The meta-tag probability distribution module 402 is used to input the to-be-classified exercise into a pre-trained meta-tag classification model to obtain a meta-tag probability distribution output by the meta-tag classification model, wherein the meta-tag is an independent phrase that characterizes the uniqueness of the tag determined based on the tag; Based on the above reasons, the BERT model is used as the initial model, and a meta-tag classification model is obtained through multiple rounds of training on a dataset in the form of "mathematical text-meta-tag". The meta-tag classification model outputs each meta-tag and its corresponding probability as the meta-tag probability distribution.

[0101] The meta-tags are predetermined based on the tags of each exercise resource and are used to characterize the part of each tag that is more unique than other tags.

[0102] Taking the tag "Application of inequality properties" as an example, it includes three independent phrases "inequality", "property" and "application". Among them, the phrase "application" refers to the practical application of the knowledge point of inequality properties. Most knowledge points involve practical application problems, which makes a large number of tags contain "application". Therefore, "application" is discarded and "inequality" and "property" are determined as the meta tags of "application of inequality properties".

[0103] On this basis, the meta-tag probability distribution obtained by using the meta-tag classification model has a better understanding of tags with high semantic overlap on the one hand, and on the other hand, for some long-tail tags, there are meta-tags among the multiple meta-tags determined that represent more exercise resources than the long-tail tags themselves, thereby improving the fitting effect of long-tail tags to a certain extent.

[0104] The labeling module 403 is used to reorganize the meta-tag probability distribution based on the mapping relationship between the tag and the meta-tag, use the reorganized meta-tag probability distribution to weight the tag probability distribution, and determine the labeling result of the to-be-classified exercise according to the weighted result.

[0105] Since one tag corresponds to multiple meta-tags, before using the meta-tag probability distribution to weight the tag probability distribution, the meta-tag probability distribution needs to be reorganized according to the mapping relationship between the tag and the meta-tag.

[0106] It can be understood that the mapping relationship dict0 between tags and meta tags is constructed when the meta tag corresponding to each tag is determined.

[0107] In a specific embodiment, Figure 2 As shown, after obtaining the tag probability distribution dict1 and the meta tag probability distribution dict2, the meta tag probability distribution dict2 is reorganized according to the predetermined dict0 to obtain the reorganized meta tag probability distribution dict3.

[0108] Specifically, it can be expressed as: ; ; In the formula, is the jth label y j Corresponding meta tag list The number of meta tags, represents the kth meta tag, Represents the probability of the kth meta tag in the meta tag probability distribution.

[0109] That is to say, for each tag, the probability sum of all meta tags corresponding to it is calculated, and then the ratio of the obtained probability sum to the number of all meta tags is calculated, and all ratios are integrated to obtain the reorganized meta tag probability distribution dict3.

[0110] It can be understood that the meta-tag probability distribution is the mapping relationship between meta-tags and probabilities, and the reorganized meta-tag probability distribution is the mapping relationship between tags and probabilities. On this basis, by combining dict3 and dict1, the reorganized meta-tag probability distribution can be weighted on the tag probability distribution to obtain the reordered tag probability distribution dict4: ; ; In the formula, is the probability of the jth label in the label probability distribution dict1, represents the probability of the jth tag in the reorganized meta-tag probability distribution dict3, Represents the probability of the jth label in the reordered label probability distribution dict4.

[0111] On this basis, the final labeling results of the exercises to be classified can be determined according to the reordered label probability distribution dict4.

[0112] Optionally, the label with the highest probability in the reordered label probability distribution dict4 is determined as the labeling result for the exercise to be classified.

[0113] Optionally, a probability threshold is set, and labels with probabilities greater than the probability threshold in the reordered label probability distribution dict4 are determined as the labeling results for the exercises to be classified. The probability threshold may be an empirical value.

[0114] Optionally, after arranging the probabilities in the reordered label probability distribution dict4 in descending order, the position where the probability gap occurs is determined, and the label corresponding to the probability before the gap is determined as the labeling result for the exercise to be classified.

[0115] The present invention splits semantically rich tags into independent meta-tags, pre-trains tag classification models and meta-tag classification models, and weights and guides the results of tag probability distribution based on meta-tag probability distribution, effectively solving the problem that existing classification algorithms are difficult to distinguish high-similarity tags, and improving the accuracy of automatic labeling and the ability to recognize long-tail tags.

[0116] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the automatic annotation method of the exercise resource, the method comprising: inputting the exercise to be classified into a pre-trained label classification model to obtain the label probability distribution output by the label classification model; inputting the exercise to be classified into a pre-trained meta-label classification model to obtain the meta-label probability distribution output by the meta-label classification model, wherein the meta-label is an independent phrase characterized by the uniqueness of the label determined based on the label; reorganizing the meta-label probability distribution based on the mapping relationship between the label and the meta-label, using the reorganized meta-label probability distribution to weight the label probability distribution, and determining the annotation result of the exercise to be classified according to the weighted result.

[0117] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0118] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the automatic labeling method of exercise resources provided by the above-mentioned methods, which method includes: inputting the exercises to be classified into a pre-trained label classification model to obtain the label probability distribution output by the label classification model; inputting the exercises to be classified into a pre-trained meta-label classification model to obtain the meta-tag probability distribution output by the meta-label classification model, wherein the meta-tag is an independent phrase characterized by the uniqueness of the tag determined based on the tag; reorganizing the meta-tag probability distribution based on the mapping relationship between the tag and the meta-tag, using the reorganized meta-tag probability distribution to weight the label probability distribution, and determining the labeling result of the exercises to be classified according to the weighted result.

[0119] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for automatically labeling exercise resources provided by the above-mentioned methods, the method comprising: inputting the exercises to be classified into a pre-trained label classification model to obtain a label probability distribution output by the label classification model; inputting the exercises to be classified into a pre-trained meta-label classification model to obtain a meta-tag probability distribution output by the meta-label classification model, wherein a meta-tag is an independent phrase determined based on the tag that characterizes the uniqueness of the tag; reorganizing the meta-tag probability distribution based on the mapping relationship between the tag and the meta-tag, weighting the label probability distribution using the reorganized meta-tag probability distribution, and determining the labeling result of the exercises to be classified according to the weighted result.

[0120] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0121] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for automatically marking exercise resources, characterized in that: include: Inputting the to-be-classified exercises into a pre-trained label classification model to obtain a label probability distribution output by the label classification model; Inputting the to-be-classified exercise into a pre-trained meta-tag classification model to obtain a meta-tag probability distribution output by the meta-tag classification model, wherein the meta-tag is an independent phrase that characterizes the uniqueness of the tag and is determined based on the tag; The meta-tag probability distribution is reorganized based on the mapping relationship between the tag and the meta-tag, the tag probability distribution is weighted using the reorganized meta-tag probability distribution, and the labeling result of the to-be-classified exercise is determined according to the weighted result.

2. The automatic annotation method for exercise resources according to claim 1, characterized in that: Before the step of inputting the to-be-classified exercises into the pre-trained label classification model to obtain the label probability distribution output by the label classification model, the step further includes: Using the mathematics text of the exercise resource as a sample and the knowledge points of the mathematics text as labels to construct a label data set; Pre-training a masked language task on the labeled dataset to obtain a preliminary model; The prepared model is fine-tuned on the labeled data set based on multi-task learning to obtain a label classification model for outputting label probability distribution.

3. The automatic annotation method for exercise resources according to claim 2 is characterized in that: The main task of the multi-task learning is defined as a label classification task, and the auxiliary task is defined as a class center learning task, wherein the class center learning task is a distance-based optimization task for gathering key features of the input text.

4. The automatic annotation method for exercise resources according to claim 1, characterized in that: Before the step of inputting the to-be-classified exercises into a pre-trained meta-tag classification model to obtain the meta-tag probability distribution output by the meta-tag classification model, the step further includes: Using the mathematics text of the exercise resource as a sample and the knowledge points of the mathematics text as labels to construct a label data set; Determine a meta tag corresponding to each tag based on a predefined division rule, and construct a meta tag dataset based on the tag dataset; Pre-training a masked language task on the labeled dataset to obtain a preliminary model; The prepared model is fine-tuned on the meta-tag dataset to obtain the meta-tag classification model for outputting the meta-tag probability distribution.

5. The automatic annotation method for exercise resources according to claim 4 is characterized in that: The predefined division rules include the following rules 1 to 4: Rule 1 is to extract independent phrases of the tag text and decompose them into at least two candidate meta tags; Rule 2 is to discard the alternative meta tags that appear alone in any tag; Rule three is to discard candidate meta tags whose occurrence frequency is greater than a preset threshold, where the occurrence frequency is the ratio of the total number of occurrences of the candidate meta tags to the total number of tags; Rule 4 is that when the tag text cannot be decomposed into at least two candidate meta tags, the candidate meta tags are determined according to the knowledge points corresponding to the tags.

6. The automatic annotation method for exercise resources according to claim 1, characterized in that: The step of determining the labeling result of the to-be-classified exercises according to the weighted result specifically includes: Determining a confidence score for each tag in the tag probability distribution according to the weighted result; Sorting the labels in the label probability distribution in descending order of the confidence scores to obtain a reordered label probability distribution; The labels whose confidence scores in the reordered label probability distribution are greater than a preset score threshold are used as the labeling results of the exercises to be classified.

7. An automatic annotation device for exercise resources, characterized in that: include: A label probability distribution module, used to input the to-be-classified exercises into a pre-trained label classification model to obtain the label probability distribution output by the label classification model; A meta-tag probability distribution module, used for inputting the to-be-classified exercise into a pre-trained meta-tag classification model to obtain a meta-tag probability distribution output by the meta-tag classification model, wherein the meta-tag is an independent phrase that characterizes the uniqueness of the tag determined based on the tag; The labeling module is used to reorganize the meta-tag probability distribution based on the mapping relationship between the tag and the meta-tag, use the reorganized meta-tag probability distribution to weight the tag probability distribution, and determine the labeling result of the to-be-classified exercise according to the weighted result.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the automatic annotation method for exercise resources according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for automatically marking exercise resources as claimed in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for automatically marking exercise resources as claimed in any one of claims 1 to 6 is implemented.

Images