A knowledge point automatic annotation modeling method and system

Abstract

The present invention discloses a knowledge point automatic annotation modeling method and system, wherein the annotation method includes a unified processing step, an ontology test question acquisition step, an input sample acquisition step, a model training step and a model acquisition step. The present invention trains a unified pre-trained language model UniLM to obtain a knowledge point automatic annotation model, which can take into account the association between labels under the premise of extracting rich semantic information of mathematics test question text, and efficiently completes the prediction of knowledge points of elementary school geometry test questions, thereby improving the credibility of automatic annotation of knowledge points; the present invention proposes an entity recognition and ontology replacement method based on matching templates to alleviate the data sparsity problem of elementary school geometry test question text in model training; the present invention performs full permutation processing on all knowledge point labels contained in the test questions to avoid erroneous feedback caused by inconsistent knowledge point label generation order during training.

Description

Technical Field

[0001] The present invention belongs to the field of smart educational resource knowledge annotation, and more specifically, relates to a knowledge point automatic annotation modeling method and system. Background Art

[0002] The main task of automatic annotation of knowledge points in math test questions is to predict the knowledge points to which the test questions belong based on the math test questions and the answer texts. This task belongs to the text multi-label classification task in the field of natural language processing, and certain text processing is required for the special domain attributes of math test questions. In various subject areas, test question resources are dominant. In order to provide each student with a more suitable personalized learning plan, it is particularly important to establish a good index structure for test question resources. This requires the use of automatic knowledge point annotation technology to automatically annotate test questions, so that test question resources can be reasonably labeled, and then scaled up to automatic annotation tasks for massive test question resources, which plays an important role in realizing cognitive diagnosis based on test questions and personalized automatic recommendation of test questions in intelligent teaching systems.

[0003] Currently, the most widely used labeling method is manual labeling, which is mainly done by hiring experts to label resources. Although the labeling results are relatively accurate, this method requires a lot of manpower investment and has certain limitations in terms of label consistency, credibility, and subsequent maintenance and updates. This means that the intelligent teaching system cannot be separated from the operation based on human achievements, which means that educational technology cannot become intelligent fundamentally.

[0004] At present, there are also some automatic knowledge point annotation methods that use the general classification model in the field of natural language processing to label and classify test question resources. However, the general classification model mainly focuses on the implementation and improvement of the algorithm, lacks the integration of domain knowledge, and ignores the characteristics of mathematics. Specifically, they lack the extraction and research of text features and semantic information of mathematics test questions; in addition, most existing classification models are based on binary classification models, while mathematics test questions are often associated with multiple and indefinite numbers of knowledge point labels, and there is also a certain correlation between labels. Most existing classification models find it difficult to capture the semantic information of test questions while considering the connection between labels, and existing automatic knowledge point annotation methods rarely pay attention to text multi-label classification.

[0005] For the automatic labeling task of elementary school mathematics geometry test questions, considering that test questions are often associated with multiple knowledge point labels, and knowledge point labels are often relatively fixed and unified, the most common approach is to regard the automatic labeling task as a multi-label multi-classification task. When dealing with multi-label classification tasks, most existing technologies use deep learning models such as recurrent neural networks (RNNs), but this type of model cannot extract semantic information from math test questions for classification, so it is necessary to find a neural network model that can better extract information such as the semantics of the test question text, which is also a major breakthrough in realizing the automatic labeling technology of knowledge points.

[0006] In order to facilitate the understanding of the present invention, the following explains relevant terms and related concepts:

[0007] Ontology: The ontology in the present invention refers to the ontology in elementary school geometry concepts, specifically: the general term for mathematical objects in elementary school mathematics geometry concepts, such as: triangles, quadrilaterals, lines, edges, etc.

[0008] Entity: The entity in the present invention refers to the entity in the elementary school geometry test questions, specifically: the specific expression name of the mathematical object in the elementary school mathematics geometry concept in the mathematics test question text, for example: triangle ABC, side AC, angle A, etc.

[0009] Ontology library: It is composed of ontologies in the elementary school geometry knowledge system, including core ontologies and attribute ontologies. Core ontologies are ontologies that cannot be used as attributes of other ontologies, such as triangles and quadrilaterals; attribute ontologies are ontologies that can be used as attributes of other ontologies, such as lines and edges. Summary of the invention

[0010] In view of the defects of the prior art, the purpose of the present invention is to provide a knowledge point automatic annotation modeling method and system, aiming to solve the problem of automatic annotation of knowledge points in elementary school mathematics geometry test questions.

[0011] To achieve the above object, the present invention provides a knowledge point automatic annotation modeling method comprising the following steps:

[0012] (1) Unified processing steps:

[0013] Adding knowledge point labels to all knowledge points in the initial test question set, and all knowledge point labels form a knowledge point set; the initial test question set consists of elementary school geometry test questions;

[0014] For each test question in the initial test question set, all the knowledge point labels contained in it are fully arranged, and the corresponding ordered set is generated for each knowledge point label arrangement method, and all ordered sets constitute the label group set of the test question; the label group set of all test questions constitutes the total label group set;

[0015] (2) Steps for obtaining the main body test questions:

[0016] For each test question in the initial test question set, the corresponding ontology test question is obtained according to the matching template; the ontology test questions of all test questions constitute an ontology test question set;

[0017] The matching template is composed of all ontologies in the ontology library and their corresponding regular expressions and ontology identifiers;

[0018] (3) Input sample acquisition steps:

[0019] Obtaining input samples of a unified pre-trained language model according to the ontology test set and the total set of label groups;

[0020] (4) Model training steps:

[0021] Input the input sample into a unified pre-trained language model, and train it according to the set number of epochs and batch input size batch_size; after each round of training, a labeled model of the training round is obtained;

[0022] (5) Model acquisition steps:

[0023] According to all the annotation models, a knowledge point automatic annotation model is obtained.

[0024] Preferably, the ontology test question acquisition step includes the following sub-steps:

[0025] (1) Entity identification sub-step:

[0026] For each entity of each test question in the initial test question set, a regular expression matching the entity is searched in the matching template, and the ontology corresponding to the regular expression that matches successfully is the ontology corresponding to the entity;

[0027] Perform a numbering operation on all entities, the purpose of which is to make the ontology identifiers of different entities in the same test question different from each other;

[0028] (2) Sub-steps of replacing the main body:

[0029] According to the repeated characters appearing in the two entities, the attribute relationship between the two corresponding entities is determined;

[0030] Replace each entity with the ontology identifier of the corresponding ontology. The ontology identifier of the core ontology consists of the ontology identifier and the number of the corresponding entity; the ontology identifier of the attribute ontology consists of the ontology identifier of the core ontology to which it belongs, the ontology identifier of the attribute ontology, and the number of the entity corresponding to the attribute ontology.

[0031] Preferably, the input sample acquisition step includes the following sub-steps:

[0032] (1) Dataset division sub-steps:

[0033] Dividing the ontology test set into a training set and a test set according to a set ratio;

[0034] Searching the total set of label groups for each ontology question in the training set, and adding the found label group set to the training set;

[0035] Searching the total set of label groups for each ontology test question in the test set, and adding the found label group set to the test set;

[0036] (2) Word segmentation sub-step:

[0037] Use the python word segmentation tool to segment all the ontology test questions in the training set to obtain a word segmentation dictionary;

[0038] (3) Short text replacement sub-step:

[0039] In the dictionary of the Python word segmentation tool, multiple different words that do not belong to the word segmentation dictionary are searched to form a short text set, and the total number of elements in the short text set is equal to that in the knowledge point set;

[0040] Establishing a short text mapping from the short text set to the knowledge point set: randomly mapping all elements in the short text set to all elements in the knowledge point set; the short text mapping is a full mapping;

[0041] Taking each label group set in the training set as an original image set, a sub-mapping is obtained according to the short text mapping, and each image in the image set is sequentially connected with at least one connector to form a label set text;

[0042] (4) Splicing sub-step:

[0043] For each ontology question in the training set, the ontology question and all the corresponding label set texts are sequentially concatenated head to tail to obtain a single sample; during the concatenation, a head tag needs to be added to the head of the ontology question, a tail tag needs to be added to the tail of the ontology question, and a tail tag needs to be added to the tail of the label set text;

[0044] All samples in the training set constitute the input samples.

[0045] Preferably, the specific process of the model training step is:

[0046] Input the input sample into the unified pre-trained language model for training according to the set number of epochs and batch input amount batch_size;

[0047] In each round of training, the BERT pre-trained model and the seq-to-seq mask method are used for fine-tuning training, and the optimizer Adam is used to continuously update the hyperparameters of the unified pre-trained language model until the cross entropy loss tends to a stable value. At this time, the annotation model of this round of training is obtained.

[0048] Preferably, the specific process of the model acquisition step is: input the test set into all the annotation models in turn, count the test question predicted knowledge point labels and the mathematics test question real knowledge point labels output by each annotation model, and use this to calculate the Hamming loss of each annotation model, and select the annotation model with the smallest Hamming loss value as the knowledge point automatic annotation model.

[0049] The present invention provides a knowledge point automatic annotation modeling system, comprising:

[0050] Unified processing module:

[0051] Used to add knowledge point labels to all knowledge points in the initial test question set, and all knowledge point labels constitute a knowledge point set; the initial test question set consists of elementary school geometry test questions;

[0052] For each test question in the initial test question set, all the knowledge point labels contained in it are fully arranged, and the corresponding ordered set is generated for each knowledge point label arrangement method, and all ordered sets constitute the label group set of the test question; the label group set of all test questions constitutes the total label group set;

[0053] Ontology test question acquisition module:

[0054] For each test question in the initial test question set, according to the matching template, the corresponding ontology test question is obtained; the ontology test questions of all test questions constitute an ontology test question set;

[0055] The matching template is composed of all ontologies in the ontology library and their corresponding regular expressions and ontology identifiers;

[0056] Input sample acquisition module:

[0057] Used to obtain input samples of a unified pre-trained language model according to the ontology test set and the total set of label groups;

[0058] Model training module:

[0059] It is used to input the input sample into the unified pre-trained language model and perform training according to the set number of epochs and batch input amount batch_size; after each round of training, a labeling model of the training round is obtained;

[0060] Model acquisition module:

[0061] Used to obtain a knowledge point automatic annotation model based on all the annotation models.

[0062] Preferably, the ontology question acquisition module includes the following submodules:

[0063] Entity recognition submodule:

[0064] For each entity of each test question in the initial test question set, searching for a regular expression matching the entity in the matching template, and the ontology corresponding to the regular expression that matches successfully is the ontology corresponding to the entity;

[0065] Perform a numbering operation on all entities, the purpose of which is to make the ontology identifiers of different entities in the same test question different from each other;

[0066] Ontology replacement submodule:

[0067] It is used to determine the attribute relationship between the two corresponding entities based on the repeated characters appearing in the two entities;

[0068] Replace each entity with the ontology identifier of the corresponding ontology. The ontology identifier of the core ontology consists of the ontology identifier and the number of the corresponding entity. The ontology identifier of the attribute ontology consists of the ontology identifier of the core ontology to which it belongs, the ontology identifier of the attribute ontology, and the number of the entity corresponding to the attribute ontology.

[0069] The input sample acquisition module includes the following submodules:

[0070] Dataset division submodule:

[0071] Dividing the ontology test set into a training set and a test set according to a set ratio;

[0072] Searching the total set of label groups for each ontology question in the training set, and adding the found label group set to the training set;

[0073] Searching the total set of label groups for each ontology test question in the test set, and adding the found label group set to the test set;

[0074] Word segmentation submodule:

[0075] Use the python word segmentation tool to segment all the ontology test questions in the training set to obtain a word segmentation dictionary;

[0076] Short text replacement submodule:

[0077] In the dictionary of the Python word segmentation tool, multiple different words that do not belong to the word segmentation dictionary are searched to form a short text set, and the total number of elements in the short text set is equal to that in the knowledge point set;

[0078] Establishing a short text mapping from the short text set to the knowledge point set: randomly mapping all elements in the short text set to all elements in the knowledge point set; the short text mapping is a full mapping;

[0079] Taking each label group set in the training set as an original image set, a sub-mapping is obtained according to the short text mapping, and each image in the image set is sequentially connected with at least one connector to form a label set text;

[0080] Splicing submodule:

[0081] For each ontology question in the training set, the ontology question and all the corresponding label set texts are sequentially concatenated head to tail to obtain a single sample; during the concatenation, a head tag needs to be added to the head of the ontology question, a tail tag needs to be added to the tail of the ontology question, and a tail tag needs to be added to the tail of the label set text;

[0082] All samples in the training set constitute the input samples.

[0083] Preferably, the specific operations included in the model training module are:

[0084] Input the input sample into the unified pre-trained language model for training according to the set number of epochs and batch input amount batch_size;

[0085] In each round of training, the BERT pre-trained model and the seq-to-seq mask method are used for fine-tuning training, and the optimizer Adam is used to continuously update the hyperparameters of the unified pre-trained language model until the cross entropy loss tends to a stable value. At this time, the annotation model of this round of training is obtained;

[0086] The specific operations included in the model acquisition module are:

[0087] The test set is input into all the annotation models in turn, and the predicted knowledge point labels of the test questions and the real knowledge point labels of the mathematics test questions output by each annotation model are counted, and the Hamming loss of each annotation model is calculated based on this, and the annotation model with the smallest Hamming loss value is selected as the knowledge point automatic annotation model.

[0088] The present invention provides a knowledge point automatic annotation modeling device, comprising a memory and a processor; the memory is used to store a computer program; the processor is used to implement the above-mentioned knowledge point automatic annotation modeling method when executing the computer program.

[0089] The present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned knowledge point automatic annotation modeling method is implemented.

[0090] Through the above technical scheme conceived by the present invention, compared with the prior art, since the present invention trains the unified pre-trained language model UniLM to obtain the knowledge point automatic labeling model, compared with the knowledge point automatic labeling model obtained by training other models, the model of the present invention can take into account the correlation between labels while extracting rich semantic information of elementary school geometry test questions text, and efficiently complete the prediction of elementary school geometry test knowledge points, and to a certain extent improves the credibility of automatic labeling of knowledge points.

[0091] The present invention takes into account the expression characteristics of elementary school geometry test questions and proposes an entity recognition and ontology replacement method based on matching templates in view of the importance of elementary school geometry ontology and the diversity of representation forms, so as to alleviate the data sparsity problem of elementary school geometry test question text in deep learning model training, and at the same time, it can mine more features from the test question text to enhance the automatic annotation effect.

[0092] Taking into account that the unified pre-trained language model has a certain order in the process of predicting knowledge point labels, the present invention performs full permutation processing on all knowledge point labels contained in the test questions, so that in the process of fine-tuning training, the unified pre-trained language model can better capture the association relationship information between the mathematics test question text and the knowledge point labels, and can avoid the incorrect feedback of the unified pre-trained language model during the training process caused by the inconsistency of the knowledge point label generation order.

[0093] In generative tasks, the generation of the current knowledge point label depends on the previous knowledge point label. Considering that the UNILM model is more accurate in generating short texts, the present invention replaces the knowledge point labels with short texts, thereby reducing the error rate of generating the current knowledge point label, thereby avoiding the generation of erroneous subsequent knowledge point labels.

[0094] In the process of model training, the present invention uses the BERT pre-training model, which can enable the unified pre-training language model to extract richer text features; the present invention selects the seq-to-seq mask method for fine-tuning training, which can enable the unified pre-training language model to integrate the association relationship between the test text and the knowledge point label, and between the knowledge point label and the knowledge point label when performing the knowledge point automatic labeling task; because the input samples of the present invention have the problems of data sparseness and much noise, the present invention selects the optimizer Adam to perform the hyperparameter update operation, and Adam can alleviate the data sparseness problem and reduce the influence of excessive data noise. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] Figure 1 is a flow chart of a knowledge point automatic annotation modeling method provided by an embodiment of the present invention;

[0096] Figure 2 It is a flow chart of the ontology question acquisition steps of the knowledge point automatic annotation modeling method provided by an embodiment of the present invention;

[0097] Figure 3 It is a flowchart of the input sample acquisition steps of the knowledge point automatic annotation modeling method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0098] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0099] like Figure 1 As shown, this embodiment provides a knowledge point automatic annotation modeling method, including a unified processing step, an ontology test question acquisition step, an input sample acquisition step, a model training step and a model acquisition step.

[0100] (1) The specific process of unified processing steps is as follows:

[0101] All knowledge points in the initial test question set are labeled with knowledge points. All knowledge point labels form a knowledge point set. The total number of knowledge point labels in the knowledge point set is 97. The initial test question set consists of elementary school geometry questions. The total number of questions in the initial test question set is 54097.

[0102] For each test question in the initial test question set, all the knowledge point labels it contains are fully arranged, and the corresponding ordered set is generated for each knowledge point label arrangement method. All ordered sets constitute the label group set of the test question; the label group set of all test questions constitutes the total label group set.

[0103] Taking into account that the unified pre-trained language model has a certain order in the process of predicting knowledge point labels, this embodiment performs a full permutation process on all knowledge point labels contained in the test questions, so that in the process of fine-tuning training, the unified pre-trained language model can better capture the association relationship information between the mathematics test question text and the knowledge point labels, and can avoid the inconsistent knowledge point label generation order causing the unified pre-trained language model to have erroneous feedback during the training process.

[0104] (2) The specific process of the ontology test question acquisition step is as follows: for each test question in the initial test question set, obtain its corresponding ontology test question according to the matching template; the ontology test questions of all test questions constitute the ontology test question set;

[0105] The matching template consists of all ontologies in the ontology library and their corresponding regular expressions and ontology identifiers; part of the matching template is shown in the matching table:

[0106] Match table

[0107]

[0108]

[0109] like Figure 2 As shown, the ontology question acquisition step includes an entity recognition sub-step and an ontology replacement sub-step:

[0110] Entity recognition sub-steps:

[0111] For each entity of each test question in the initial test question set, a regular expression matching the entity is searched in the matching template, and the ontology corresponding to the regular expression that matches successfully is the ontology corresponding to the entity;

[0112] All entities are numbered, one entity corresponds to one number, and the number is a single Arabic numeral; the purpose of the numbering operation is to make the entity identifiers of different entities in the same test question different from each other;

[0113] Ontology replacement sub-steps:

[0114] According to the repeated characters appearing in the two entities, the attribute relationship between the two corresponding entities is determined;

[0115] Replace each entity with the ontology identifier of the corresponding ontology. The ontology identifier of the core ontology consists of the ontology identifier and the number of the corresponding entity; the ontology identifier of the attribute ontology consists of the ontology identifier of the core ontology to which it belongs, the ontology identifier of the attribute ontology, and the number of the entity corresponding to the attribute ontology.

[0116] The following is an example of body replacement:

[0117] Question: It is known that triangle ABC is an isosceles triangle, where AB=AC=5. If ∠ABC is 60°, find the length of BC.

[0118] Main question: It is known that triangle #1 is an isosceles triangle, where line #1 of triangle #1 is equal to line #2 of triangle #1, which is equal to 5. If angle #1 of triangle #1 is 60 degrees, find the length of line #3 of triangle #1.

[0119] This embodiment takes into account the expression characteristics of elementary school geometry test questions and proposes an entity recognition and ontology replacement method based on matching templates in view of the importance of elementary school geometry ontology and the diversity of representation forms, so as to alleviate the data sparsity problem of elementary school geometry test question text in the deep learning model training, and at the same time, it can mine more features from the test question text to enhance the automatic annotation effect.

[0120] (3) The specific process of the input sample acquisition step is as follows: according to the ontology test set and the total set of label groups, the input samples of the unified pre-trained language model are obtained;

[0121] like Figure 3 As shown, the input sample acquisition step includes a data set partitioning sub-step, a word segmentation sub-step, a short text replacement sub-step, and a splicing sub-step:

[0122] Dataset partitioning sub-steps:

[0123] The ontology test set is divided into a training set and a test set according to the set ratio of 8:2;

[0124] Search the total set of label groups for each ontology test question in the training set, and add the found label group set to the training set;

[0125] Search the total set of label groups for each ontology question in the test set, and add the found label group set to the test set;

[0126] Word segmentation sub-step:

[0127] Use the Python word segmentation tool to segment all the ontology test questions in the training set and obtain a word segmentation dictionary;

[0128] The Python word segmentation tool is a very popular word segmentation tool with the characteristics of high performance, accuracy, and scalability.

[0129] Short text replacement sub-steps:

[0130] Randomly search the dictionary of the Python word segmentation tool to obtain multiple different words that do not belong to the word segmentation dictionary, and form a short text set. The total number of elements in the short text set is equal to that in the knowledge point set.

[0131] Establish a short text mapping from the short text set to the knowledge point set: randomly map all elements in the short text set to all elements in the knowledge point set; the short text mapping is a full mapping;

[0132] Taking each label group set in the training set as the original image set, a sub-mapping is obtained according to the short text mapping, and each image in the image set is sequentially connected with at least one connector to form a label set text;

[0133] In generative tasks, the generation of the current knowledge point label depends on the previous knowledge point label. Considering that the UNILM model is more accurate in generating short texts, the present invention replaces the knowledge point labels with short texts, thereby reducing the error rate of generating the current knowledge point label, thereby avoiding the generation of erroneous subsequent knowledge point labels.

[0134] Splicing sub-steps:

[0135] For each ontology question in the training set, the ontology question and all its corresponding label set texts are concatenated head to tail to obtain a single sample; when concatenating, a head tag [CLS] needs to be added to the head of the ontology question, a tail tag [CLS] needs to be added to the tail of the ontology question, and a tail tag [CLS] needs to be added to the tail of the label set text;

[0136] All samples in the training set constitute the input samples, and the total number of input samples is 105265.

[0137] (4) The specific process of the model training step is:

[0138] The input samples are input into the unified pre-trained language model and trained according to the set number of epochs and batch input batch_size; after each round of training, a labeled model of this round of training is obtained; among them, the set number of epochs is 20, and the batch input batch_size is 16.

[0139] The UniLM model is a research result of Microsoft Research based on the pre-trained model Bert, and is called the unified pre-trained language model. It can complete unidirectional, sequence-to-sequence, and bidirectional prediction tasks, and can excellently handle abstract summarization, generative question answering, and language generation tasks.

[0140] In each round of training, the BERT pre-trained model and the seq-to-seq mask method are used for fine-tuning training, and the optimizer Adam is used to continuously update the hyperparameters of the unified pre-trained language model until the cross entropy loss tends to a stable value. At this time, the annotation model of this round of training is obtained.

[0141] After the model generates a prediction value, the predicted label of the category is compared with the true label, and the cross entropy loss function is used to calculate the loss generated, which is the cross entropy loss.

[0142] In the model training process, this embodiment uses the BERT pre-training model, which can enable the unified pre-training language model to extract richer text features; this embodiment selects the seq-to-seq mask method for fine-tuning training, which can enable the unified pre-training language model to integrate the association between the test text and the knowledge point label, and between the knowledge point label and the knowledge point label when performing the knowledge point automatic labeling task; because the input samples of this embodiment have data sparsity and high noise problems, this embodiment selects the optimizer Adam to perform hyperparameter update operations, and Adam can alleviate the data sparsity problem and reduce the impact of excessive data noise.

[0143] (5) The specific process of the model acquisition step is:

[0144] The test set is input into all the annotation models in turn, and the predicted knowledge point labels and the real knowledge point labels of the math questions output by each annotation model are counted, and the Hamming loss of each annotation model is calculated, and the annotation model with the smallest Hamming loss value is selected as the knowledge point automatic annotation model. In this embodiment, the annotation model with a Hamming loss value of 0.0063 is selected as the final knowledge point automatic annotation model.

[0145] Hamming loss is used to examine the misclassification of samples on a single label. The smaller the value of this indicator, the better the classification performance of the model. Its optimal value is 0, which means that all labels of every sample are classified correctly.

[0146] This embodiment trains the unified pre-trained language model UniLM to obtain a knowledge point automatic labeling model. Compared with the knowledge point automatic labeling model obtained by training other models, the model of this embodiment can take into account the correlation between labels while extracting rich semantic information of elementary school geometry test questions text, and efficiently complete the prediction of elementary school geometry test knowledge points, and to a certain extent improves the credibility of automatic labeling of knowledge points.

[0147] The present invention provides a knowledge point automatic annotation modeling system, comprising:

[0148] Unified processing module:

[0149] It is used to label all knowledge points in the initial test question set, and all knowledge point labels constitute a knowledge point set; the initial test question set consists of elementary school geometry test questions;

[0150] For each test question in the initial test question set, all the knowledge point labels contained in it are fully arranged, and the corresponding ordered set is generated for each knowledge point label arrangement method, and all ordered sets constitute the label group set of the test question; the label group set of all test questions constitutes the total label group set;

[0151] Ontology test question acquisition module:

[0152] For each test question in the initial test question set, according to the matching template, the corresponding ontology test question is obtained; the ontology test questions of all test questions constitute the ontology test question set;

[0153] The matching template consists of all ontologies in the ontology library and their corresponding regular expressions and ontology identifiers;

[0154] Input sample acquisition module:

[0155] Used to obtain input samples of a unified pre-trained language model based on the ontology test set and the total set of label groups;

[0156] Model training module:

[0157] It is used to input the input samples into the unified pre-trained language model and train according to the set number of epochs and batch input size batch_size; after each round of training, a labeled model of the training round is obtained;

[0158] Model acquisition module:

[0159] Used to obtain the automatic annotation model of knowledge points based on all annotation models.

[0160] The ontology question acquisition module includes the following sub-modules:

[0161] Entity recognition submodule:

[0162] For each entity of each test question in the initial test question set, a regular expression matching the entity is searched in the matching template, and the ontology corresponding to the regular expression that matches successfully is the ontology corresponding to the entity;

[0163] Perform a numbering operation on all entities. The purpose of the numbering operation is to make the ontology identifiers of different entities in the same test question different from each other.

[0164] Ontology replacement submodule:

[0165] It is used to determine the attribute relationship between the two corresponding entities based on the repeated characters appearing in the two entities;

[0166] Replace each entity with the ontology identifier of the corresponding ontology. The ontology identifier of the core ontology consists of the ontology identifier and the number of the corresponding entity. The ontology identifier of the attribute ontology consists of the ontology identifier of the core ontology to which it belongs, the ontology identifier of the attribute ontology, and the number of the entity corresponding to the attribute ontology.

[0167] The input sample acquisition module includes the following submodules:

[0168] Dataset division submodule:

[0169] Divide the ontology test set into a training set and a test set according to a set ratio;

[0170] Search the total set of label groups for each ontology test question in the training set, and add the found label group set to the training set;

[0171] Search the total set of label groups for each ontology question in the test set, and add the found label group set to the test set;

[0172] Word segmentation submodule:

[0173] Use the Python word segmentation tool to segment all the ontology test questions in the training set and obtain a word segmentation dictionary;

[0174] Short text replacement submodule:

[0175] In the dictionary of the Python word segmentation tool, multiple different words that do not belong to the word segmentation dictionary are searched to form a short text set. The total number of elements in the short text set is equal to that in the knowledge point set.

[0176] Establish a short text mapping from the short text set to the knowledge point set: randomly map all elements in the short text set to all elements in the knowledge point set; the short text mapping is a full mapping;

[0177] Taking each label group set in the training set as the original image set, a sub-mapping is obtained according to the short text mapping, and each image in the image set is sequentially connected with at least one connector to form a label set text;

[0178] Splicing submodule:

[0179] For each ontology question in the training set, the ontology question and all its corresponding label set texts are concatenated head to tail to obtain a single sample; when concatenating, a head tag needs to be added to the head of the ontology question, a tail tag needs to be added to the tail of the ontology question, and a tail tag needs to be added to the tail of the label set text;

[0180] All samples in the training set constitute the input samples.

[0181] The specific operations included in the model training module are:

[0182] Input the input sample into the unified pre-trained language model and train it according to the set number of epochs and batch input batch_size;

[0183] In each round of training, the BERT pre-trained model and the seq-to-seq mask method are used for fine-tuning training, and the optimizer Adam is used to continuously update the hyperparameters of the unified pre-trained language model until the cross entropy loss tends to a stable value. At this time, the annotation model of this round of training is obtained;

[0184] The specific operations included in the model acquisition module are:

[0185] The test set is input into all the annotation models in turn, and the predicted knowledge point labels of the test questions and the actual knowledge point labels of the mathematics test questions output by each annotation model are counted. The Hamming loss of each annotation model is calculated based on this, and the annotation model with the smallest Hamming loss value is selected as the automatic knowledge point annotation model.

[0186] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A knowledge point automatic annotation modeling method, It is characterized in that The following steps are involved: (1) Unified processing steps: Adding knowledge point labels to all knowledge points in the initial test question set, and all knowledge point labels form a knowledge point set; the initial test question set consists of elementary school geometry test questions; For each test question in the initial test question set, all the knowledge point labels contained in it are fully arranged, and the corresponding ordered set is generated for each knowledge point label arrangement method, and all ordered sets constitute the label group set of the test question; the label group set of all test questions constitutes the total label group set; (2) Steps for obtaining the main body test questions: For each test question in the initial test question set, the corresponding ontology test question is obtained according to the matching template; the ontology test questions of all test questions constitute an ontology test question set; The matching template is composed of all ontologies in the ontology library and their corresponding regular expressions and ontology identifiers; (3) Input sample acquisition steps: Obtaining input samples of a unified pre-trained language model according to the ontology test set and the total set of label groups; (4) Model training steps: Input the input sample into a unified pre-trained language model, and train it according to the set number of epochs and batch input size batch_size; after each round of training, a labeled model of the training round is obtained; (5) Model acquisition steps: According to all the annotation models, a knowledge point automatic annotation model is obtained; The ontology test question acquisition step includes the following sub-steps: (1) Entity identification sub-step: For each entity of each test question in the initial test question set, a regular expression matching the entity is searched in the matching template, and the ontology corresponding to the regular expression that matches successfully is the ontology corresponding to the entity; Perform a numbering operation on all entities, the purpose of which is to make the ontology identifiers of different entities in the same test question different from each other; (2) Sub-steps of ontology replacement: According to the repeated characters appearing in the two entities, the attribute relationship between the two corresponding entities is determined; Replace each entity with the ontology identifier of the corresponding ontology. The ontology identifier of the core ontology consists of the ontology identifier and the number of the corresponding entity; the ontology identifier of the attribute ontology consists of the ontology identifier of the core ontology to which it belongs, the ontology identifier of the attribute ontology, and the number of the entity corresponding to the attribute ontology.

2. The knowledge point automatic annotation modeling method according to claim 1, It is characterized in that The input sample acquisition step includes the following sub-steps: (1) Dataset division sub-steps: Dividing the ontology test set into a training set and a test set according to a set ratio; Searching the total set of label groups for each ontology question in the training set, and adding the found label group set to the training set; Searching the total set of label groups for each ontology test question in the test set, and adding the found label group set to the test set; (2) Word segmentation sub-step: Use the python word segmentation tool to segment all the ontology test questions in the training set to obtain a word segmentation dictionary; (3) Short text replacement sub-step: In the dictionary of the Python word segmentation tool, multiple different words that do not belong to the word segmentation dictionary are searched to form a short text set, and the total number of elements in the short text set is equal to that in the knowledge point set; Establishing a short text mapping from the short text set to the knowledge point set: randomly mapping all elements in the short text set to all elements in the knowledge point set; the short text mapping is a full mapping; Taking each label group set in the training set as an original image set, a sub-mapping is obtained according to the short text mapping, and each image in the image set is sequentially connected with at least one connector to form a label set text; (4) Splicing sub-steps: For each ontology question in the training set, the ontology question and all the corresponding label set texts are sequentially concatenated head to tail to obtain a single sample; during the concatenation, a head tag needs to be added to the head of the ontology question, a tail tag needs to be added to the tail of the ontology question, and a tail tag needs to be added to the tail of the label set text; All samples in the training set constitute the input samples.

3. The knowledge point automatic annotation modeling method according to claim 1, It is characterized in that The specific process of the model training step is: Input the input sample into the unified pre-trained language model for training according to the set number of epochs and batch input amount batch_size; In each round of training, the BERT pre-trained model and the seq-to-seq mask method are used for fine-tuning training, and the optimizer Adam is used to continuously update the hyperparameters of the unified pre-trained language model until the cross entropy loss tends to a stable value. At this time, the annotation model of this round of training is obtained.

4. The knowledge point automatic annotation modeling method according to claim 1, It is characterized in that The specific process of the model acquisition step is: input the test set into all the annotation models in turn, count the test question predicted knowledge point labels and the mathematics test question real knowledge point labels output by each annotation model, and use this to calculate the Hamming loss of each annotation model, and select the annotation model with the smallest Hamming loss value as the knowledge point automatic annotation model.

5. A knowledge point automatic annotation modeling system, It is characterized in that include: Unified processing module: Used to add knowledge point labels to all knowledge points in the initial test question set, and all knowledge point labels constitute a knowledge point set; the initial test question set consists of elementary school geometry test questions; For each test question in the initial test question set, all the knowledge point labels contained in it are fully arranged, and the corresponding ordered set is generated for each knowledge point label arrangement method, and all ordered sets constitute the label group set of the test question; the label group set of all test questions constitutes the total label group set; Ontology test question acquisition module: For each test question in the initial test question set, according to the matching template, the corresponding ontology test question is obtained; the ontology test questions of all test questions constitute an ontology test question set; The matching template is composed of all ontologies in the ontology library and their corresponding regular expressions and ontology identifiers; Input sample acquisition module: Used to obtain input samples of a unified pre-trained language model according to the ontology test set and the total set of label groups; Model training module: It is used to input the input sample into the unified pre-trained language model and perform training according to the set number of epochs and batch input amount batch_size; after each round of training, a labeling model of the training round is obtained; Model acquisition module: Used to obtain a knowledge point automatic annotation model based on all the annotation models; The ontology question acquisition module includes the following submodules: Entity recognition submodule: For each entity of each test question in the initial test question set, searching for a regular expression matching the entity in the matching template, and the ontology corresponding to the regular expression that matches successfully is the ontology corresponding to the entity; Perform a numbering operation on all entities, the purpose of which is to make the ontology identifiers of different entities in the same test question different from each other; Ontology replacement submodule: It is used to determine the attribute relationship between the two corresponding entities based on the repeated characters appearing in the two entities; Replace each entity with the ontology identifier of the corresponding ontology. The ontology identifier of the core ontology consists of the ontology identifier and the number of the corresponding entity. The ontology identifier of the attribute ontology consists of the ontology identifier of the core ontology to which it belongs, the ontology identifier of the attribute ontology, and the number of the entity corresponding to the attribute ontology. The input sample acquisition module includes the following submodules: Dataset division submodule: Dividing the ontology test set into a training set and a test set according to a set ratio; Searching the total set of label groups for each ontology question in the training set, and adding the found label group set to the training set; Searching the total set of label groups for each ontology test question in the test set, and adding the found label group set to the test set; Word segmentation submodule: Use the python word segmentation tool to segment all the ontology test questions in the training set to obtain a word segmentation dictionary; Short text replacement submodule: In the dictionary of the Python word segmentation tool, multiple different words that do not belong to the word segmentation dictionary are searched to form a short text set, and the total number of elements in the short text set is equal to that in the knowledge point set; Establishing a short text mapping from the short text set to the knowledge point set: randomly mapping all elements in the short text set to all elements in the knowledge point set; the short text mapping is a full mapping; Taking each label group set in the training set as an original image set, a sub-mapping is obtained according to the short text mapping, and each image in the image set is sequentially connected with at least one connector to form a label set text; Splicing submodule: For each ontology question in the training set, the ontology question and all the corresponding label set texts are sequentially concatenated head to tail to obtain a single sample; during the concatenation, a head tag needs to be added to the head of the ontology question, a tail tag needs to be added to the tail of the ontology question, and a tail tag needs to be added to the tail of the label set text; All samples in the training set constitute the input samples.

6. The knowledge point automatic annotation modeling system according to claim 5, It is characterized in that The specific operations included in the model training module are: Input the input sample into the unified pre-trained language model for training according to the set number of epochs and batch input amount batch_size; In each round of training, the BERT pre-trained model and the seq-to-seq mask method are used for fine-tuning training, and the optimizer Adam is used to continuously update the hyperparameters of the unified pre-trained language model until the cross entropy loss tends to a stable value. At this time, the annotation model of this round of training is obtained; The specific operations included in the model acquisition module are: The test set is input into all the annotation models in turn, and the predicted knowledge point labels of the test questions and the real knowledge point labels of the mathematics test questions output by each annotation model are counted, and the Hamming loss of each annotation model is calculated based on this, and the annotation model with the smallest Hamming loss value is selected as the knowledge point automatic annotation model.

7. A knowledge point automatic annotation modeling device, It is characterized in that It comprises a memory and a processor; the memory is used to store a computer program; the processor is used to implement the knowledge point automatic annotation modeling method as described in any one of claims 1 to 4 when executing the computer program.

8. A computer-readable storage medium, It is characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the knowledge point automatic annotation modeling method according to any one of claims 1 to 4 is implemented.

Images