Method, System, and Storage Medium for Evaluating the Answerability of Text Automatic Generation Problems

Abstract

A method, system and storage medium for evaluating the answerability of questions automatically generated from text. The evaluation method includes the following steps: obtaining a data set and expanding the data set to obtain enhanced data; annotating and voting on the enhanced data using multiple question-answering models to obtain an annotated data set; encoding the context and question in the annotated data set, and fusing the context features with the question features to obtain an embedded representation of the question after fusion with the context; and using the embedded representation of the question after fusion with the context to evaluate the answerability of the question based on a contrastive learning model. Also disclosed is a system and storage medium for evaluating the answerability of questions automatically generated from text. By introducing contextual information and a contrastive learning model, the present invention solves the problem that the existing methods for evaluating the answerability of questions are not highly correlated with human evaluations and that contextual information is not introduced during evaluation, thereby achieving a full evaluation of the answerability of the question.

Description

Technical Field

[0001] The present invention belongs to the field of Internet online learning, and in particular relates to a method, system and storage medium for evaluating the answerability of questions automatically generated from text. Background Art

[0002] With the rapid development of internet technology, researchers in the field of education are constantly exploring and utilizing it to improve educational effectiveness. The widespread availability of online resources and the prevalence of online learning have provided learners with more convenient and diverse learning methods. Online learning platforms not only offer a wealth of learning resources but also allow learners to choose the learning method that best suits them, improving learning efficiency. This has made online learning platforms popular among students and parents, and they play an increasingly important role in modern education.

[0003] In traditional education, teachers assess students' understanding and mastery of knowledge by asking them questions. However, in large-scale online learning, generating test questions becomes more challenging due to the time and effort required, coupled with the dramatic increase in the amount of online learning materials. Consequently, many researchers, both domestically and internationally, have begun exploring how to use computers to automatically generate questions to improve teaching efficiency and effectiveness. Automatic question generation requires efficient algorithms and techniques, which is quite challenging. As a result, question generation models still generate a large number of unanswerable questions. Current evaluation methods, which mostly assess question similarity, suffer from the following problems: First, the correlation between answerability evaluation results and human evaluation is low; second, the evaluation fails to consider contextual information, making it difficult to fully assess the answerability of generated questions. Therefore, fully assessing the answerability of questions remains a pressing challenge. Summary of the Invention

[0004] The purpose of the present invention is to provide a method, system and storage medium for evaluating the answerability of questions automatically generated from text. By introducing contextual information and a comparative learning model, the present invention solves the problem that existing methods for evaluating the answerability of questions are not highly correlated with human evaluation and do not introduce contextual information during evaluation.

[0005] In order to achieve the above object, the present invention has the following technical solutions:

[0006] In a first aspect, a method for evaluating the answerability of questions automatically generated from text is provided, comprising the following steps:

[0007] Obtain the data set and expand the data set to obtain enhanced data;

[0008] A labeled dataset is obtained by labeling and voting on the augmented data using multiple question-answering models;

[0009] Encoding the context and question in the labeled dataset, and fusing the context features with the question features to obtain an embedded representation of the question after fusing the context;

[0010] The answerability of questions is evaluated based on a contrastive learning model using the embedded representation of the question after integrating the context.

[0011] As a preferred solution, the step of expanding the data set includes:

[0012] Randomly select a certain proportion of questions that do not belong to the same context as the source data for augmentation data, and use the method of obtaining relevant context to augment the context data;

[0013] A certain proportion of contexts are randomly selected as the source data for augmented data, and the method of obtaining related questions is used to expand the questions of the selected contexts.

[0014] As a preferred solution, the method for obtaining relevant context includes:

[0015] Use a search engine to retrieve several fragments related to the question, then calculate the relevance through a relevance algorithm, sort the relevance to obtain several more relevant contexts, and combine them into a new unlabeled sample.

[0016] As a preferred solution, the method for obtaining relevant questions includes:

[0017] Use semantic matching to embed semantic representations of all questions in the current data and generate a question semantic library. When a query question is input, the question semantic library outputs the question with the semantics closest to the query question.

[0018] Use the method of mutual translation, using translation technology to convert the source question into another language, and then translate the other language back to the original language through translation;

[0019] Using the synonym replacement method, some words in the question are replaced with synonyms to generate new questions while keeping the semantics close to the original question. During the replacement, the words in the question are traversed and a certain proportion of them are randomly replaced with synonyms.

[0020] As a preferred solution, in the step of annotating and voting on the enhanced data by using multiple question-answering models, the question-answering models select Distilbert-Base-Cased-Distilled-Squad, Roberta-base forQA and ChatGPT, and adopt the principle of minority obeys majority. Distilbert-Base-Cased-Distilled-Squad and Roberta-base for QA directly input context and corresponding questions to the model to answer, while ChatGPT asks questions in the form of templates, and constructs a voting mechanism for the above question-answering models.

[0021] As a preferred solution, the step of encoding the context and questions in the annotated dataset includes:

[0022] Generate a sequence representation of the context and add "cls" at the beginning of the text. The expression is as follows:

[0023] C=[cls,t1,t2,…t maxlen-1 ]

[0024] In the formula, t represents the vocabulary and punctuation in the context, cls is a special word, located at the beginning of the sentence, used to mark the entire sentence;

[0025] Use the following formula to find the number of the word in the vocabulary:

[0026] input=V lookup (C)

[0027] Where V lookup Represents a search method that returns the index of a word in the vocabulary;

[0028] Map the input to a word embedding vector as follows:

[0029] E c =WordEmbedding(input)

[0030] In the formula, WordEmbedding means mapping the sequence into a vector sequence, E c is the embedded representation of the context;

[0031] The position information of the word is encoded as follows:

[0032] P C =pos(input)*W p

[0033] Where, P C Represents the position embedding representation of the context, pos represents the method of obtaining the position of the character in the sentence, WP Indicates the calculation of P C The weight parameter of

[0034] According to the formula, E c With P C Perform the summation:

[0035] X c =E c +P C

[0036] Where, X c Denotes the embedding vector E c With position vector P C sum;

[0037] X c Use the self-attention mechanism to transform into Q c , V c , K c :

[0038] Q c =W q X c

[0039] V c =W v X c

[0040] K c =W k X c

[0041] Where W q , W v , W k Respectively represent the calculation of Q c , V c , K c The weight parameter of

[0042] After the self-attention transformation, Q c , V c , K c Z aggregated by the activation function softmax c and X c Add up to get the self-attention matrix The calculation expression is as follows:

[0043]

[0044]

[0045] Normalize the self-attention matrix as follows:

[0046]

[0047]

[0048] In the formula, LayerNorm means normalizing the matrix;

[0049] After mapping activation using the feedforward network, we get Then normalize:

[0050]

[0051] Use the feature vector of "cls" at the first position as the context representation e c ;

[0052]

[0053] The above steps are expressed as e c =BERT1[0], BERT means using the BERT model to extract features from text;

[0054] The same method is used to embed the BERT model used for anchor point questions, positive example questions, and negative example questions:

[0055] e a =BERT2[0]

[0056] e p =BERT2[0]

[0057] e n =BERT2[0]

[0058] Where, e a is the anchor sample embedding representation, e p is the embedding representation of the positive sample, e n Embedding representation for negative samples.

[0059] As a preferred solution, the steps of fusing context features with question features include:

[0060] In the input, replace e c Respectively with e a , e p , e n To perform the fusion:

[0061] t a =e c +e a

[0062] t p =e c +e p

[0063] t n =e c +e n

[0064] Where, t a is the representation of the anchor sample after fusion of context, t p is the representation of the positive sample after fusion of context, t n It is the representation of negative samples after fusion of context;

[0065] Then use multi-layer perceptron to learn features:

[0066] h a =MLP(t a )

[0067] h p =MLP(t p )

[0068] h n =MLP(t n )

[0069] Where h a is the hidden state vector of the anchor sample, h p is the hidden state vector of the positive sample, h n Represents the negative sample hidden state vector.

[0070] As a preferred solution, the answerability evaluation of the question based on the contrastive learning model using the embedded representation of the question after integrating the context includes:

[0071] In the text classification task, a given text sample is converted into a vector representation, and then Triplet loss is used to minimize the distance between text samples of the same category while maximizing the distance between different categories. In this way, samples with similar characteristics are mapped to similar embedding spaces. Contrastive learning is achieved through this loss function. The calculation expression is as follows:

[0072]

[0073] Where N represents the length of the hidden state vector of each sample, i represents the i-th hidden state vector in the sample, α is the interval parameter, and d is the distance function used to calculate the similarity between two variables.

[0074] In a second aspect, a system for evaluating the answerability of questions automatically generated from text is provided, comprising:

[0075] The data enhancement module is used to obtain and expand the data set to obtain enhanced data;

[0076] Annotation and voting module, configured to annotate and vote on the augmented data using multiple question-answering models to obtain annotated datasets;

[0077] An encoding fusion module is used to encode the context and question in the annotated dataset and fuse the context features with the question features to obtain an embedded representation of the question after fusion of the context;

[0078] The contrastive evaluation module is used to evaluate the answerability of questions based on the contrastive learning model by utilizing the embedded representation of the question after integrating the context.

[0079] According to a third aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for evaluating the answerability of the text-automatically generated questions is implemented.

[0080] Compared with the prior art, the present invention has at least the following beneficial effects:

[0081] Answerability evaluation is performed based on contrastive learning and context perception. By introducing contextual information and contrastive learning models, the problem that existing methods for evaluating the answerability of questions are not highly correlated with human evaluation and do not introduce contextual information is solved. Relevant data sets are obtained from texts in the education field, the data in the data sets are evaluated and analyzed, and the effect of contrastive learning is enhanced by expanding the data sets. A method of answering questions and voting using multiple question-answering models is used to annotate each context-question pair, which combines the respective advantages of multiple question-answering models and improves the annotation effect of enhanced data. The context and question in the annotated data set are encoded, and the context features are fused with the question features. The context information is integrated into the input so that the model can perceive the context. When the answerability of a question is evaluated based on a contrastive learning model, the input is the context-question pair in the expanded data set, and the output is the answerability score of the question. The maximum score is 1, and the closer it is to 1, the higher the possibility that the question can be answered. The present invention achieves a full evaluation of the answerability of the question. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only show some embodiments of the present invention. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0083] Figure 1 Flowchart of a method for evaluating the answerability of questions according to an embodiment of the present invention;

[0084] Figure 2Flowchart of data set expansion in the method of an embodiment of the present invention;

[0085] Figure 3 Flowchart of answerability score calculation in the method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0086] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, ordinary technicians in this field can also derive other embodiments without making any creative work.

[0087] The answerability evaluation method of the text-automatically generated questions of the present invention mainly includes obtaining relevant data sets from texts in the field of education, and evaluating and analyzing the data in the data sets. Then, some contexts and questions in the data sets are extracted, and the contexts related to the questions are obtained by using search engines and correlation calculation methods to expand the contexts; questions with similar meanings to the questions are obtained by using semantic matching, mutual translation and synonym replacement methods to expand the questions; and each context-question pair is annotated by answering questions and voting using multiple question-answering models to expand the data sets. The model algorithm based on contrastive learning evaluates the answerability of the questions, and the input of this module is the context-question pair in the expanded data set. The output is the answerability score of the question, and the maximum score is 1. The closer to 1, the higher the possibility that the question can be answered.

[0088] See also Figure 1 The answerability evaluation method of the text-generated question in the embodiment of the present invention specifically includes the following steps:

[0089] Step 1: Obtain the dataset and expand the dataset:

[0090] Through research on data sets, it was found that the only data sets containing unanswerable questions are SQuAD 2.0 and DuoRC. SQuAD (Stanford Question Answering Dataset) is a data set based on reading comprehension tasks, designed to test the reading comprehension ability of machines. SQuAD 1.1 is the first version of the data set, and each question has a corresponding context. According to statistical data, unanswerable negative samples account for a small proportion in question-answering data sets. In order to solve this problem, an embodiment of the present invention adopts a data expansion method to enhance the effect of contrastive learning by increasing the number of negative samples.

[0091] like Figure 2 As shown, the data expansion steps are as follows:

[0092] 1. Data Processing

[0093] To extract context-question pairs from the dataset, the embodiment uses the following two processing methods:

[0094] (1) Randomly select 20% of the questions that do not belong to the same context as the source data of the augmented data, and use the method of obtaining relevant context to augment these data.

[0095] (2) Randomly select 20% of the contexts as the source data for augmented data, and use the method of obtaining related questions to expand the questions of these contexts.

[0096] 2. Data augmentation

[0097] After preliminary data processing, the embodiment of the present invention adopts two methods to enhance the data to support subsequent text classification and comparative learning.

[0098] (1) Related context expansion method

[0099] The relevant context expansion method is to use a search engine to retrieve questions. The search engine will return several fragments related to the question, and then use the correlation algorithm to calculate the correlation between the top ten results and the context, sort them according to the relevance, select the top three most relevant contexts, and combine them into a new unlabeled sample.

[0100] (2) Related Questions Expansion Method

[0101] The method of expanding related questions is to expand a given question in order to generate a question that is semantically closest to it. The embodiment of the present invention adopts three means: First, semantic matching is used to perform semantic embedding representation on all questions in the current data to generate a question semantic library. When a query question is input, the semantic library will output the question that is semantically closest to the query question. Then, a mutual translation method is used, which uses mature translation technology to convert the source question into another language, and then translates it back to the original language through translation. Finally, the method of synonym replacement is used to generate new questions by replacing some words in the question with synonyms, and keep the semantics close to the source question. When replacing, the words in the question are traversed, and 30% of the words are randomly replaced with synonyms.

[0102] 3. Question-Answer Model Voting

[0103] Each context-question pair is annotated by answering questions and voting using a question-answering model. The embodiment of the present invention uses three different question-answering models for annotation, namely: Distilbert-Base-Cased-Distilled-Squad, Roberta-base for QA, and ChatGPT. The principle of minority obeys majority is adopted, and the question-answering models used have better performance in question-answering related fields. Distilbert-Base-Cased-Distilled-Squad and Roberta-base for QA directly input the context-question into the model to answer it, while ChatGPT adopts template questions in this article, and the template form is: "Can the following questions be answered from the passage? just answer YES OR NO." By constructing the above question-answering model voting mechanism, the enhanced data is processed.

[0104] See also Figure 3 ,Step 2: Context and question encoding of the data processed in step 1.

[0105] The method of the embodiment of the present invention uses the BERT model to encode the context. The encoding process will be described in detail below. First, a sequence representation of the context is generated, and "cls" is added at the beginning of the text:

[0106] C=[cls,t1,t2,…t maxlen-1 ]

[0107] In the formula, t represents the vocabulary and punctuation in the context, cls is a special word located at the beginning of the sentence, and this position is used to mark the entire sentence.

[0108] Then find the number of the word in the vocabulary:

[0109] input=V lookup (C)

[0110] Where V lookup Represents a search method that returns the index of a word in the vocabulary.

[0111] Then map the input into a word embedding vector:

[0112] E c =WordEmbedding(input)

[0113] In the formula, WordEmbedding is a word embedding technology used to map the sequence into a vector sequence, E c is the embedding representation of the context.

[0114] In addition to embedding the words, the position information of the words also needs to be encoded. The expression is as follows:

[0115] P C =pos(input)*W p

[0116] Where, P C Represents the position embedding representation of the context, and pos represents the method of obtaining the position of the character in the sentence. p Indicates the calculation of P C The weight parameter of .

[0117] E c With P C Perform the summation:

[0118] X c =E c +P C

[0119] Where, X c Denotes the embedding vector E c With position vector P C sum.

[0120] X c Use the self-attention mechanism to transform into Q c , V c , K c :

[0121] Q c =W q X c

[0122] V c =W v X c

[0123] K c =W k X c

[0124] Where W q , W v , W k Respectively represent the calculation of Q c , V c , K c The weight parameter of .

[0125] After the self-attention transformation, Q c , V c , K c Z aggregated by the activation function softmax cand X c Add up to get the self-attention matrix The formula is as follows:

[0126]

[0127]

[0128] Normalize the self-attention matrix as follows:

[0129]

[0130]

[0131] In the formula, LayerNorm means normalizing the matrix.

[0132] Finally, the feedforward network is used to map the activation and obtain Then normalize.

[0133]

[0134] Finally, the feature vector of “cls” at the first position is used as the representation of the context e c For the convenience of description, BERT is used in the following text to represent the feature extraction of text using the BERT model.

[0135]

[0136] The above steps are abbreviated as e x =BERT1[0]. After encoding the context, we need to encode the question. Next, we embed the BERT model for three types of questions: anchor questions, positive questions, and negative questions:

[0137] e a =BERT2[0]

[0138] e p =BERT2[0]

[0139] e n =BERT2[0]

[0140] Where, e a is the anchor sample embedding representation, e p is the embedding representation of the positive sample, e n Embedding representation for negative samples.

[0141] Step 3: Fusion of context features and question features.

[0142] This step fuses context features with question features, integrating context information into the input so that the model can perceive the context. c Respectively with e a , e p , e n to integrate.

[0143] t a =e c +e a

[0144] t p =e c +e p

[0145] t n =e c +e n

[0146] Where, t a is the representation of the anchor sample after fusion of context, t p is the representation of the positive sample after fusion of context, t n It is the representation of negative samples after fusion of context.

[0147] Then the feature learning is performed through a multi-layer perceptron (MLP).

[0148] h a =MLP(t a )

[0149] h p =MLP(t p )

[0150] h n =MLP(t n )

[0151] Where h a is the hidden state vector of the anchor sample, h p is the hidden state vector of the positive sample, h n represents the negative sample hidden state vector.

[0152] Step 4: Calculate the loss function.

[0153] Triplet loss is a loss function commonly used to train neural network models. Its goal is to cluster data points of the same category together while separating data points of different categories. In text classification tasks, for a given text sample, it can be converted into a vector representation, and then triplet loss is used to minimize the distance between text samples of the same category while maximizing the distance between different categories. This can map samples with similar characteristics to similar embedding spaces, thereby improving the performance of text classification tasks. This step uses this loss function to implement contrastive learning, which is calculated as follows:

[0154]

[0155] Where N represents the length of the hidden state vector of each sample, i represents the i-th hidden state vector in the sample, α is the margin parameter, and d is the distance function used to calculate the similarity between two variables.

[0156] Step 5: Experimental results and analysis

[0157] Correlation analysis is a method used to verify the correlation between algorithm results and human annotation scores.

[0158] Dataset: simQG dataset, which annotates the answerability scores of questions. Correlation analysis is used to detect the correlation between the scores output by the answerability model and the scores of manual annotations.

[0159] Comparison Method: This example experiment compares the Q-Metric method, which measures the answerability of questions based on similarity. While the metric can be based on lexical similarity, this example experiment used BLEU4 as its foundation. The experimental setup and platform remained consistent with the present invention.

[0160] Evaluation Metrics: Pearson, Spearman, and Kendall correlation coefficients were used for comparison. In addition to answerability evaluation methods, other evaluation methods in the field of question generation were also compared. These methods used the context and question as input for similar evaluation methods.

[0161] Table 1 Experimental results

[0162]

[0163] Table 1 shows that the answerability assessment method proposed in this paper outperformed the Q-BLEU4 model in human-evaluated correlation analysis experiments. Specifically, it scored higher than the Q-BLEU4 model on all three evaluation metrics (Pearson, Spearman, and Kendall). The model of this paper scored 0.134 and 0.137 higher than Q-BLEU4 in the Pearson and Spearman coefficients, respectively, and 0.138 higher in the Kendall coefficient. Other similarity assessment methods, such as BLEU4, ROUGE-L, and METEOR, which are based on lexical similarity, produced lower results. This is because the context and question are input during the comparison step, and the context is much longer than the question, further leading to lower results. This also confirms that the existing methods mentioned are incapable of assessing answerability. Furthermore, semantic similarity assessment methods such as BLEURT and BertScore showed higher results than lexical-based methods. This is because methods based on pretrained language models have anisotropy, resulting in higher results even when the input context is much longer than the question. In general, the method proposed in this invention is closer to the results of manual evaluation than other methods.

[0164] Another embodiment of the present invention further provides a system for evaluating the answerability of questions automatically generated from text, comprising:

[0165] The data enhancement module is used to obtain and expand the data set to obtain enhanced data;

[0166] Annotation and voting module, configured to annotate and vote on the augmented data using multiple question-answering models to obtain annotated datasets;

[0167] An encoding fusion module is used to encode the context and question in the annotated dataset and fuse the context features with the question features to obtain an embedded representation of the question after fusion of the context;

[0168] The contrastive evaluation module is used to evaluate the answerability of questions based on the contrastive learning model by utilizing the embedded representation of the question after integrating the context.

[0169] Another embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for evaluating the answerability of the text-automatically generated question is implemented.

[0170] Exemplarily, the instructions stored in the memory may be divided into one or more modules / units, which are stored in a computer-readable storage medium and executed by the processor to implement the answerability assessment method for automatically generated text questions of the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the server.

[0171] The electronic device may be a computing device such as a smartphone, laptop, PDA, or cloud server. The electronic device may include, but is not limited to, a processor and memory. Those skilled in the art will appreciate that the electronic device may include more or fewer components, or a combination of certain components, or different components. For example, the electronic device may also include input / output devices, network access devices, buses, and the like.

[0172] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0173] The memory may be an internal storage unit of the server, such as a hard disk or memory of the server. The memory may also be an external storage device of the server, such as a plug-in hard disk equipped on the server, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Furthermore, the memory may include both an internal storage unit of the server and an external storage device. The memory is used to store the computer-readable instructions and other programs and data required by the server. The memory may also be used to temporarily store data that has been output or is about to be output.

[0174] It should be noted that the information interaction, execution process, etc. between the above-mentioned module units are based on the same concept as the method embodiment. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0175] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0176] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunications signal and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk.

[0177] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0178] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A method for evaluating the answerability of questions automatically generated from text, characterized in that: The following steps are involved: Obtain the data set and expand the data set to obtain enhanced data; A labeled dataset is obtained by labeling and voting on the augmented data using multiple question-answering models; Encoding the context and question in the labeled dataset, and fusing the context features with the question features to obtain an embedded representation of the question after fusing the context; Using the embedded representation of the question after integrating the context, the answerability of the question is evaluated based on the contrastive learning model; The steps of encoding the context and questions in the labeled dataset include: Generate a sequence representation of the context and add "cls" at the beginning of the text. The expression is as follows: C=[cls,t1,t2,…t maxlen-1 ] In the formula, t represents the vocabulary and punctuation in the context, cls is a special word, located at the beginning of the sentence, used to mark the entire sentence; Use the following formula to find the number of the word in the vocabulary: input=V lookup (C) Where V lookup Represents a search method that returns the index of a word in the vocabulary; Map the input to a word embedding vector as follows: E c =WordEmbedding(input) In the formula, WordEmbedding means mapping the sequence into a vector sequence, E c is the embedded representation of the context; The position information of the vocabulary is encoded as follows: P C =pos(input)*W p Where, P C Represents the position embedding representation of the context, pos represents the method of obtaining the position of the character in the sentence, W p Indicates the calculation of P C The weight parameter of According to the formula, E c With P C Perform the summation: X c =And c +P C Where, X c Denotes the embedding vector E c With position vector P C sum; X c Use the self-attention mechanism to transform into Q c , V c , K c : Q c =W q X c V c =W v X c K c =W k X c Where W q , W v , W k Respectively represent the calculation of Q c , V c , K c The weight parameter of After the self-attention transformation, Q c , V c , K c Z aggregated by the activation function softmax c and X c Add up to get the self-attention matrix The calculation expression is as follows: Normalize the self-attention matrix as follows: In the formula, LayerNorm means normalizing the matrix; After mapping activation using the feedforward network, we get Then normalize: Use the feature vector of "cls" at the first position as the context representation e c ; The above steps are expressed as e c =BERT1[0], BERT means using the BERT model to extract features from text; The same method is used to embed the BERT model used for anchor point questions, positive example questions, and negative example questions: And a =BERT2[0] And p =BERT2[0] And n =BERT2[0] Where, e a is the anchor sample embedding representation, e p is the embedding representation of the positive sample, e n Embedding representation for negative samples.

2. The answerability evaluation method for automatically generated text questions according to claim 1, characterized in that: The steps of expanding the data set include: Randomly select a certain proportion of questions that do not belong to the same context as the source data for augmentation data, and use the method of obtaining relevant context to augment the context data; A certain proportion of contexts are randomly selected as the source data for augmented data, and the method of obtaining related questions is used to expand the questions of the selected contexts.

3. The answerability evaluation method for automatically generated text questions according to claim 2, characterized in that: The method for obtaining the relevant context includes: Use a search engine to retrieve several fragments related to the question, then calculate the relevance through a relevance algorithm, sort the relevance to obtain several more relevant contexts, and combine them into a new unlabeled sample.

4. The answerability evaluation method for automatically generated text questions according to claim 2, characterized in that: The method for obtaining relevant questions includes: Use semantic matching to embed semantic representations of all questions in the current data and generate a question semantic library. When a query question is input, the question semantic library outputs the question with the semantics closest to the query question. Use the method of mutual translation, using translation technology to convert the source question into another language, and then translate the other language back to the original language through translation; Using the synonym replacement method, some words in the question are replaced with synonyms to generate new questions while keeping the semantics close to the original question. During the replacement, the words in the question are traversed and a certain proportion of them are randomly replaced with synonyms.

5. The answerability evaluation method for automatically generated text questions according to claim 1, characterized in that: In the step of annotating and voting on the enhanced data by using multiple question-answering models, the question-answering models select Distilbert-Base-Cased-Distilled-Squad, Roberta-base for QA and ChatGPT, and adopt the principle of minority obeys majority. Distilbert-Base-Cased-Distilled-Squad and Roberta-base for QA directly input context and corresponding questions to the model, while ChatGPT asks questions in the form of templates, and constructs a voting mechanism for the above question-answering models.

6. The answerability evaluation method for automatically generated text questions according to claim 1, characterized in that: The steps of fusing context features with question features include: In the input, replace e c Respectively with e a , e p , e n To perform the fusion: t a =and c +e a t p =and c +e p t n =and c +e n Where, t a is the representation of the anchor sample after fusion of context, t p is the representation of the positive sample after fusion of context, t n The representation of negative samples after fusion of context; Then use multi-layer perceptron to learn features: h a =MLP(t a ) h p =MLP(t p ) h n =MLP(t n ) Where h a is the hidden state vector of the anchor sample, h p is the hidden state vector of the positive sample, h n represents the negative sample hidden state vector.

7. The answerability evaluation method for automatically generating questions from text according to claim 6, characterized in that: The answerability evaluation of the question based on the contrastive learning model using the embedded representation of the question after integrating the context includes: In the text classification task, a given text sample is converted into a vector representation, and then Tripletloss is used to minimize the distance between text samples of the same category while maximizing the distance between different categories. In this way, samples with similar characteristics are mapped to similar embedding spaces. Contrastive learning is achieved through this loss function. The calculation expression is as follows: Where N represents the length of the hidden state vector of each sample, i represents the i-th hidden state vector in the sample, α is the interval parameter, and d is the distance function used to calculate the similarity between two variables.

8. A system for evaluating the answerability of questions automatically generated from text, characterized in that: include: The data enhancement module is used to obtain and expand the data set to obtain enhanced data; Annotation and voting module, configured to annotate and vote on the augmented data using multiple question-answering models to obtain annotated datasets; An encoding fusion module is used to encode the context and question in the annotated dataset and fuse the context features with the question features to obtain an embedded representation of the question after fusion of the context; The steps of encoding the context and questions in the labeled dataset include: Generate a sequence representation of the context and add "cls" at the beginning of the text. The expression is as follows: C=[cls,t1,t2,…t maxlen-1 ] In the formula, t represents the vocabulary and punctuation in the context, cls is a special word, located at the beginning of the sentence, used to mark the entire sentence; Use the following formula to find the number of the word in the vocabulary: input=V lookup (C) Where V lookup Represents a search method that returns the index of a word in the vocabulary; Map the input to a word embedding vector as follows: E c =WordEmbedding(input) In the formula, WordEmbedding means mapping the sequence into a vector sequence, E c is the embedded representation of the context; The position information of the vocabulary is encoded as follows: P C =pos(input)*W p Where, P C Represents the position embedding representation of the context, pos represents the method of obtaining the position of the character in the sentence, W p Indicates the calculation of P C The weight parameter of According to the formula, E c With P C Perform the summation: X c =And c +P C Where, X c Denotes the embedding vector E c With position vector P C sum; X c Use the self-attention mechanism to transform into Q c , V c , K c : Q c =W q X c V c =W v X c K c =W k X c Where W q , W v , W k Respectively represent the calculation of Q c , V c , K c The weight parameter of After the self-attention transformation, Q c , V c , K c Z aggregated by the activation function softmax c and X c Add up to get the self-attention matrix The calculation expression is as follows: Normalize the self-attention matrix as follows: In the formula, LayerNorm means normalizing the matrix; After mapping activation using the feedforward network, we get Then normalize: Use the feature vector of "cls" at the first position as the context representation e c ; The above steps are expressed as e c =BERT1[0], BERT means using the BERT model to extract features from text; The same method is used to embed the BERT model used for anchor point questions, positive example questions, and negative example questions: And a =BERT2[0] And p =BERT2[0] And n =BERT2[0] Where, e a is the anchor sample embedding representation, e p is the embedding representation of the positive sample, e n Embedding representation for negative samples; The contrastive evaluation module is used to evaluate the answerability of questions based on the contrastive learning model by utilizing the embedded representation of the question after integrating the context.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for evaluating the answerability of a question automatically generated from text as claimed in any one of claims 1 to 7 is implemented.

Images