Quality evaluation method and system without reference problem based on context similarity calculation

Abstract

The invention discloses a non-reference problem quality evaluation method and system based on context similarity calculation, and the method comprises the steps: carrying out the feature extraction in each layer through a pre-training language model by using a trainable adaptive weight according to an input problem and context, connecting the weighted feature representations, and carrying out the feature extraction; generating a three-dimensional semantic representation of the question and the context; obtaining a correlation matrix between the context and the problem, generating a mask matrix by setting a threshold value, sparsifying the correlation matrix by using the mask matrix, and determining the content related to the problem in the context; the question features and content features related to the question in the context are fused from different levels; and calculating a similarity score between the fused problem features and content features related to the problem in the context to obtain an evaluation result. According to the method, the content related to the problem is determined from the context, the correlation degree of the generated problem and the context is evaluated, and the evaluation limitation caused by the fact that an existing method depends on manual annotation is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and particularly relates to a method and system for evaluating the quality of questions without reference based on context similarity calculation. Background Art

[0002] In recent years, with the continuous development of various information technologies, Natural Language Processing (NLP) has shown great potential in the field of artificial intelligence and has become an important driving force for promoting innovation. NLP aims to enable computers to better understand, process, and generate human natural language. As a form of human language expression, questions have also become one of the focuses of attention in the NLP field. In an educational scenario, questions are regarded as basic tools for evaluating teaching quality and promoting knowledge learning. Classroom teaching is the main way for students to learn various knowledge. By asking questions, the mastery of knowledge by students can be tested, thereby evaluating teaching quality. At the same time, when students answer questions, they will focus their attention on important learning materials and strengthen knowledge learning by repeatedly using core concepts. However, constructing a large number of high-quality questions manually is a quite time-consuming task.

[0003] The question generation task emerged as the times require, aiming to solve the challenges faced when constructing a large number of high-quality questions manually. Its definition is: given a text in natural language, automatically generate relevant and grammatically correct questions. The question generation task not only has broad prospects in the education field but also plays a positive role in promoting other NLP tasks. For example, the question generation task can be used as a dual task of the question-answering task to quickly construct a large number of question-answer pairs, thereby alleviating the problem of insufficient corpus in the question-answering task. In the field of dialogue tasks, the question generation task can be embedded as a part of the dialogue system to assist the dialogue system in more effectively understanding user needs and automatically generating relevant questions accordingly, thereby improving the naturalness and interactivity of the human-computer dialogue system.

[0004] The question generation task has developed rapidly in various fields. It has become increasingly important to reliably evaluate the quality of generated questions through a sound evaluation method. However, existing methods rely on manually annotated questions as references, which brings limitations to the evaluation of question quality. On the one hand, existing methods do not consider the key input context when generating questions. The question generation model can propose different questions by focusing on different contents in the same context, while the manually annotated questions can only represent one possible output. This may lead to the incorrect punishment of appropriate questions that are not similar to the manually annotated questions by existing methods. On the other hand, when the application model in the real scenario generates questions, since there is no manually annotated question as a reference, the quality of the generated questions cannot be evaluated at this time. Therefore, it is necessary to design an evaluation method that does not rely on manually annotated questions to solve the above limitations. Summary of the Invention

[0005] An object of the present invention is to provide a method and system for non-reference question quality assessment based on context similarity calculation for the problems in the above-mentioned prior art. Instead of using manually annotated questions as references, it determines the content related to the question from the context, calculates the similarity between the generated question and the relevant content, and evaluates the relevance between the generated question and the context.

[0006] To achieve the above object, the present invention has the following technical solutions:

[0007] In the first aspect, a method for non-reference question quality assessment based on context similarity calculation is provided, including:

[0008] According to the input question and context, use a pre-trained language model to extract features with trainable adaptive weights in each layer, and connect the weighted feature representations to generate a three-dimensional semantic representation of the question and context;

[0009] Based on the three-dimensional semantic representation of the question and context, obtain a correlation matrix between the context and the question, generate a mask matrix by setting a threshold, use the mask matrix to sparsify the correlation matrix, and determine the content in the context related to the question;

[0010] Fuse the question features and the content features in the context related to the question from different levels;

[0011] Calculate the similarity score between the question features after fusion and the content features in the context related to the question to obtain the evaluation result.

[0012] As a preferred solution, for a given context C = {w1, w2,..., w m} and question Q = {w1, w2,..., w n}, use a pre-trained BERT model as the encoder. By adding a [CLS] token in front of the text, adjust the text to the standard input of the BERT model, and the expression is as follows:

[0013] U c = {[CLS], w1, w2,..., w m}

[0014] U q = {[CLS], w1, w2,..., w n}

[0015] In the formula: U c is the text input to the encoder layer after preprocessing the context C; w i is the i-th word of the context or question; m is the length of the context C; U qThe text that is the input to the encoder layer after preprocessing for question Q; n is the length of question Q;

[0016] Each layer of the BERT model corresponds to a Transformer block. Assuming there are H Transformer blocks in the BERT model, the semantic representations of the context and the question are obtained through the h-th Transformer block, and the expression is as follows:

[0017]

[0018] In the formula: is the representation of the i-th token corresponding to the context of the h-th Transformer block; i ∈ [1, 2,..., m]; is the representation of the j-th token corresponding to the context of the h-th Transformer block; j ∈ [1, 2,..., n].

[0019] As a preferred solution, the feature extraction using trainable adaptive weights in each layer by the pre-trained language model and connecting the weighted feature representations to generate the three-dimensional semantic representation of the question and the context includes the following steps: Connect the representations of different layers of the BERT model using trainable weights, and obtain the unnormalized variable through the following formula:

[0020]

[0021] In the formula: is the unnormalized attention variable of the context; W1 h 、 are the trainable parameters of the h-th Transformer block; is the bias; σ is the sigmoid activation function;

[0022] Calculate the normalized attention weights, and the expression is as follows:

[0023]

[0024] In the formula: is the attention weight corresponding to the context; is the attention weight corresponding to the question;

[0025] Weighted concatenation of the representations of each layer according to the obtained weights to generate the three-dimensional semantic representation of the question and the context:

[0026]

[0027] In the formula: X c is the semantic representation of the context; X c ∈ R H×m×d; H is the number of blocks in the Transformer encoder; X q is the semantic representation of the question; X q ∈R H×n×d .

[0028] As a preferred solution, the problem features and the content features related to the problem in the context are interacted through matrix multiplication, and the result is scaled. The correlation matrix between the context and the problem is obtained through the following formula:

[0029]

[0030] In the formula: Q Rel is the query vector corresponding to the problem features ; Φ(,) represents the linear transformation layer; is the i-th feature in X q ; is the weight matrix initialized randomly; K Rel is the key vector corresponding to the context features ; is the i-th feature in X c ; M Rel is the correlation matrix between the problem and the context; D Rel is the dimension corresponding to K Rel .

[0031] As a preferred solution, the threshold is obtained through the average value of the weights, and the calculation expression is as follows:

[0032]

[0033] In the formula: t = {t1, t2,..., t n} is a vector with a dimension of n; t i is the threshold corresponding to each row of the correlation matrix;

[0034] By introducing a mask matrix, the correlation matrix is sparsified; for each row of the correlation matrix, if the value within the row is greater than the threshold of the corresponding row, the corresponding value is retained, and the corresponding value of the mask matrix is 1; otherwise, the value within the corresponding row is ignored, and the corresponding value of the mask matrix is 0. The expression is as follows:

[0035]

[0036] In the formula: M Mask is the mask matrix;

[0037] Element-wise multiplication of the mask matrix and the correlation matrix is performed to achieve sparsification. The sparsification of the correlation matrix retains relatively high attention scores and filters out relatively low attention scores, filtering out content with low relevance to the context. The expression is as follows:

[0038] M Spa = M Mask ⊙ M Rel

[0039] In the formula: M Spa is the sparsified matrix; M Mask is the mask matrix;

[0040] The weights of the matrix are balanced through Softmax calculation to obtain the features after context sparsification. The features after context sparsification contain the content relevant to the question and the context; the content relevant to the question and the context in each layer is determined respectively, and the content relevant to each layer is concatenated to obtain the overall context-related content. The calculation expression is as follows:

[0041]

[0042] M = Softmax(M Spa )

[0043]

[0044] In the formula: V Rel is the key vector corresponding to the context feature ; is the randomly initialized weight matrix; M is the weight of the balanced matrix to obtain the final matrix; is the result of context feature sparsification; is the feature of context sparsification, that is, the feature of context-related content;

[0045] As a preferred solution, based on the Inception architecture, the question features and the content features related to the question in the context are fused from different levels;

[0046] Multi-scale features are obtained through convolutional kernels of different sizes, and the multi-scale features are expanded by dilated convolution to increase the receptive field. The expression is as follows:

[0047]

[0048] In the formula: is the result of the i-th convolution of the question feature; is the i-th convolution with different kernel sizes; is the dilation rate of ri The i-th convolution; Have the same kernel size; Have different kernel sizes; i ∈ [1, 2, 3]; j ∈ [1, 2, 3]; Is the result of the j-th convolution of the problem features;

[0049] Use a 1×1 convolutional layer to encode the feature vectors and concatenate all the output feature vectors. The expression is as follows:

[0050]

[0051] In the formula: Is the feature representation of the problem; φ0 is a 1x1 convolutional layer; Is the feature representation of the context-related content.

[0052] As a preferred solution, in the step of calculating the similarity score between the problem features after the calculation fusion and the content features related to the problem in the context, the content features related to the problem in the context and the problem features are used to calculate the similarity through element-wise multiplication, and are combined with the features themselves and input into the global average pooling layer and the fully connected layer to obtain the similarity score. The expression is as follows:

[0053]

[0054] In the formula: v is the vector after the fusion of the problem and the relevant content; GAP() is the global average pooling operation;

[0055] Use cross-entropy loss to predict the similarity of the vector v after the fusion of the problem and the relevant content through a linear layer. The expression is as follows:

[0056] y p = Softmax(W4ReLU(W3(v)+b3)+b4)

[0057] In the formula: y p Is the predicted label; W3, W4 are trainable weights; b3, b4 are biases.

[0058] In a second aspect, a reference-free problem quality assessment system based on context similarity calculation is provided, including:

[0059] An adaptive feature extraction module, which is used to extract features by using trainable adaptive weights in each layer through a pre-trained language model according to the input problem and context, and concatenate the weighted feature representations to generate a three-dimensional semantic representation of the problem and the context;

[0060] A sparsification filtering module, which is used to obtain a correlation matrix between the context and the question based on the three-dimensional semantic representation of the question and the context, generate a mask matrix by setting a threshold, sparsify the correlation matrix using the mask matrix, and determine the content in the context related to the question;

[0061] A multi-level feature fusion module, which is used to fuse the question features and the content features related to the question in the context from different levels;

[0062] A similarity score calculation and evaluation module, which is used to calculate the similarity score between the question features and the content features related to the question in the context after fusion, and obtain an evaluation result.

[0063] In a third aspect, there is provided an electronic device, including:

[0064] A memory that stores at least one instruction; and a processor that executes the instruction stored in the memory to implement the method for evaluating the quality of a question without reference based on context similarity calculation.

[0065] In a fourth aspect, there is provided a computer-readable storage medium, in which at least one instruction is stored, and the at least one instruction is executed by a processor in an electronic device to implement the method for evaluating the quality of a question without reference based on context similarity calculation.

[0066] Compared with the prior art, the present invention has at least the following beneficial effects:

[0067] In order to retain as much complete information of the text as possible, the present invention uses trainable adaptive weights in each layer of the pre-trained language model for feature extraction, and connects the weighted feature representations to generate a three-dimensional semantic representation of the question and the context. In order to determine the content related to the question from the context, the present invention obtains a correlation matrix between the context and the question, generates a mask matrix by setting a threshold, sparsifies the correlation matrix using the mask matrix, determines the content in the context related to the question, and filters out the context features unrelated to the question. In order to fully explore the relationship between the features at different levels of the text and obtain a better representation of the text, the present invention fuses the question features and the content features related to the question in the context from different levels, so as to better understand the features of the question and the context. Finally, by calculating the similarity score between the question features and the content features related to the question in the context after fusion, an evaluation result of the correlation degree between the question and the context is obtained. The method for evaluating the quality of a question without reference based on context similarity calculation of the present invention does not require manual annotation of questions as a reference, but determines the content related to the question from the context, calculates the similarity between the generated question and the relevant content, and evaluates the correlation degree between the generated question and the context, thus effectively solving the evaluation limitation brought by the existing methods that need to rely on manual annotation of questions. Brief Description of the Drawings

[0068] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention, and those of ordinary skill in the art can obtain other related drawings based on these drawings without creative efforts.

[0069] Figure 1 It is a flowchart of a reference-free question quality assessment method based on context similarity calculation according to an embodiment of the present invention;

[0070] Figure 2 It is a flowchart of determining the content related to the question in the context by sparsification according to an embodiment of the present invention;

[0071] Figure 3 It is a flowchart of fusing question features and relevant content features from different levels according to an embodiment of the present invention. Detailed Embodiments

[0072] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all of them. Based on the embodiments of the present invention, those of ordinary skill in the art can obtain other embodiments without creative efforts.

[0073] Please refer to Figure 1 , an embodiment of the present invention provides a reference-free question quality assessment method based on context similarity calculation, including:

[0074] S1. According to the input question and context, use a pre-trained language model to extract features with trainable adaptive weights in each layer, and connect the weighted feature representations to generate a three-dimensional semantic representation of the question and context;

[0075] S2. Obtain a correlation matrix between the context and the question based on the three-dimensional semantic representation of the question and context, generate a mask matrix by setting a threshold, and use the mask matrix to sparsify the correlation matrix to determine the content related to the question in the context;

[0076] S3. Fuse the question features and the content features related to the question in the context from different levels;

[0077] S4. Calculate the similarity score of the fused question features and the content features related to the question in the context to obtain the evaluation result.

[0078] In a possible implementation, to help the model effectively retain semantic information at different levels, step S1 maps the semantic information to a higher-dimensional space, thereby transforming the original two-dimensional features into three-dimensional features, comprehensively and effectively retaining semantic information at different levels. To achieve this goal, an adaptive weighting method is introduced to extract features from the representations of each layer. At the same time, a stacking and concatenation method is used to combine features at different levels to provide more complete text information for the next sparsification filtering. Compared with the traditional pooling operation method for compressing semantic representations at different levels, the embodiment of the present invention proposes to use trainable adaptive weights in each layer for feature extraction and connect these weighted feature representations, effectively retaining semantic information at different levels and providing support for determining relevant content in the next step.

[0079] For a given context C = {w1, w2,..., w m} and question Q = {w1, w2,..., w n}, the embodiment of the present invention uses a pre-trained BERT model as an encoder to obtain the representation of the input sentence. By adding the [CLS] token at the front of the text, the text is adjusted to the standard input of the BERT model, and the expression is as follows:

[0080] U c = {[CLS], w1, w2,..., w m}

[0081] U q = {[CLS], w1, w2,..., w n}

[0082] In the formula: U c is the text input to the encoder layer after preprocessing the context C; w i is the i-th word of the context or the question; m is the length of the context C; U q is the text input to the encoder layer after preprocessing the question Q; n is the length of the question Q.

[0083] Each layer of the BERT model corresponds to a Transformer block. Assuming there are H Transformer blocks in the BERT model, the semantic representations of the context and the question can be obtained through the h-th Transformer block, and the expression is as follows:

[0084]

[0085] In the formula: is the representation of the i-th token corresponding to the context of the h-th Transformer block; i ∈ [1, 2,..., m]; is the representation of the jth token corresponding to the hth Transformer block context; j∈[1,2,...,n].

[0086] In order to retain the semantic features of text at different layers as much as possible, the representations of different layers are connected using trainable weights.

[0087] First, obtain the non-normalized variables, the expression is as follows:

[0088]

[0089] Where: is the non-normalized attention variable of the context; W1 h , is the trainable parameter of the h-th Transformer block; is the bias; σ is the sigmoid activation function.

[0090] Secondly, calculate the normalized attention weight, the expression is as follows:

[0091]

[0092] Where: is the corresponding attention weight of the context; is the attention weight corresponding to the question.

[0093] Finally, the representation of each layer is weighted and concatenated according to the calculated weights to generate a three-dimensional semantic representation of the question and context for the next step of sparse filtering. The expression is as follows:

[0094]

[0095] Where: X c is the semantic representation of the context; X c ∈R H×m×d ; H is the number of blocks in the Transformer encoder; X q is the semantic representation of the problem; X q ∈R H×n×d .

[0096] See also Figure 2, in a possible implementation, to determine the relevant content of the context with respect to the question, step S2 generates a threshold through a function and uses this threshold to generate a mask matrix. The context features are sparsified through this mask matrix to determine the relevant content in the context. Since the time and space complexity of attention calculation is relatively high, introducing a complex threshold generation algorithm will increase the computational burden. Therefore, a lightweight algorithm is selected to generate the threshold, that is, using the average value of the weights. This not only simplifies the calculation but also simulates the influence of experience on attention in human visual processing.

[0097] First, step S2 interacts the question features and the context features through matrix multiplication and scales the result, aiming to obtain the correlation matrix between the context and the question. The calculation expression is as follows:

[0098]

[0099] In the formula: Q Rel is the query vector corresponding to the question features ; Φ(,) represents the linear transformation layer; is the i-th feature in X q ; is the randomly initialized weight matrix; K Rel is the key vector corresponding to the context features ; is the i-th feature in X c ; M Rel is the correlation matrix between the question and the context; D Rel is the dimension corresponding to K Rel .

[0100] Secondly, since all features in the question and the context participate in the generation of the weights of the correlation matrix, those context features that are not relevant to the question will also participate in the weight calculation. This not only affects the weight distribution of attention but also reduces the accuracy of the similarity score between the relevant content of the question and the context. Therefore, the embodiments of the present invention take measures to filter out irrelevant features. The influence of irrelevant features is alleviated by setting a threshold, and the threshold is obtained through the average value of the weights. The calculation expression is as follows:

[0101]

[0102] In the formula: t = {t1, t2,..., t n} is a vector with a dimension of n; t i is the threshold corresponding to each row of the correlation matrix.

[0103] A mask matrix corresponding to the correlation matrix can be obtained based on a threshold, and the correlation matrix can be sparsified through this matrix. To prevent the hard-coded method of setting values to 0 from having an adverse effect on the gradient update and backpropagation of the model, the embodiment of the present invention adopts the method of the mask matrix, and sparsifies the correlation matrix by introducing a mask matrix. For each row of the correlation matrix, if the value within the row is greater than the threshold of the corresponding row, the value is retained and the corresponding value in the mask matrix is 1; otherwise, the value is ignored and the corresponding value in the mask matrix is 0. The expression is as follows:

[0104]

[0105] In the formula: M Mask is the mask matrix.

[0106] After that, element-wise multiplication of the mask matrix and the correlation matrix is performed to achieve sparsification. Sparsification of the correlation matrix can retain relatively high attention scores and filter out relatively low attention scores, that is, filter out content with low relevance to the context. The expression is as follows:

[0107] M Spa = M Mask ⊙ M Rel

[0108] In the formula: M Spa is the sparsified matrix; M Mask is the mask matrix.

[0109] Finally, Softmax calculation is performed to balance the weights of the matrix to obtain the final matrix, and context-sparsified features are obtained based on this matrix. The sparsified context features contain content that is relatively relevant to the question and the context. Since both the context and question features are three-dimensional, the relevant content between the question and the context in each layer is determined respectively, and the relevant content of each layer is concatenated to obtain the overall context-related content. The expression is as follows:

[0110]

[0111] M = Softmax(M Spa )

[0112]

[0113] In the formula: V Rel is the key vector corresponding to the context feature ; is the randomly initialized weight matrix; M is the final matrix obtained by balancing the weights of the matrix; is the context feature after sparsification; Features for context sparsification, i.e., context-related content features;

[0114] Please refer to Figure 3 , in a possible implementation, the problems and features in the context involved in steps S1 and S2 are all operated on the hierarchical dimension, and the layers are independent of each other, and the relationships between features of different levels are not mined and utilized. To make full use of such three-dimensional features and mine the relationships between features of different levels, multi-level feature fusion is performed through step S3. Step S3 can effectively extract features of different scales and sizes of problems and contexts, so as to better understand the features of problems and contexts. Step S3 is based on the Inception architecture, uses a multi-branch structure, and uses convolutional layers of different sizes. In addition, to further expand the receptive field, dilated convolutions widely used in image semantic segmentation are also applied.

[0115] For the three-dimensional structure of problem and context features, in order to better understand problem features and context-related content features, multi-scale features are obtained through convolutional kernels of different sizes, and the multi-scale features are used to expand the receptive field through dilated convolutions. The expression is as follows:

[0116]

[0117] In the formula: is the result of the i-th convolution of the problem feature; is the i-th convolution with different kernel sizes; is the i-th convolution with a dilation rate of r i ; have the same kernel size; have different kernel sizes; i ∈ [1, 2, 3]; j ∈ [1, 2, 3]; is the result of the j-th convolution of the problem feature.

[0118] In addition to dilated convolutions, in order to help the model understand the global semantic information of the text, a 1×1 convolutional layer is used to encode the feature vectors, and all output feature vectors are concatenated. The expression is as follows:

[0119]

[0120] In the formula: is the feature representation of the problem; φ0 is the 1x1 convolutional layer; is the feature representation of the context-related content.

[0121] In a possible implementation manner, the method for evaluating the quality of questions without reference based on context similarity calculation in the embodiments of the present invention is to calculate the similarity between the question and the relevant content by determining the relevant content of the context and the question to evaluate the relevance degree between the generated question and the context when there is no manually annotated question as a reference. Finally, the relevant content features and question features are subjected to similarity calculation through element multiplication, and are combined with the features themselves, and they are input into the global average pooling layer and the fully connected layer to obtain the final similarity score. The expression for calculating the similarity score in step S4 is as follows:

[0122]

[0123] In the formula: v is the vector after fusing the question and the relevant content; GAP() is the global average pooling operation.

[0124] The vector v after fusing the question and the relevant content is a high-level representation of the similarity between the question and the relevant content. The final similarity is predicted through a linear layer, and the cross-entropy loss is adopted. The expression is as follows:

[0125] y p = Softmax(W4ReLU(W3(v)+b3)+b4)

[0126] In the formula: y p is the predicted label; W3 and W4 are trainable weights; b3 and b4 are biases.

[0127] The method for evaluating the quality of questions without reference based on context similarity calculation in the present invention is further described through experiments as follows:

[0128] The SQuAD dataset, which is widely used in QA and QG research, is used. The SimQG dataset is a benchmark dataset for the question quality evaluation task. The SimQG dataset contains the quality evaluation results of machine-generated questions annotated by crowd workers. More specifically, the dataset contains 500 pieces of data, and each piece of data contains context, reference question, generated question, similarity score, answerability score, fluency score, and relevance score. Among them, the relevance score indicates whether the question is relevant to the article and whether it is only based on the content of the article. The score ranges from 0 to 1, and the closer it is to 1, the better.

[0129] Comparison methods: including methods based on similarity calculation and methods based on dissimilarity calculation. The inputs of these methods are both context and generated questions. The comparison experiment aims to compare the performance of these evaluation methods without manually annotated questions as references to verify the effectiveness of the methods proposed in the embodiments of the present invention. The comparison methods include: the representative evaluation metric BLEU based on exact matching n-grams, the ROUGE-L method based on the longest common subsequence, the improved METEOR method based on BLEU, the BERTScore method based on the BERT embedding vectors of each word in the generated text and the reference text, the MoverScore method for calculating the difference by the earth mover's distance, the BLEURT method trained based on synthetic data, the QSTS method based on multi-angle similarity calculation, the GPTScore method based on the generative pre-trained model, and the QRelScore method for calculating the relevance between the generated question and the context.

[0130] Evaluation metrics: The performance of the automatic evaluation method can be measured by calculating the correlation between the automatic evaluation result and the manual evaluation result. Three correlation coefficients are adopted, including the Pearson correlation coefficient, the Spearman correlation coefficient, and the Kendall correlation coefficient.

[0131] Table 1 Experimental results

[0132]

[0133] It can be analyzed from the results of the comparison experiment shown in Table 1 that:

[0134] In the comparison experiment, the evaluation methods based on similarity calculation include BLEU4, ROUGE-L, METEOR, BERTScore, MoverScore, BLEURT, QSTS, and the methods based on dissimilarity calculation include GPTScore, QRelScore. Among them, QRelScore is a question relevance evaluation method designed for the question generation task. The experimental results show that the method of the present invention is significantly better than other evaluation methods in terms of the three correlation coefficients of Pearson, Spearman, and Kendall, and is also higher than the QRelScore method for calculating the relevance between the question and the context designed for the question generation task. This indicates that the evaluation result of the method of the present invention is closer to the human evaluation result, and at the same time proves the effectiveness of the reference-free question quality evaluation method of the present invention.

[0135] According to the results in the table, the correlation coefficients of other methods based on similarity calculation are much lower than those of the method proposed in the present invention. This indicates that when there is no manual annotation of problems as a reference, it is difficult to evaluate the quality of problems for existing problem quality assessment methods based on similarity calculation. In addition, GPTScore has a certain performance improvement compared to the methods based on similarity calculation, but there is still a certain gap compared to the QRelScore method designed for the problem generation task and the method of the present invention. Compared with the QRelScore method, the three correlation coefficients of the method of the present invention are respectively 0.033, 0.037, and 0.028 higher, which indicates that the method of the present invention has better performance than the QRelScore method. When calculating the correlation between the problem and the context, the QRelScore method does not consider the influence of a large amount of noise information in the context on the calculation results. The present invention determines the content related to the problem from the context through the method of sparsification filtering, and then calculates the similarity between the generated problem and the relevant content to evaluate the relevance between the problem and the context, obtaining better performance, which indicates the effectiveness of determining context-related content through sparsification filtering.

[0136] Another embodiment of the present invention provides a reference-free problem quality assessment system based on context similarity calculation, including:

[0137] An adaptive feature extraction module, configured to extract features by using trainable adaptive weights in each layer of a pre-trained language model according to the input problem and context, and connect the weighted feature representations to generate a three-dimensional semantic representation of the problem and the context;

[0138] A sparsification filtering module, configured to obtain a correlation matrix between the context and the problem based on the three-dimensional semantic representation of the problem and the context, generate a mask matrix by setting a threshold, sparsify the correlation matrix by using the mask matrix, and determine the content in the context related to the problem;

[0139] A multi-level feature fusion module, configured to fuse the problem features and the content features related to the problem in the context from different levels;

[0140] A similarity score calculation and evaluation module, configured to calculate the similarity score between the problem features after fusion and the content features related to the problem in the context to obtain an evaluation result.

[0141] Another embodiment of the present invention further provides an electronic device, including:

[0142] A memory storing at least one instruction; and a processor, configured to execute the instruction stored in the memory to implement the reference-free problem quality assessment method based on context similarity calculation.

[0143] Another embodiment of the present invention further provides a computer-readable storage medium, in which at least one instruction is stored, and the at least one instruction is executed by a processor in an electronic device to implement the no-reference question quality assessment method based on context similarity calculation.

[0144] Exemplarily, the instructions stored in the memory can be divided into one or more modules / units. The one or more modules / units are stored in the computer-readable storage medium and executed by the processor to complete the no-reference question quality assessment method based on context similarity calculation of the present invention. The one or more modules / units can be a series of computer-readable instruction segments capable of completing specific functions, and these instruction segments are used to describe the execution process of the computer program in the server.

[0145] The electronic device can be a computing device such as a smart phone, a notebook, a palm computer, and a cloud server. The electronic device may include, but is not limited to, a processor and a memory. Those skilled in the art can understand that the electronic device may further include more or fewer components, or combine certain components, or different components. For example, the electronic device may further include input / output devices, network access devices, a bus, etc.

[0146] The processor may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0147] The memory may be an internal storage unit of the server, such as the hard disk or memory of the server. The memory may also be an external storage device of the server, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the server. Further, the memory may also include both the internal storage unit and the external storage device of the server. The memory is used to store the computer-readable instructions and other programs and data required by the server. The memory may also be used to temporarily store the data that has been output or will be output.

[0148] It should be noted that for the content such as information interaction and execution process between the above-mentioned module units, since it is based on the same concept as the method embodiment, for its specific functions and the resulting technical effects, reference can be specifically made to the method embodiment section, and details will not be elaborated here.

[0149] Those skilled in the art can clearly understand that for the convenience and conciseness of description, only the above-mentioned division of each functional unit and module is used as an example. In actual applications, the above-mentioned functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working process of the units and modules in the above system can refer to the corresponding process in the foregoing method embodiment, and details will not be elaborated here.

[0150] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the above method embodiments of this application, a computer program can be used to instruct the relevant hardware to complete. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can at least include: any entity or device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium that can carry the computer program code to the photographing device / terminal device. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disc, etc.

[0151] In the above embodiments, the descriptions of each embodiment have their own emphases. For the parts not elaborated or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0152] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A no-reference question quality assessment method based on context similarity calculation, characterized in that: include: Based on the input question and context, the pre-trained language model uses trainable adaptive weights in each layer to extract features and connect the weighted feature representations to generate a three-dimensional semantic representation of the question and context. Based on the three-dimensional semantic representation of the question and the context, the correlation matrix between the context and the question is obtained, a mask matrix is ​​generated by setting a threshold, and the mask matrix is ​​used to sparse the correlation matrix to determine the content in the context that is relevant to the question; Integrate the problem features and the content features related to the problem in the context from different levels; The similarity scores of the fused question features and the content features related to the question in the context are calculated to obtain the evaluation results.

2. The method for quality assessment of no-reference questions based on context similarity calculation according to claim 1, characterized in that: For a given context C = {w1,w2,...,w m } and the problem Q = {w1,w2,...,w n }, use the pre-trained BERT model as the encoder, and adjust the text to the standard input of the BERT model by adding the [CLS] tag before the text. The expression is as follows: U c ={[CLS],w1,w2,...,w m } U q ={[CLS],w1,w2,...,w n } Where: U c The text input to the encoder layer after preprocessing for context C; w i is the i-th word in the context or question; m is the length of the context C; U q The text input to the encoder layer after preprocessing for question Q; n is the length of question Q; Each layer of the BERT model corresponds to a Transformer block. Assuming that there are H Transformer blocks in the BERT model, the semantic representation of the context and question is obtained through the hth Transformer block, as shown below: Where: is the representation of the i-th token corresponding to the h-th Transformer block context; i∈[1,2,...,m]; is the representation of the jth token corresponding to the hth Transformer block context; j∈[1,2,...,n].

3. The method for quality assessment of reference-free questions based on context similarity calculation according to claim 2, characterized in that: The method of extracting features using a trainable adaptive weight in each layer of the pre-trained language model and connecting the weighted feature representations to generate a three-dimensional semantic representation of the question and context includes the following steps: The representations of different layers of the BERT model are connected using trainable weights to obtain non-normalized variables using the following formula: Where: is the non-normalized attention variable of the context; W1 h 、W2 h is the trainable parameter of the h-th Transformer block; is the deviation; σ is the sigmoid activation function; Calculate the normalized attention weight, the expression is as follows: Where: is the attention weight corresponding to the context; is the attention weight corresponding to the question; The representation of each layer is weighted concatenated according to the obtained weights to generate a three-dimensional semantic representation of the question and context: Where: X c is the semantic representation of the context; X c ∈R H×m×d ; H is the number of blocks in the Transformer encoder; X q is the semantic representation of the problem; X q ∈R H×n×d .

4. The method for quality assessment of no-reference questions based on context similarity calculation according to claim 1, characterized in that: The question features and the content features related to the question in the context are interacted through matrix multiplication, and the results are scaled to obtain the correlation matrix between the context and the question through the following formula: Where: Q Rel Problem Characteristics The corresponding query vector; Φ(,) represents the linear transformation layer; For X q The i-th feature in ; i∈[1,H]; is a randomly initialized weight matrix; K Rel Contextual Features The corresponding key vector; For X c The i-th feature in ; M Rel is the correlation matrix between the question and the context; D Rel K Rel The corresponding dimension.

5. The method for quality assessment of reference-free questions based on context similarity calculation according to claim 4, characterized in that: The threshold is obtained by the average value of the weights, and the calculation expression is as follows: Where: t={t1,t2,...,t n } is a vector of dimension n; t i is the threshold corresponding to each row of the correlation matrix; By introducing a mask matrix, the correlation matrix is ​​sparsed; for each row of the correlation matrix, if the value in the row is greater than the threshold of the corresponding row, the corresponding value is retained and the corresponding value of the mask matrix is ​​1; otherwise, the value in the corresponding row is ignored and the corresponding value of the mask matrix is ​​0. The expression is as follows: Where: M Mask is the mask matrix; The mask matrix is ​​element-wise multiplied with the relevance matrix to achieve sparseness. The relevance matrix sparseness retains relatively high attention scores and filters out relatively low attention scores, filtering out content with low context relevance. The expression is as follows: M Spa =M Mask ⊙M Rel Where: M Spa is the sparse matrix; M Mask is the mask matrix; The weight of the matrix is ​​balanced by Softmax calculation to obtain the context-sparse features, where the context-sparse features contain the content related to the question and the context; Determine the content related to the context in each layer of the question, and connect the related content of each layer in series to obtain the content related to the overall context. The calculation expression is as follows: M=Softmax(M Spa ) Where: V Rel Contextual Features The corresponding key vector; is a randomly initialized weight matrix; M is the weight of the balanced matrix to obtain the final matrix; It is a context feature The result after sparseness; It is the context-sparse features, that is, the features of context-related content; 6. The method for quality assessment of no-reference questions based on context similarity calculation according to claim 1, characterized in that: Based on the Inception architecture, the problem features and the content features related to the problem in the context are integrated from different levels; Multi-scale features are obtained through convolution kernels of different sizes, and the receptive field of multi-scale features is expanded through dilated convolution. The expression is as follows: Where: is the result of the i-th convolution of the problem feature; is the i-th convolution with different kernel sizes; The expansion rate is r i The i-th convolution; The core size is the same; The kernel sizes are different; i∈[1,2,3]; j∈[1,2,3]; is the result of the j-th convolution of the problem feature; Use a 1×1 convolutional layer to encode the feature vector and concatenate all the output feature vectors. The expression is as follows: Where: Characterize the problem; φ0 is a 1x1 convolutional layer; It is a feature representation of context-related content.

7. The method for quality assessment of no-reference questions based on context similarity calculation according to claim 6, characterized in that: In the step of calculating the similarity score between the fused question feature and the question-related content feature in the context, the question-related content feature in the context and the question feature are similarly calculated by element multiplication, and are merged together with the feature itself, input into the global average pooling layer and the fully connected layer, and the similarity score is obtained. The expression is as follows: Where: v is the vector after the question and related content are fused; GAP() is the global average pooling operation; The cross entropy loss is used to predict the similarity of the vector v after the problem and the related content are fused through the linear layer. The expression is as follows: <h2 style=";text-align:left;direction:ltr">y<h2 style=";text-align:left;direction:ltr"> p <h2 style=";text-align:left;direction:ltr"> =Softmax(W4 ReLU(W3(v)+b3)+b4) Where: y p is the predicted label; W3 and W4 are trainable weights; b3 and b4 are biases.

8. A no-reference question quality assessment system based on context similarity calculation, characterized in that: include: An adaptive feature extraction module is used to extract features based on the input question and context using trainable adaptive weights in each layer through a pre-trained language model, and concatenate the weighted feature representations to generate a three-dimensional semantic representation of the question and context; A sparse filtering module is used to obtain a correlation matrix between the context and the question based on the three-dimensional semantic representation of the question and the context, generate a mask matrix by setting a threshold, and use the mask matrix to sparse the correlation matrix to determine the content in the context that is relevant to the question; Multi-level feature fusion module, used to fuse question features and content features related to the question in the context from different levels; The similarity score calculation evaluation module is used to calculate the similarity scores of the fused question features and the content features related to the question in the context to obtain the evaluation results.

9. An electronic device, characterized in that: include: A memory storing at least one instruction; and A processor executes instructions stored in the memory to implement the reference-free question quality assessment method based on context similarity calculation as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, and the at least one instruction is executed by a processor in an electronic device to implement the reference-free question quality assessment method based on context similarity calculation as described in any one of claims 1 to 7.

Images